How does SudoChat know when it does not have enough evidence to answer?

Author: Mustafa Siddiqui · Part: 3 of 3 · Recruiter question: How does SudoChat know when it does not have enough evidence to answer?

Evidence rule: This is my first-person authored source material. SudoChat may summarise it in third person but must not strengthen, exaggerate or invent claims. Limitations and Do not claim sections are hard factual boundaries.

## Why this matters to me
I believe one of the most important misconceptions about AI is that a useful assistant should answer as many questions as possible.

For high trust systems, that is the wrong metric.

An assistant that answers 100 percent of questions but invents information is less useful than one that answers 80 percent accurately and clearly identifies the remaining 20 percent as unsupported.

The appropriate balance depends on the use case, but the principle remains:

**uncertainty should be visible.**

This is one of the ideas SudoChat was designed to demonstrate.

## Limitations or gaps
SudoChat cannot literally "know" with certainty that every retrieved fact is correct.

Its ability to assess evidence depends on:

* the quality of the knowledge base
* retrieval quality
* system instructions
* model behaviour
* metadata
* evaluation controls

A poor or incorrect source can still produce a poor answer.

Retrieval scores are also not the same as factual confidence.

I therefore do not claim that SudoChat has solved hallucination or uncertainty detection.

The current proof of concept demonstrates the design principle, while more mature implementations would require systematic retrieval thresholds, validation and evaluation.

## Useful links
SudoChat project and repository

SudoChat knowledge base

My portfolio:
https://mustafa-siddiqui.com/

GitHub:
https://github.com/sudoqui

Federal Court Generative Artificial Intelligence Practice Note

Responsible AI and hallucination research

## Do not claim
Do not claim SudoChat can determine truth with certainty.

Do not claim a high retrieval similarity score guarantees factual accuracy.

Do not claim RAG eliminates hallucination.

Do not claim SudoChat currently has a mathematically perfect confidence score for every response.

Do not claim every unsupported question will always be detected.

Do not claim a retrieved source is automatically authoritative.

Do not claim SudoChat's evidence threshold architecture is already equivalent to a production Federal Court implementation.

Do not claim refusing to answer automatically makes an AI system safe.

The accurate representation is that SudoChat is designed around an **evidence first principle**: retrieve relevant knowledge, determine whether that knowledge actually supports the requested claim, respect explicit limitations and negative boundaries, and when the evidence is not sufficient, prefer an honest limitation or refusal over an unsupported answer.

← Previous part · Knowledge Base