How do teams choose the right reference source for hallucination testing?

How to match the reference source to the hallucination test

The right source is the one that defines the answer boundary for the task you are evaluating. If the system is retrieving evidence, the retrieved context is the test source; if it is summarising a fixed document set, the source documents are the test source; if it answers open-ended prompts, a gold dataset is the better benchmark. The source should make the evaluation fair, repeatable, and aligned to how the application is actually supposed to answer.

That choice matters because hallucination is not a single phenomenon. A model can be wrong by inventing facts, by failing to stay within supplied evidence, or by producing an answer that is acceptable in one setting but unusable in another. A source that is too broad will hide errors, while one that is too narrow will overstate failure and create noisy evaluations.

Teams usually get the best results by starting from the task contract, not the model. Ask what the system is supposed to know at answer time, what it is supposed to ignore, and whether the goal is factual grounding, document fidelity, or answer consistency. Once that boundary is clear, the reference source should mirror it as closely as possible.

When retrieved context, source documents, or gold data each make sense

For RAG systems, retrieved context is usually the most defensible reference source because the model is expected to answer from whatever evidence was fetched at runtime. The test should therefore measure whether the answer stays inside the retrieved evidence, whether unsupported claims appear, and whether the retrieval layer itself is bringing back the right material. NIST AI Risk Management Framework is useful here because it frames evaluation around trustworthy system behaviour, not just model output in isolation.

For summarisation, the source documents themselves are the truth boundary. The evaluator should compare the summary to the original text, looking for omissions, distortions, unsupported additions, and overconfident paraphrases that change meaning. This is different from RAG, where a good answer may legitimately rely on a subset of retrieved evidence rather than the whole corpus.

For open-domain question answering, a gold dataset is often the right reference because there is usually a known correct answer or a bounded set of acceptable answers. In that case, the key question is whether the model matches the target answer rather than whether it stayed faithful to a particular document. When the task allows multiple valid phrasings, teams should define acceptance criteria in advance so they do not mistake alternative wording for hallucination.

Make the reference source match the evaluation goal, not the model output

The strongest evaluation source is the one that reflects the failure mode you care about. If you want to know whether the system is inventing facts from thin air, use a dataset or answer key that can expose unsupported content. If you want to know whether it follows the provided evidence, use the underlying documents or retrieval output. If you want to know whether the system can handle ambiguity, compare repeated outputs across runs and check whether variation stays within acceptable bounds.

Some teams try to use a single reference source for every test, and that usually produces misleading results. A benchmark built for summarisation will not fairly judge retrieval quality, and a document set will not always tell you whether an open-domain answer is semantically correct. The source should be selected for what the application is doing, then kept stable enough that test results are comparable over time.

When there is no single correct answer, repeated outputs can be a better reference than a fixed gold response because they show whether the system is stable, bounded, and appropriately diverse. In those cases, hallucination testing is less about exact match and more about whether the model drifts into unsupported claims, contradictions, or unsafe overprecision.

Practitioner Guidance

What to verify: Before you lock in a reference source, verify that it matches the answer contract the user will actually experience. If the application can answer only from retrieved evidence, do not judge it against outside knowledge; if it is supposed to summarise a fixed document, do not score it against an abstract gold answer.

Decision rule: If the test needs to detect unsupported claims, choose a source that exposes grounding gaps; if the test needs to detect factual correctness, choose a source with an explicit answer key. If both matter, run separate evaluations rather than forcing one source to do both jobs.

Common mistake: Teams often evaluate the model against the most convenient reference, not the most faithful one. That shortcut inflates confidence, hides real failure modes, and makes regression results hard to trust.

Practitioner takeaway: The right reference source is the one that matches the system’s truth boundary, because hallucination testing is only useful when the reference mirrors the way the application is supposed to know, answer, and fail.

Related resources from NHI Mgmt Group

• How should teams choose the right signature assurance level for regulated workflows?
• How should security teams choose the right signer authentication method?
• How should teams choose a testing deployment model for regulated workloads?
• How should teams govern open source AI projects when many contributors can change the stack?