Use separate tests for summarisation, retrieval, tool use, and agentic interaction. A single benchmark score hides the fact that a model can perform well in one setting while failing in another. Governance teams need task-specific thresholds, because the risk profile changes when outputs are merely descriptive versus when they can trigger actions.
Why hallucination risk needs workflow-specific testing
Hallucination is not a single property of a model, it is a property of a model used in a specific workflow. Summarisation, retrieval, tool use, and agentic interaction each create different failure modes, so a one-number benchmark can look reassuring while hiding operationally dangerous behaviour. Teams should evaluate the workflow, not just the model, because the same output error has very different consequences depending on how it will be used.
The practical question is whether the system needs a faithful description, a grounded answer, a tool call, or an action sequence. Those are different control problems, and they should not share the same acceptance threshold. A workflow that only drafts text can tolerate a different error profile from one that can search data, trigger side effects, or continue acting without direct human review.
How to separate tests by workflow
Start by defining the task class and the failure that matters most. For summarisation, test whether the system preserves source meaning and omits unsupported additions. For retrieval, test whether it retrieves relevant evidence and resists inventing facts when the corpus is thin or ambiguous. For tool use, test whether the model chooses the correct tool, passes the right arguments, and refuses to fabricate tool outputs.
For agentic interaction, include multi-step scenarios because hallucination can appear as a mistaken plan, a bad assumption carried forward across turns, or a fabricated justification for an action. This is where Agentic AI Risk Assessment Template becomes useful: it treats autonomy, tools, credentials, data, oversight, and logging as separate risk inputs rather than collapsing them into one generic score. That same logic should shape your evaluation design.
A useful pattern is to create a test matrix with one axis for workflow type and another for consequence level. A descriptive workflow should be judged on truthfulness and source fidelity. A workflow that can retrieve or act should also be judged on grounding, escalation behaviour, and whether the system knows when to defer. That is the difference between a harmless factual slip and an unsafe operational decision.
What good thresholds look like in practice
Thresholds should reflect business impact, not just model quality. A team may accept a small rate of unsupported detail in low-stakes summarisation, but the same rate may be unacceptable when the output informs customer advice, compliance decisions, or downstream automation. Task-specific thresholds let governance teams distinguish nuisance error from material risk.
For higher-impact workflows, score more than one dimension: factual accuracy, citation or retrieval grounding, refusal quality, and action correctness. A model that is “mostly right” in text generation may still be unsuitable if it overstates certainty, cites irrelevant evidence, or takes an unsafe next step. The threshold should therefore be set on the complete workflow outcome, not on a general model score.
AI Risk Assessment Guide is a useful companion when teams need to turn those thresholds into a repeatable governance process, because it ties use-case scoping, scoring, treatment, and reassessment together. For hallucination risk, the important judgement is whether the workflow is merely informative or whether its outputs can influence decisions, permissions, or actions.
Risk and Threat Considerations
Hallucination becomes more serious as the workflow moves from description to decision support and then to action. In a retrieval or agentic setting, a confident but false output can create hidden operational error, bad escalation, or unsafe tool use, especially when users trust the system’s tone more than its evidence.
Failure mechanism: The system produces plausible but unsupported content, and the surrounding workflow treats it as if it were grounded. In agentic flows, that error can compound across steps, because one fabricated assumption can shape later retrieval, tool calls, or user-facing conclusions.
Impact: The result can be incorrect decisions, wasted investigation, customer harm, policy violations, or unintended actions. The more autonomy the workflow has, the more important it is to measure hallucination as an operational risk rather than a purely model-level quality issue.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST AI RMF and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | GV.OV-01 — Governance Oversight | Hallucination testing needs workflow-level oversight and risk thresholds. |
| MAP 1.1 — Context and Purpose | The answer depends on scoping each AI workflow before measuring risk. | |
| MEASURE 1.1 — Validity and Reliability | Separate tests need measurements that differ by task and output type. | |
| Recommendation — Set workflow-specific hallucination thresholds and review them through AI risk oversight. Define the workflow, intended use, and failure consequences before evaluating hallucination risk. Measure hallucination separately for summarisation, retrieval, tool use, and agentic actions. | ||
| NIST SP 800-53 Rev 5 | RA-3 — Risk Assessment | Task-specific thresholds are a risk assessment problem tied to system use and impact. |
| AU-12 — Audit Record Generation | Agentic workflows need evidence of outputs, tool calls, and action decisions. | |
| Recommendation — Assess hallucination risk per workflow and set acceptance thresholds by impact. Log model outputs and tool actions so hallucination failures can be investigated later. | ||
Practitioner Guidance
What to prioritise: Separate the evaluation by workflow before you compare models. A single blended benchmark usually obscures the most important failure mode, which is how the model behaves under the exact conditions where the business will use it.
What to verify: Confirm that each test set matches the real workflow, including the presence or absence of sources, tools, multi-turn context, and action authority. If those conditions are not represented, the score is not a reliable decision input.
Decision rule: If the workflow can only inform a human, optimise for source fidelity and clarity. If it can retrieve, call tools, or act, require stricter thresholds for grounding, refusal, and step-by-step correctness before go-live.
Practitioner takeaway: Hallucination risk is not one risk, it is a set of workflow-specific risks, and governance should judge each workflow on the kind of mistake that would actually matter in production.
Related resources from NHI Mgmt Group
- Guide: AI Hallucination Business Risk Guide
- How should security teams evaluate agentic AI workflows that use multiple tools and maintain state across turns?
- How should security teams implement AI risk management across training, inference, and AI agent workflows?
- Why does AI governance create risk when policy and enforcement are split across different teams?
- How should security teams distinguish legitimate data workflows from high-risk exfiltration activity across SaaS, AI apps, and endpoints?