Why do groundedness and faithfulness need separate thresholds in production?

Why Separate Thresholds Matter in Evaluation and Governance

Groundedness and faithfulness answer different operational questions, so combining them into one score can mask where the system is actually failing. Groundedness is about whether the output is supported by the supplied evidence. Faithfulness is about whether the model stayed accurate to that evidence when composing the answer. When teams collapse those checks, they lose the ability to distinguish retrieval problems from generation problems, which weakens triage, makes regression analysis noisy, and can leave audit evidence too vague to support a defensible control decision.

That distinction matters most in production settings where model quality is not just a product concern but a trust and accountability issue. If a system retrieves the right source material but still paraphrases it badly, the fix is different from a case where the retriever missed the right source entirely. NHI Management Group treats that separation as a governance boundary, not a cosmetic scoring preference. In practice, many security teams discover the difference only after a production incident has already been logged as a single “answer quality” failure.

For teams assessing identity-linked or agentic workflows, the same logic applies to non-human access paths and delegated actions: evidence support and evidence interpretation are separate failure surfaces. Where identity-bearing automation is involved, the OWASP Non-Human Identity Top 10 is often the more relevant external lens than a generic AI-quality discussion.

How the Two Thresholds Work Together in Practice

A production pipeline usually benefits from evaluating groundedness first and faithfulness second, because the two checks diagnose different layers of the system. If groundedness is low, the model may be answering from weak, incomplete, or irrelevant retrieval results. If groundedness is high but faithfulness is low, the evidence may be present but the generation layer may have overstated, merged, or distorted it. Keeping the thresholds separate lets teams route incidents to the right owner instead of sending every failure to the same queue.

A practical pattern is to treat the thresholds as two independent gates rather than two numbers that are averaged together. That means one gate can trigger retrieval review, indexing review, chunking review, or source-selection review, while the other can trigger prompt, decoding, evaluation, or model-behaviour review. This is especially useful when the system has multiple knowledge sources or when a retrieval layer is periodically refreshed. A single blended score can improve superficially while hiding a worsening imbalance between source selection and answer fidelity.

• Use groundedness to test whether the evidence set is sufficient for the answer you expect.
• Use faithfulness to test whether the final wording stays inside that evidence set.
• Track the two scores separately across releases so regressions do not get averaged away.
• Compare failures by source class, topic, and model version to see whether the issue is upstream or downstream.

Teams often get the best signal when they review low-groundedness cases as retrieval defects and low-faithfulness cases as composition defects, rather than treating both as generic “hallucinations.” The guidance breaks down when the evidence itself is ambiguous or internally inconsistent, because then neither threshold can reliably distinguish model error from source uncertainty.

When a Single Score Becomes Misleading

Tighter evaluation usually increases measurement overhead, so organisations have to balance diagnostic clarity against the cost of running and reviewing two controls. That trade-off is worth making when the application carries audit, safety, or trust consequences, but it may be unnecessary for low-stakes summarisation where only a coarse quality signal is needed.

One common edge case is when the retrieved context is accurate but incomplete. The answer may appear grounded and faithful to the available context while still being operationally wrong because the context set was too narrow. Another edge case is when the model uses correct facts but presents them with an implication the source does not support. That can look like a weak answer quality score if the thresholds are merged, but it is actually a distinct integrity failure. Industry consensus is still evolving on how to weight these two dimensions across different tasks, so teams should label their threshold policy explicitly rather than assuming one universal cutoff fits every use case.

The same caution applies when the system serves regulated or high-trust workflows, because compliance reviewers usually need to know whether the issue was missing evidence, misread evidence, or both. A single combined score can tell you that the answer was bad, but not why. In operational terms, that is the difference between a useful control and a dashboard number.

Practitioner Guidance

What to prioritise: Separate the thresholds wherever the system has distinct retrieval and generation stages. That gives teams a clean failure map and prevents remediation from being aimed at the wrong layer.

What to verify: Check that low scores are classifiable after the fact. If reviewers cannot tell whether a failure came from evidence absence or evidence distortion, the thresholds are not doing enough work for production use.

Decision rule: If the use case needs defensible review, escalation, or auditability, keep the gates separate; if the use case is low impact and only needs rough quality trending, a blended score may be sufficient.

Common mistake: Treating a single quality number as if it can explain both retrieval health and model behaviour. That shortcut tends to hide the first real degradation until users or reviewers notice it indirectly.

Practitioner takeaway: The real value of separate thresholds is not stricter scoring, but clearer diagnosis, because production teams can only fix what they can reliably distinguish.

Related resources from NHI Mgmt Group

• Why do RAG systems need separate groundedness and faithfulness thresholds?
• How should teams separate readiness from liveness in production services?
• Who is accountable when a production model drifts below approved thresholds?
• How do organisations separate AI experimentation from governed production use?