LLM evaluation gaps show why production monitoring still fails

At a glance

What this is: This is a guide to LLM evaluation and production monitoring, arguing that passed offline evals do not guarantee safe or accurate real-world behaviour.

Why it matters: IAM and AI governance teams need this distinction because LLMs, agents, and RAG systems can change behaviour after launch, so control points must extend into runtime monitoring and policy enforcement.

Context

LLM evaluation is a governance problem as much as a model-quality problem. A system can score well on a curated test set and still fail once real users, new prompts, and changing retrieval data reshape its behaviour. For production AI, the question is not whether a model can pass evals, but whether the organisation can keep quality, safety, and operational performance under control after deployment.

That matters for AI security and identity governance because runtime behaviour now depends on tools, data sources, and policy enforcement, not just the model weights. When LLMs are embedded in workflows, the monitoring layer becomes part of the control plane. The article's core message is typical of mature AI programmes: testing is necessary, but insufficient without ongoing observability.

Key questions

Q: How should teams respond when an LLM passes offline evals but fails in production?

A: Treat the production failure as a monitoring and control problem, not a benchmark problem. The right response is to separate quality, safety, and operational metrics, replay real traces, and add runtime checks around retrieval, tools, and policy enforcement. Offline success proves only that the model matched a test set, not that it will remain reliable after deployment.

Q: Why do LLMs that look safe in testing still break after launch?

A: Because testing uses a bounded dataset, while production introduces new prompts, changing user behaviour, and shifting retrieval content. That creates drift in both output quality and safety. If the monitoring layer does not watch runtime traces, the system can keep failing long after the initial evals passed.

Q: What are the signs that RAG evaluation is missing the real problem?

A: The common signs are answers that are well written but unsupported, responses that cite the wrong source context, and retrieval rankings that bury the best evidence too far down the list. If faithfulness, answer relevance, and contextual precision are not all measured, the system can appear trustworthy while still producing grounded-looking errors.

Q: How do organisations keep agent workflows under control in production?

A: By evaluating task completion, tool correctness, and traceability instead of judging only the generated text. Agent controls should show which tool was called, whether parameters were valid, and whether the end-to-end task actually succeeded. Without those signals, a fluent response can hide a broken workflow.

Technical breakdown

Why offline LLM evals miss production failure modes

Offline evaluation measures model behaviour against a fixed test set, but production systems operate in a changing environment. Prompt patterns shift, retrieval corpora drift, users ask edge-case questions, and downstream tools can fail in ways a static benchmark never exercises. That is why a model can look strong on BLEU, ROUGE, or judge-based scoring and still hallucinate, misroute tool calls, or answer the wrong question in production. The real limitation is not the metric itself but the assumption that one test pass represents lasting safety and quality.

Practical implication: treat offline evals as a release gate, not as evidence of ongoing control.

How RAG metrics separate grounding from relevance

Retrieval-Augmented Generation adds a retrieval step before generation, so it needs two different kinds of checks. Faithfulness asks whether the answer is supported by the retrieved context, while answer relevance asks whether the answer actually addresses the user’s question. A response can be faithful but irrelevant, or relevant but unsupported. Contextual precision adds another layer by checking whether the best source material appears early enough in the retrieved ranking to influence generation. Without all three, teams can miss a system that sounds accurate while using the wrong evidence.

Practical implication: measure retrieval quality and answer quality separately, then tune them as one control surface.

Why agent workflow metrics matter more than text metrics

Agentic systems are judged by task completion, tool selection, and parameter correctness, not just by the text they produce. A workflow can generate fluent prose while calling the wrong API, passing malformed arguments, or misreading tool output. That is why span-level traces matter: they show where the chain broke, which tool was selected, and whether the agent stayed aligned with the intended objective. In practice, the evaluation unit is the workflow, not the sentence.

Practical implication: instrument tool calls and end-to-end task success, not only model responses.

Threat narrative

Attacker objective: The objective is to exploit the gap between benchmark success and runtime behaviour so the system produces misleading, unsafe, or untrustworthy outputs in production.

1. Entry occurs when production prompts, retrieval content, or tool inputs differ from the curated evaluation set and expose failure modes that were never exercised offline.
2. Credential or control abuse is not the main issue here, but the operational equivalent is the model or agent overstepping its intended behaviour because monitoring does not catch drift fast enough.
3. Impact appears as hallucinated answers, misgrounded RAG responses, unsafe output, or failed task completion after deployment, despite a successful pre-launch eval.
4. The attacker objective is not a single adversary here, but the systemic outcome is the same: user-facing AI behaviour escapes the assumptions that governance and testing relied on.

Breaches seen in the wild

• McKinsey AI platform breach: McKinsey AI platform hack exposed 46M chats and sensitive data.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Continuous monitoring is now the real AI governance boundary. Offline evaluation answers whether a model looked safe in a controlled test environment, but production AI has to survive drift, new prompts, changing retrieval sources, and tool failures. The article is right to separate quality, safety, and operational metrics because no single score captures runtime risk. The practitioner conclusion is clear: treat monitoring as the living control, not the final reporting layer.

LLM observability is becoming a control-plane problem, not a model-science problem. Once RAG systems and agents enter production, the important question is whether the organisation can prove what the system saw, what it retrieved, what it called, and why it answered the way it did. That is an identity and governance issue as much as a model issue because the system's runtime behaviour depends on trusted tools, data paths, and delegated actions. Practitioners should demand traceability across the full workflow, not just model-level metrics.

Faithfulness gaps create a specific governance failure mode: grounded-looking falsehoods. A response can appear compliant with source material structure while still introducing unsupported claims, which makes RAG especially dangerous when users assume citations equal truth. That failure mode deserves its own named concept, the grounded answer trust gap. Security and AI teams should therefore evaluate retrieval, ranking, and generation together, because the control failure is in the chain, not only in the model output.

Agent metrics need to move from text scoring to task assurance. When an agent can choose tools, sequence actions, and interpret results, sentence-level quality says little about whether the workflow actually achieved the intended outcome. This is where governance frameworks such as the NIST AI RMF and NIST AI 600-1 matter in practice, because they push organisations toward measurable monitoring and managed risk. The practitioner conclusion is to certify workflows, not just model outputs.

The operational economics of evaluation will shape AI governance design. If every trace requires an external scoring call, monitoring becomes a budget problem as well as a security one. That pushes enterprises toward in-environment scoring and policy enforcement, which in turn changes how they think about auditability, latency, and control ownership. The field is moving toward continuous assurance that can run at production volume, and teams that cannot instrument this will rely on blind spots they no longer control.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Grounded answer trust gap: production AI fails when answers look supported but are not actually traceable to the retrieved context, and that makes runtime observability a governance requirement rather than a quality enhancement. For teams building RAG systems and agents, the control objective shifts from model approval to evidence-backed operation.

The monitoring stack has to move closer to execution if organisations want to avoid blind spots in AI workflows. External scoring alone will not scale cleanly at enterprise volume, which is why in-environment evaluation and policy enforcement are becoming the practical pattern for continuous assurance.

AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026. That scale makes runtime control of AI systems and their surrounding secrets estate a governance issue, not just a model-quality issue.

For practitioners

• Define separate metrics for quality, safety, and operations Use different measures for factual accuracy, harmful output detection, and latency or cost so one score does not hide a failure in another control layer.
• Refresh evaluation datasets on a fixed schedule Rebuild test sets regularly with new production examples, edge cases, and failures so the benchmark reflects current user behaviour and failure modes.
• Instrument RAG pipelines end to end Track retrieval ranking, faithfulness, and answer relevance together so you can see whether the system retrieved the right context and used it correctly.
• Measure agent workflows at the task level Capture tool selection, parameter validity, and task completion in traces so broken steps are visible even when the final text looks acceptable.
• Shift to in-environment scoring where scale matters Run evaluation models inside your own environment for high-volume traces so monitoring can stay continuous without external API costs or data exposure.

Key takeaways

• Production LLM failures often appear only after launch because static evaluation does not model real user behaviour, retrieval drift, or tool failure.
• RAG systems need separate checks for grounding, question relevance, and retrieval ranking, or they can produce polished but unsupported answers.
• Agentic workflows require task-level monitoring, because a fluent response can hide wrong tool calls, malformed parameters, or failed end-to-end completion.

Key terms

• Reference-based metric: A reference-based metric compares model output to a human-written or otherwise known correct answer. It works best when the task has a stable target, such as translation or summarisation, and becomes less useful when many valid answers exist.
• Reference-free metric: A reference-free metric scores output without a gold standard answer. It is useful for open-ended tasks like chatbots or creative generation, but it depends on a rubric, judge model, or other scoring mechanism to keep the result consistent and defensible.
• Faithfulness: The degree to which an answer is supported by the retrieved context. In practice, it measures whether the model stayed grounded in supplied evidence rather than inventing details, which makes it a critical signal for high-trust AI workflows and audit-sensitive applications.
• Contextual Precision: Contextual precision is the degree to which an analytic system can explain not just that something failed, but where, when, and under what conditions it failed. It is essential when decision-makers need confidence to act on a narrow population instead of treating every anomaly the same.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and secrets management. It is a practical fit for practitioners who need to connect runtime AI control with identity and access governance.