AI agent evaluation at production scale needs stronger governance

At a glance

What this is: This is an analysis of how Uber’s production agent experience shows that evals fail when they stop at pre-launch scoring and do not feed back into ongoing governance.

Why it matters: It matters because AI agent programmes need operational controls for traces, ownership, and dataset refresh, not just model-quality checks, and those controls increasingly affect identity, access, and delegated action risk.

Context

AI agent evaluation is the control problem of turning real behaviour into durable governance, not just a one-time quality score. In Uber’s case, the gap appeared when offline tests missed an agent that misread ambient speech and rerouted a ride, showing that production behaviour can expose failure modes the initial test set never anticipated.

For IAM, NHI, and agentic AI programmes, the same pattern matters because autonomous or semi-autonomous systems need continuous evidence about what they actually did, who owns the response, and how findings become new tests. Static review processes often assume the important behaviour is already known before deployment, which is exactly what production disproves.

Key questions

Q: How should teams handle when an agent evaluation misses a real production failure?

A: They should treat the miss as a governance signal, not just a model bug. The failure should become a new trace-backed test case, the dataset should be updated, and the ownership model should force a named team to review whether the current evaluator still reflects live behaviour. That prevents a static test suite from becoming a false source of confidence.

Q: Why do production traces matter so much when teams are improving AI agents over time?

A: Production traces show what the agent actually did, including tool calls, retrieved context, intermediate reasoning, and final outputs. That makes them a better signal than static test cases alone. Teams can turn trace failures into new evaluation examples, update datasets with representative traffic, and use the results to improve prompts, models, and tool routing without guessing at the root cause.

Q: What do security and product teams get wrong about agent evaluation suites?

A: They often assume the suite is finished once the dashboard exists. In practice, the real work is maintaining relevant examples, refreshing thresholds, and deciding who responds when a regression appears. Without that operating model, the suite becomes a scorecard that certifies launches instead of improving the agent over time.

Q: How do you know if agent evaluation is actually working?

A: It is working when it explains failures well enough to change the system, not just when it outputs a score. Good evaluation should point to the layer that failed, produce a reproducible trace, and feed confirmed failures into regression tests. If the score does not change engineering decisions, it is not yet operational.

Technical breakdown

Why offline agent evaluations miss production behaviour

Offline evaluations tend to reflect the team’s assumptions about user intent, tool use, and conversation shape. That works until the agent meets ambient speech, unexpected interruptions, or multi-step trajectories that were never in the test set. In the Uber example, input and output logs were insufficient because the failure sat in the path between them. The real issue was not whether the agent produced a plausible final answer, but whether its intermediate reasoning and tool choices matched the actual task boundary.

Practical implication: instrument traces from the first deployment so production trajectories, not just prompts and outputs, become the basis for evaluation.

How production traces become the raw material for better evals

A trace is more than a debug artefact. It records the sequence of tool calls, retries, plan changes, and partial outputs that explain why an agent succeeded or failed. Once teams have that record, they can promote real failures into test examples, create evaluators aligned to actual behaviour, and keep datasets from drifting away from the product. This is especially important for multi-turn agents, where generic synthetic conversations often look realistic but miss the interruptions and edge cases that matter operationally.

Practical implication: treat production failures as dataset candidates and refresh eval suites on a defined governance cadence.

Why evaluation quality is measured by decisions, not scores

A high score does not prove that the evaluation system is useful. The better test is whether an evaluation result changes a release decision, prompts a design fix, or reveals a missing capability. If teams optimise toward the score itself, the eval suite can become a certification ritual that rewards test-taking behaviour instead of product learning. That shift is a governance failure because it hides whether the organisation is actually improving control over the agent or merely improving its scorecard.

Practical implication: review whether an eval result changed a build, ship, or redesign decision before treating the suite as effective.

NHI Mgmt Group analysis

Production evaluation is becoming an identity and governance problem, not just a model-quality problem. Once an agent can act across tools, decide when to continue, and shape user outcomes, the evaluation system is effectively governing delegated behaviour. That makes ownership, traceability, and response rights part of the control surface, especially where agent decisions affect customer-facing actions or internal workflows. Practitioners should treat eval design as part of the agent identity model, not a post-build QA step.

Continuous feedback loops are now the named concept that matters: evaluation drift. When datasets, prompts, and evaluators stop reflecting live behaviour, the control system begins measuring an earlier version of the product. Uber’s example shows that production traces, reviewed failures, and updated test cases must move together or confidence becomes stale. The practitioner conclusion is simple: if the dataset is not evolving, the evaluation programme is already behind.

Cross-functional ownership is the difference between visible failures and governed improvement. Engineering can verify tool execution, but conversational design, operations, and product teams often understand whether the behaviour was acceptable in context. That is why agent governance has to widen beyond the build team. The practical conclusion is that evaluation criteria, exception handling, and response ownership should be shared across the people closest to the real workflow.

Agent evals should be judged by their influence on product decisions and risk decisions. A mature programme does not merely detect anomalies; it changes what gets built next, what gets blocked, and what gets retested. That is the standard for governance in agentic systems because static thresholds cannot keep pace with changing behaviour. Practitioners should measure whether eval outputs alter development and release decisions, not only whether they generate alerts.

From our research library:

• Gartner predicts that more than 40% of agentic AI projects will be cancelled by the end of 2027, citing rising costs, unclear value and insufficient risk controls.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Evaluation drift is the core governance risk in production AI agents. Once the test set lags live behaviour, a high score can simply mean the system is getting better at the benchmark rather than safer in the real world. Teams need a refresh cycle for traces, examples, and thresholds so the control keeps pace with the product.

AI agent governance now depends on who can see, review, and act on traces. That is where the identity and authorisation angle becomes real: the right people need access to evidence, while the agent’s delegated actions must remain bounded by defined ownership and approval paths. Without that, evaluation becomes observation without control.

Half of companies using generative AI will deploy agentic AI by 2027, according to Deloitte's 2025 Technology Predictions. That scale makes the move from launch-time testing to continuous evaluation unavoidable, because more agents will be making decisions in live workflows with real operational consequences.

For practitioners

• Start tracing at first deployment Make full trajectory tracing a default part of every agent deployment so tool calls, retries, and intermediate outputs are captured before incidents force reconstruction from partial logs.
• Promote production failures into test cases Review live failures, turn them into evaluation examples, and refresh datasets on a schedule so the offline suite reflects current user behaviour rather than launch-time assumptions.
• Assign shared ownership for evaluation upkeep Give product, design, operations, and engineering defined responsibility for evaluator review, threshold changes, and exception handling so no single team owns the whole feedback loop.
• Measure whether evals change decisions Track whether an evaluation result altered a release, design choice, model change, or blocked deployment, because decision impact is a stronger signal than a passing score.
• Use behavioural metrics as early warning signals Watch for spikes in session length, repeated actions, or contradiction between tool output and agent response, then tie those signals back to the underlying traces.

Key takeaways

• AI agent evaluation fails when teams rely on pre-launch scores and do not convert production behaviour into updated tests and ownership workflows.
• The production lesson is that traces, not dashboards alone, reveal the failures that matter for multi-turn agents and tool-using systems.
• A mature eval programme changes product decisions, which is the clearest sign that governance is actually working.

Key terms

• Agent Evaluation Lifecycle: The staged approach to testing and monitoring AI agents as they move from prototype to production. Each phase uses different evidence, from fast heuristics early on to production sampling and post-incident analysis later, because the value and cost of evaluation change as exposure increases.
• Evaluator Drift: The condition where an evaluation system no longer matches real operational quality or user satisfaction. It can happen when a judge model, heuristic, or rubric keeps scoring outputs well even as users begin rejecting them, creating false confidence in the programme.
• Production Tracing: Production tracing records an agent’s full execution path, including tool calls, retries, and intermediate states. It gives teams evidence about how the system reached a result, which is essential for debugging, evaluation, and governance when final outputs alone do not explain behaviour.
• Evaluation Feedback Loop: A recurring process for improving LLM evals by comparing automated scores, human review, and real user feedback. It helps teams refine test cases, scoring rules, and thresholds over time instead of treating evaluation as a one-time setup. The loop is central to making evals more accurate and operationally useful.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, IAM, and secrets management. It helps practitioners build the governance discipline needed when autonomous systems start acting inside real production workflows.