AI agent evaluation at production scale needs stronger governance

At a glance

What this is: This is an analysis of how production-scale AI agent evaluation breaks down when teams rely on tooling instead of governance, with the key finding that real-world traces must feed continuous testing.

Why it matters: It matters because IAM, NHI, and AI governance teams need controls that account for runtime behavior, delegated actions, and unanticipated agent decisions, not just pre-launch checks.

👉 Read Arize's analysis of how Uber evaluates AI agents at production scale

Context

AI agent evaluation is often treated as a testing problem, but the deeper issue is governance: teams need a way to turn live behavior into controls, not just scores. At production scale, the gap appears when logs, dashboards, and offline test sets fail to capture how an agent actually behaves once users, tools, and interruptions change the interaction.

That matters for identity and access programmes because AI agents increasingly sit in the path of delegated decisions, tool calls, and data access. When agent behaviour changes in production, the question is not only whether the model worked, but what identity, privilege, or workflow boundary failed to constrain it. Uber’s example is typical of mature teams discovering this too late, not an isolated edge case.

Key questions

Q: What breaks when AI agent evaluation is only a launch gate?

A: Evaluation becomes a certification exercise instead of a learning system. Teams optimise for passing scores, but production behaviour can still drift, new user patterns go unseen, and important failures never enter the test set. The result is false confidence: the dashboard looks healthy while the agent keeps behaving in ways the original evaluation never anticipated.

Q: Why do AI agents complicate access governance more than ordinary automation?

A: AI agents complicate access governance because they can branch at runtime, wait on external services, and continue later with the same operational context. That means privilege is not just granted at launch, it persists across a live session that must be observable, resumable, and attributable.

Q: How do security teams know whether an agent evaluation process is working?

A: Look for evidence that evaluation results change decisions. If the team updates prompts, policies, datasets, or release criteria after a finding, the process is working. If scores only generate reports and never alter build choices, the evaluation function is decorative rather than operational.

Q: Who should own AI agent evaluation in production environments?

A: Engineering should own instrumentation and system behaviour, but product, operations, and domain experts should own the judgment criteria. The best evaluation programmes distribute responsibility because acceptable behaviour depends on customer impact, workflow context, and business risk, not just technical correctness.

Technical breakdown

Why production traces matter more than final outputs

A final answer only tells you what the agent said, not how it got there. Production traces capture the full trajectory, including tool calls, retries, intermediate plans, and model disagreements, which is essential for diagnosing failures in multi-turn systems. Without that path-level evidence, evaluation stays blind to the decision point where behaviour drifted. In practice, this is the difference between validating an output and understanding the control path that produced it.

Practical implication: instrument agent traces from first deployment so evaluation can inspect the full decision path, not just the final response.

How production failures become reusable evaluation data

Offline datasets age quickly because they reflect assumptions made before the agent meets real users. Production failures expose the cases those assumptions missed, then human review can promote the best examples into the offline set or into synthetic simulations. That creates a feedback loop where the test suite evolves with the product instead of becoming a static certification layer. For multi-turn agents, this is especially important because context, interruptions, and user intent shift from one deployment cycle to the next.

Practical implication: build a process that converts reviewed production failures into the next evaluation dataset and simulation set.

Cross-functional evaluation is a control model, not a workflow detail

Agent quality is not judged by engineering alone. Conversational design, product, operations, and domain subject matter often determine whether behaviour is acceptable, especially when human experience or downstream business impact is involved. If only engineers define success, the evaluation suite can miss interaction failures that matter operationally. That makes evaluation a governance control around acceptable behaviour, not just a technical test harness. The better model gives the closest experts a formal role in approving what good looks like.

Practical implication: assign evaluation ownership across engineering, product, and operations so behavioural failures are judged by the right reviewers.

NHI Mgmt Group analysis

Production AI agent evaluation is becoming an identity governance problem. Once agents can call tools, access data, and trigger downstream actions, evaluation is no longer just about model quality. It becomes a control over whether delegated behaviour stays within intended scope, which is exactly where IAM and NHI governance intersect. Programmes that ignore runtime behaviour will keep discovering failures after the fact, when access or actions have already been taken.

The real failure mode is evaluation that stops at launch approval. Teams often have platforms, dashboards, and judge models, but no operating model that turns production evidence into new controls. That creates evaluation debt: the organisation appears covered because tests exist, yet the test set no longer represents live behaviour. The governance lesson is simple. If production traces do not reshape policy, ownership, and test design, the evaluation process is performative.

Continuous tracing is the named control gap teams keep underestimating. Without full path visibility, a team cannot tell whether an agent failed because of user ambiguity, tool misuse, or a poor policy boundary. That makes it impossible to separate product defects from governance defects. For practitioners, the priority is not more dashboards. It is a trace-backed evaluation model that makes every unexpected action evidence for the next control update.

Multi-turn agent behaviour exposes a boundary problem that conventional QA misses. The most dangerous failures are often not obvious errors, but subtle rerouting, overreach, or context contamination that only appears in production. In identity terms, the issue is whether the agent remained inside its authorised intent. When teams cannot answer that, they do not have an evaluation system. They have an observation layer without enforcement value.

What this signals

Evaluation is becoming the governance layer between agent capability and delegated risk. As agents gain broader tool access, teams need a trace-backed control model that shows not only what the system returned, but what it attempted, retried, and touched. The practical shift is toward evidence-led governance, where runtime traces inform policy updates and control exceptions instead of sitting in an observability silo.

Continuous evaluation will increasingly resemble identity lifecycle management for agents. The useful pattern is not a one-time approval, but a repeatable loop of introduction, monitoring, refinement, and retirement when behaviour no longer fits the intended boundary. Teams that already manage NHI or agentic AI risk should expect evaluation data to become part of access review, exception handling, and change management.

Production agent programmes should expect the biggest failures to surface as boundary drift rather than obvious breakage. The organisations that respond fastest will be the ones that connect evaluation to reviewable evidence, and then use that evidence to adjust tooling, ownership, and release criteria.

For practitioners

• Instrument full agent traces at deployment Capture tool calls, retries, intermediate reasoning artifacts where allowed, and session trajectories from the first release so production can be reconstructed later.
• Promote production failures into the next test set Create a review loop where human annotators turn real failures into offline examples and synthetic scenarios, then retire stale test cases that no longer reflect live behaviour.
• Assign evaluation ownership across functions Require engineering, product, operations, and domain reviewers to sign off on what acceptable agent behaviour means before the evaluation suite is considered authoritative.
• Treat evaluation regressions as release blockers only when they change decisions Measure whether a failed evaluator would alter a roadmap item, model choice, policy boundary, or workflow design. If not, refine the evaluator instead of declaring victory.

Key takeaways

• AI agent evaluation fails when organisations treat it as a scorecard instead of a control loop.
• Production traces, reviewed failures, and cross-functional ownership are the controls that make agent evaluation useful.
• For IAM and NHI teams, the important question is whether agent behaviour stays inside intended delegated scope at runtime.

Key terms

• Production Tracing: Production tracing records an agent’s full execution path, including tool calls, retries, and intermediate states. It gives teams evidence about how the system reached a result, which is essential for debugging, evaluation, and governance when final outputs alone do not explain behaviour.
• Evaluation Debt: Evaluation debt is the gap between the tests a team has and the behaviours the product now exhibits in production. It grows when datasets, thresholds, and judges are not updated as the system changes, leaving organisations with scores that no longer reflect real risk.
• Cross-functional Evaluation: Cross-functional evaluation is the practice of involving engineering, product, operations, and subject-matter experts in judging agent behaviour. It recognises that acceptable outcomes depend on business context, user experience, and operational impact, not just technical correctness.
• Boundary Drift: Boundary drift is the gradual or sudden expansion of an agent’s actions beyond its intended scope. It can appear as rerouted tasks, unnecessary tool use, sensitive data exposure, or repeated retries that indicate the system is no longer respecting the original operating boundary.

What's in the full article

Arize's full analysis covers the operational detail this post intentionally leaves for the source:

• The evaluation defaults Uber used to make tracing automatic across managed and code-first agent deployments.
• The production-to-dataset workflow for turning reviewed failures into reusable offline tests and synthetic simulations.
• The interface changes that let product, design, and operations teams participate in evaluator review and maintenance.
• The criteria Uber used to decide whether an evaluation result should change a product decision, release, or model choice.

👉 The full Arize article covers the production tracing model, evaluator defaults, and the decision criteria behind continuous improvement.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and agentic AI identity. It helps security and identity practitioners build controls for delegated access, lifecycle oversight, and runtime accountability.