AI agent evals only work when the transcripts explain the score

At a glance

What this is: This is an analysis of how AI agent evaluations can mislead teams when scores obscure harness defects, lucky execution paths, and weak causal evidence.

Why it matters: It matters because IAM and security teams building agentic AI need evaluation methods that prove behavior, not just performance, before agents are allowed to act with meaningful access.

By the numbers:

• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes and as quickly as 9 minutes in some cases.
• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.

👉 Read Arize's analysis of agent evals, transcript review, and grading reliability

Context

AI agent evaluation is the practice of testing not only whether an agent completes a task, but also whether the path it took was reliable, reproducible, and appropriate for production use. The problem is that a single score can hide a brittle harness, a flawed grader, or a lucky sequence of tool calls that will not hold under real conditions. That matters for agent governance because once an agent can call tools, query data, or operate across systems, the question is no longer just whether it succeeded.

In identity and access terms, the evaluation problem is also a control problem. Agents are increasingly treated as systems with runtime authority, which means their behavior must be assessed with the same discipline used for privileged workflows, secrets, and delegated access. A score that cannot be traced back to evidence leaves teams unable to decide whether the agent is safe to connect to sensitive tools or whether it merely looked safe in a benchmark.

Arize’s published discussion is typical of the current market moment: teams are discovering that agentic AI needs better test design before it can be trusted in production. The underlying issue is not unique to one model or one harness, but to any environment where access, state, and tool execution change the meaning of a passing result.

Key questions

Q: How should security teams inventory AI agents before granting production access?

A: Start by building a register that links each agent to its owner, the identities it uses, the systems it can reach, and the data it can touch. Do not treat model deployment as proof of governance. Discovery must also capture hidden tokens, inherited service accounts, and any workflow that lets an agent act outside direct human review.

Q: Why do AI agent evaluations produce false confidence in production readiness?

A: AI agent evaluations can create false confidence because they compress multi-step behaviour into a single score. That score may hide environment changes, compensating workarounds, or grader mistakes. The result is a benchmark that looks precise but does not explain whether the agent is genuinely reliable enough for sensitive workflows or privileged tasks.

Q: What do teams get wrong about LLM-as-a-judge for agent evals?

A: Teams often treat an LLM-as-a-judge as if it were neutral by default. In practice, it needs calibration against human-labeled examples, version control, and disagreement review. Without that discipline, the judge can reward the wrong behaviour and turn the evaluation layer itself into a source of drift.

Q: How do you know whether an agent benchmark is measuring real capability?

A: You know a benchmark is measuring real capability when the team can explain the score from the transcript upward. If the result changes because of a harness defect, a changed tool, or random variation, the benchmark is not yet a dependable capability measure. The strongest signal is reproducible behaviour across repeated runs and stable conditions.

Technical breakdown

Why agent evals need trajectory-level evidence

Agent evaluations differ from ordinary software tests because the system can make multiple decisions before reaching an answer. A model may interpret the task, choose a tool, revise its plan, and continue with state that carries forward across the whole run. That means a passing outcome can come from genuine capability, a lucky path, or a broken harness that rewards the wrong behavior. Trajectory evidence makes the cause visible by showing which tool calls, inputs, and environmental conditions shaped the result. Without that evidence, the score is informative but not trustworthy.

Practical implication: teams should review tool-call history and environment state before accepting any agent benchmark result.

Regression evals vs capability evals in agent systems

Regression evals protect known-good behavior from slipping after a model, prompt, tool, or harness change. Capability evals test the frontier, asking what the agent can do now that it could not do before. Both matter because production teams need safety and growth at the same time. Regression suites should trend toward saturation, while capability suites should stay challenging enough to reveal room for improvement. If teams mix those purposes, they either miss failures or overfit to stale success criteria.

Practical implication: separate your safety suite from your growth suite so release decisions and roadmap decisions do not use the same metric.

How LLM-as-a-judge becomes reliable

LLM-as-a-judge can help scale evaluation, but only if it is calibrated against human-labeled examples first. The judge must be tested against expert decisions, with disagreements reviewed to expose weak rubrics or missing context. Versioning matters too, because changing the judge model or prompt can change the benchmark itself. In agentic systems, a weak judge is especially dangerous because it can reward the wrong trajectory while appearing objective. The model is not the only thing being evaluated. The grading system is part of the control plane.

Practical implication: version your judge rubric, model, and labels together, then recalibrate whenever the task distribution shifts.

NHI Mgmt Group analysis

Agent evaluation is becoming an access governance issue, not just a model quality issue. Once agents can call tools, query data, or continue a task across stateful workflows, a misleading eval can greenlight runtime access that was never really proven safe. That makes evaluation evidence part of the control stack for agentic AI, alongside IAM, secrets, and permissioning. Practitioners should treat benchmark design as a prerequisite for access decisions, not a downstream QA exercise.

Trajectory blindness creates a false sense of confidence in agentic systems. A score that captures only outcome can conceal the exact failure mode that matters: a flawed harness, a lucky sequence, or a compensating path that will not repeat. In governance terms, this is a provenance problem, because the system cannot explain why the result occurred. The named concept here is trajectory credibility gap: the distance between a passing score and the evidence needed to trust the path that produced it. Teams need to close that gap before they expand agent privileges.

Eval datasets should be built from real production traces, because synthetic cases miss the messy edge conditions that drive incident risk. Public benchmarks provide structure, but the substance has to come from actual trajectories, customer failures, and expert review. That is the same pattern security teams use when they build detections from incident data rather than theory. For agent governance, the practical conclusion is clear: if the agent is going to interact with sensitive systems, the test set must reflect how those systems behave in the real world.

Model-based grading needs the same change control discipline as any other decision engine. An LLM judge can help with scale, but it also becomes a policy surface that can drift when prompts, tasks, or external tools change. That is familiar territory for identity teams, where approval logic and access rules must be versioned, audited, and recalibrated. The practitioner takeaway is to treat judge outputs as governed evidence, not as neutral truth, and to keep human review in the loop for edge cases.

The strongest agent programmes will be the ones that can explain failure, not just report success. Mature governance means a team can move from an aggregate score to the exact tool call, grader decision, or environment change that altered the outcome. That auditability is what makes agentic AI operationally fit for regulated, high-risk, or privileged workflows. Practitioners should aim for evaluation systems that support traceability first and headline metrics second.

What this signals

Trajectory credibility gap: agent programmes will need a new governance layer that ties benchmark scores to replayable evidence, because a passing result without traceability does not justify expanded access. Teams should expect more scrutiny of judge calibration, tool-state capture, and repeatability as agentic AI moves into regulated workflows.

The practical signal is that evaluation stacks now behave like security control planes. For identity-led teams, that means any agent granted access to data, APIs, or operational tools should also have auditable evaluation artifacts, with transcript review and versioned grading aligned to the same rigor used for privileged access decisions.

For practitioners

• Separate safety and frontier evals Build one regression suite for previously approved behaviours and a separate capability suite for new tasks, then gate releases on both. This prevents a single score from masking either broken existing workflows or overconfident expansion into new ones.
• Review trajectories before accepting gains Require transcript inspection for any material benchmark jump so teams can confirm whether the change came from better reasoning, a harness defect, or a lucky tool path. Tie the review to the exact tool calls, state transitions, and grader decisions that produced the score.
• Version the grading stack as a controlled system Track the judge model, prompt, rubric, and label set together, and recalibrate whenever task mix or external tools change. If the grading layer drifts, the evaluation no longer measures the same thing from one release to the next.
• Mine production traces for evaluation cases Use real agent transcripts, incident reports, and unusual tool sequences to seed new test cases. Production traces reveal the failures synthetic benchmarks usually miss, especially where environment state or long-running context changes the outcome.

Key takeaways

• AI agent evals can hide flawed harnesses and lucky execution paths if teams rely on outcome scores alone.
• Reliable agent governance requires trajectory evidence, calibrated graders, and separate regression and capability suites.
• As agents gain runtime access, evaluation quality becomes part of the access control decision, not just a model QA task.

Key terms

• AI Agent Evaluation: A structured assessment of whether an AI agent behaves as intended under controlled conditions. In practice, it checks output quality, policy adherence, and task performance, but it does not by itself prove resilience against adversarial prompts, tool abuse, or runtime identity misuse.
• Trajectory Evaluation: Trajectory evaluation checks the sequence of decisions an agent makes from start to finish rather than only the final answer. It is used to confirm task decomposition, tool choice, state retention, and termination logic across multi-step workflows.
• LLM-as-a-judge: A control pattern where one language model evaluates another model's prompts, tool calls, or outputs against policy. It is not content moderation alone. In practice, it acts as a runtime decision layer that can allow, block, redact, or escalate based on semantic context and organisational rules.
• Capability Eval: An evaluation designed to reveal what an agent can do now that it could not reliably do before. Unlike regression testing, which protects existing behaviour, capability evaluation explores the frontier of performance and helps teams decide whether a new workflow is ready for productisation.

What's in the full article

Arize's full article covers the operational detail this post intentionally leaves for the source:

• Detailed examples of agent transcript review and side-by-side run comparison for debugging score changes
• Practical advice on building eval datasets from production traces, incidents, and expert-labelled cases
• Further guidance on calibrating LLM-as-a-judge and handling disagreement between the model and human reviewers
• Expanded explanation of how capability evals influence product strategy and release decisions

👉 Arize's full piece covers the transcript examples, eval dataset workflow, and judge calibration advice in more depth

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and secrets management. It helps practitioners connect identity controls to the runtime systems now shaping access decisions.