AI agent evals only work when the transcripts explain the score

At a glance

What this is: Arize's coverage of Anthropic's guidance shows that agent eval scores can look precise while hiding harness defects, lucky runs, and misread capability changes.

Why it matters: IAM, AI security, and governance teams need transcript-level evidence because agent scores alone do not prove reliable behaviour, safe tool use, or durable control over changing environments.

Context

AI agent evaluation is the practice of testing whether an agent completed a task and whether it did so through a reliable sequence of tool calls, instructions, and environmental interactions. The problem is that a single score can flatten very different causes into one number, which makes governance decisions fragile when agents operate across tools, state, and external systems.

This matters for identity and AI governance because agents are not just models generating text. They act through tools, permissions, and runtime context, so teams need evidence about how access was used, not only whether a task passed. Transcript review becomes the control that explains whether a result reflects capability, chance, or a broken test harness.

Key questions

Q: What breaks when AI agent evals rely on scores without transcript review?

A: Teams can mistake a harness bug, lucky execution path, or biased grader for real model improvement. The score may be correct as a number, but it no longer explains the cause. Transcript review is what separates durable capability from accidental success, which is essential when agents act through tools and changing environment state.

Q: Why do AI agent evals need to track tool calls and environment state?

A: Because agents operate across a trajectory, and the same result can emerge from very different behaviour. Tool calls, state changes, retries, and external system responses all affect the outcome. Without those details, teams cannot tell whether the agent improved, the harness changed, or the judge rewarded the wrong path.

Q: How do security teams know whether an agent evaluation process is working?

A: Look for evidence that evaluation results change decisions. If the team updates prompts, policies, datasets, or release criteria after a finding, the process is working. If scores only generate reports and never alter build choices, the evaluation function is decorative rather than operational.

Q: What should teams do when an AI agent gets the right answer by a lucky path?

A: Treat it as a control failure, not a success. The right response is to inspect the transcript, identify the workaround or harness defect that enabled the result, and decide whether the behaviour should be blocked, retrained, or added as a regression case. Otherwise the same lucky path will reappear later.

Technical breakdown

Why agent eval scores can hide the real cause

Agent evals measure outcomes over a trajectory, not just a final answer. That trajectory can include model prompts, tool calls, environment state, grader logic, and retry behaviour. If any one of those layers changes, the score may move for reasons that have nothing to do with the model’s underlying capability. A harness bug, a lucky path through the tools, or a grader that rewards the wrong evidence can all produce a false gain. The practical lesson is that the score is only a signal. The explanation lives in the trace, where the causal chain can be inspected.

Practical implication: Treat every meaningful score change as a trace-review problem before you accept it as a real capability shift.

Regression evals versus capability evals in production

Regression evals protect behaviour the product already depends on, while capability evals test the frontier of what the agent may now be able to do. Regression cases usually come from customer incidents and repaired bugs, so they should saturate at a high pass rate. Capability cases are different: they deliberately sit near the edge of current performance so teams can see whether a model, prompt, or tool change expands the envelope. Mature programmes need both, because a safe release and a new feature are different decisions. One tells you what must not break. The other tells you what the system may be ready to do next.

Practical implication: Maintain separate suites so release safety and product expansion are measured with different expectations and different thresholds.

How transcript review explains score changes

Transcript review connects the aggregate metric to the exact tool calls, state transitions, and grader decisions that produced it. That matters because two runs can share the same outcome while reaching it through very different behaviour. One agent may solve a task cleanly, while another stumbles into success after risky retries or a harness quirk. Without the transcript, those differences collapse into the same pass. With the transcript, teams can see whether the score moved because the agent improved, the environment changed, or the judge evaluated the runs inconsistently. The trace is what turns evaluation from a number into evidence.

Practical implication: Build review workflows that let teams jump from a score change to the specific run segments that explain it.

NHI Mgmt Group analysis

Transcript evidence is the governance boundary for agent evaluation. A score can show that behaviour changed, but it cannot show why without trace-level inspection. When agents operate through tools, permissions, and external state, the governance question is whether the system behaved consistently for the right reason. The practitioner implication is simple: no team should treat an eval result as decision-grade until the transcript supports it.

Agent evals create a control problem, not just a measurement problem. The deeper risk is that organisations can optimise to the benchmark instead of the behaviour they actually want. Once a harness defect or grader bias enters the loop, the system may reward workaround behaviour and label it progress. The practitioner implication is to design evals as controls over trajectories, not as vanity metrics over outcomes.

Capability growth and evaluation integrity must advance together. As agents take on longer workflows, more tools, and more external systems, the surface area for false confidence expands. That makes transcript review, calibrated judging, and environment-aware testing part of the same governance stack. The practitioner implication is to tie every model or harness change to a reproducible evidence trail.

Evaluation data becomes a security artefact when agents hold operational power. The moment an agent can call tools, query systems, or act on behalf of a workflow, its traces become evidence of access use and decision quality. That creates a natural intersection with AI governance and agentic identity: the system needs to show not only what happened, but what was authorised to happen. The practitioner implication is to treat eval telemetry as part of identity and AI control evidence.

Trajectory analysis is the named concept teams should formalise. Trajectory analysis means judging an agent by the sequence of actions, tool calls, and intermediate states that led to success or failure. It matters because a single outcome can hide a brittle or unsafe path. The practitioner implication is to make trajectory analysis the default lens for production-grade agent evaluation.

What this signals

AI agent evaluation is moving from scorekeeping to evidence management. The practical change for teams is that the transcript, not the summary metric, becomes the authoritative record of why a run passed or failed. That shifts governance toward traceability, reproducibility, and reviewable decision points.

Evaluation-driven development will expose weak harnesses faster than weak models. In mature programmes, the biggest gains often come from better tasks, better graders, and better trace inspection, not just from model swaps. Teams that do not govern the benchmark itself will optimise the wrong thing.

Agent governance now overlaps with identity governance. Once an agent can call tools and operate on live systems, the evaluation question includes what it was authorised to do and whether it stayed within that boundary. The control objective is not merely accuracy, but accountable execution.

For practitioners

• Separate outcome scoring from trajectory review Score whether the task was completed, then review how the agent got there, including tool calls, environment state, and grader decisions. This prevents a lucky pass from being mistaken for durable capability.
• Build distinct regression and capability suites Use regression cases to protect known customer-critical behaviour and capability cases to probe the frontier. Refresh each suite when the model, harness, tools, or user distribution changes.
• Calibrate LLM judges against expert labels Compare model judges with a small human reference set, inspect disagreement cases, and version the rubric with the judge prompt and output format.
• Instrument transcript viewers for long trajectories Give reviewers search, filtering, annotation, and side-by-side comparison across tool calls and state transitions so they can identify where a score change came from.
• Treat the harness as part of the evaluated system Include tools, permissions, external services, and mutable state in the test design so a passing score reflects the real operating environment.

Key takeaways

• A high eval score can conceal a flawed harness, a lucky route through the task, or a grader that rewards the wrong behaviour.
• Production-grade agent governance depends on tracing outcomes back to the exact tool calls, environment state, and scoring decisions behind them.
• Teams should separate regression protection from frontier capability testing, then require transcript evidence before treating any gain as real.

Key terms

• Agent Evaluation Lifecycle: The staged approach to testing and monitoring AI agents as they move from prototype to production. Each phase uses different evidence, from fast heuristics early on to production sampling and post-incident analysis later, because the value and cost of evaluation change as exposure increases.
• Trajectory Review: The inspection of an agent’s step-by-step path through a task, including tool calls, intermediate observations, retries, and stopping decisions. It is the practical method for explaining why a score changed and for distinguishing a real capability gain from a lucky run or harness defect.
• LLM-as-a-judge: A control pattern where one language model evaluates another model's prompts, tool calls, or outputs against policy. It is not content moderation alone. In practice, it acts as a runtime decision layer that can allow, block, redact, or escalate based on semantic context and organisational rules.
• Regression Eval: A regression eval is a repeatable test case built from prior failures. It uses captured inputs, expected outputs, and a scorer to verify that a previously observed problem does not reappear after a change to prompts, models, tools, or retrieval logic.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security and identity practitioners build the control discipline needed for systems that act through tools and credentials.