AI agent testing needs trajectory scoring, not exact-match checks

At a glance

What this is: This is a practical guide to testing AI agents, showing that reliable evaluation must score tool use, multi-step trajectories, and final state rather than rely on exact output matches.

Why it matters: It matters to IAM and NHI practitioners because agentic systems increasingly make identity-relevant decisions through tools and permissions, so testing must catch unsafe delegation, broken workflows, and state-changing errors before production.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, sharing sensitive data, and revealing access credentials.
• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes, and as quickly as 9 minutes in some cases.

👉 Read Braintrust's practical guide to testing AI agents

Context

AI agent testing is hard because the same request can produce several valid execution paths, and a system can still fail even when the final response looks acceptable. Once an agent can choose tools, pass arguments, and change state, quality assurance has to examine the whole trajectory, not just the answer text. That is why AI agent testing now overlaps directly with identity governance, because tool permissions and credential boundaries shape what the agent can do.

Most teams still inherit a software-testing mindset that assumes one input should produce one expected output. That model breaks for agents, where correctness depends on tool selection, sequence, side effects, and recovery from errors. The article's starting point is typical for any team building agentic workflows, especially where the agent can touch customer data, payments, tickets, or other systems governed by access controls.

Key questions

Q: How should teams test AI agents that can use tools and change state?

A: Test the decision path, not just the final answer. Score tool selection, argument quality, execution order, and final state across repeated runs, because agents can reach the same outcome through different valid trajectories. Use controlled dependencies for anything state-changing, then promote real production failures into the regression suite so the test set keeps pace with live behaviour.

Q: Why do exact-match tests fail for AI agents?

A: They assume one request should produce one fixed output, but agents can legitimately take different paths or phrase the result differently. That means a run can be correct even when the text varies, or incorrect even when the text sounds plausible. Evaluation should therefore use criteria and thresholds that reflect the action the system must actually perform.

Q: What breaks when AI agent tests ignore tool order and intermediate steps?

A: A run may appear successful while the agent skips a required check, repeats a call, or sends the wrong argument to a downstream tool. Those failures often only appear in the trajectory, not the closing message. The result is a false pass that can hide bad state changes until production exposes them.

Q: Who is accountable when an AI agent makes the wrong change?

A: Accountability sits with the governance chain that approved the access model, not with the agent alone. Teams need a trace from requester to policy decision to identity issuance to action results. If that chain is missing, incident review becomes guesswork and access governance cannot be defended to auditors.

Technical breakdown

Why exact-match testing fails for AI agents

Exact-match assertions work for deterministic code, but agents introduce non-determinism, multiple valid paths, and state-changing side effects. The right tool call, the right order, and the right final state can all matter at once, while wording differences in the final response may be irrelevant. That is why agent evals score criteria instead of comparing a single expected string. A useful scorer evaluates the choice, the argument payload, and the outcome across repeated trials so teams can see both correctness and consistency.

Practical implication: score the behaviour that matters, not just the final text, and require repeat runs before approving an agent change.

Trajectory evaluation and trace-based debugging

A trajectory captures the full chain from request to final state, including intermediate tool calls and loop iterations. This matters because a run can look successful at the end while hiding earlier failures such as wrong IDs, repeated calls, or skipped dependencies. Trace instrumentation lets each span record inputs, outputs, and errors so the failing step can be tied back to the prompt, tool schema, or model instruction that caused it. Trajectory testing is therefore as much about debugging as it is about pass or fail status.

Practical implication: instrument every tool hop so failed runs expose the exact step that broke, not just a generic failure result.

Regression suites and production scoring

Agent quality improves when pre-release tests and production monitoring feed each other. Regression suites hold a stable dataset that can block merges when scores fall below threshold, while production scoring surfaces unfamiliar language, tool states, and new failure modes that the suite did not cover. The strongest pattern is to turn reviewed production failures into new evaluation cases. That keeps the suite aligned to real behaviour instead of letting it drift toward synthetic examples that are too clean to be useful.

Practical implication: promote real failures into the regression suite and use them as release gates for future changes.

Threat narrative

Attacker objective: The objective is to force an agent to take an incorrect action while the final response still appears plausible to the human reviewer or automated test.

1. Entry begins when a user request reaches an agent that can select tools and act on live systems, creating an opportunity for unsafe or mis-scoped execution.
2. Escalation occurs when the agent chooses the wrong tool, passes the wrong arguments, or repeats a dependency in a way that changes state incorrectly.
3. Impact appears when the workflow refunds the wrong order, contacts the wrong customer, or otherwise makes irreversible changes that a final-output check would not catch.

NHI Mgmt Group analysis

AI agent testing has become an identity control problem as much as a software QA problem. Once an agent can choose tools, pass tokens, and trigger state changes, the question is no longer only whether it answered correctly. The real issue is whether it stayed inside the permissions, scopes, and execution boundaries the organisation intended. That makes tool-call observability and privilege scoping part of the control plane, not just the test harness. Practitioners should treat agent evaluation as a governance function, not a one-off engineering check.

Trajectory scoring is the right named concept for agent quality because the path, not just the result, determines risk. A model can hit the right end state while making the wrong intermediate decisions, and that is where most operational and security failures live. This aligns closely with OWASP Agentic AI guidance and with the logic behind trace-based controls in NIST AI Risk Management Framework. Teams should require evidence of correct tool order, argument integrity, and outcome consistency before release.

Production traces should become evaluation assets, especially where agents touch customer, payment, or identity workflows. The article's core lesson is that real failures are more valuable than abstract test cases because they reveal how the agent behaves under live context, not synthetic perfection. That is a strong fit for identity governance, where the boundary between allowed and disallowed action must remain auditable across repeated runs. Practitioners should build feedback loops that convert observed failures into policy-enforced tests.

Exact-match testing gives a false sense of certainty when agents can legitimately take different routes. Security and QA teams need to measure decision quality, not output sameness, because brittle assertions can miss both overreach and underperformance. In governance terms, this means the approval standard must shift from 'did the model say the right thing' to 'did the system use the right authority, in the right sequence, with the right result.' Practitioners should align release gates to that broader control objective.

Agent evaluation will increasingly sit alongside identity lifecycle controls for workloads and assistants. As more workflows involve human approval, delegated tokens, and tool-mediated access, the quality of the agent's decisions affects the security of the identity boundary itself. That makes NHI governance and agent testing converge around the same question: what was the agent allowed to do, and how do we prove it only did that? Practitioners should plan for shared ownership across IAM, engineering, and QA.

What this signals

AI agent testing is converging with identity governance because the same workflow now includes tool permissions, delegated authority, and state-changing actions. As deployment accelerates, the practical control question is no longer whether the model can answer the task, but whether the agent can prove it stayed inside the authority it was granted. Teams that cannot trace decisions will struggle to defend either release quality or access accountability.

Trajectory confidence gap: when the evaluation only checks the final response, the organisation can miss the failed step that actually caused the unsafe action. That gap matters most in workflows that touch credentials, customer records, or payment systems. Practitioners should expect evaluation pipelines to absorb more trace data, more thresholding, and more identity-aware policy checks as agent estates mature.

For practitioners

• Instrument tool calls in every agent span Record tool name, arguments, return values, and errors for each hop so evaluation can isolate the exact failure point instead of only flagging the end result.
• Score trajectories with pass thresholds Use scorer outputs and repeated trials to measure whether the agent reaches the right state consistently across multiple valid execution paths.
• Version regression cases from real failures Turn reviewed production traces, support incidents, and near misses into versioned evaluation cases so the suite evolves with actual agent behaviour.
• Stub state-changing dependencies before release Replace live payment, email, database, or ticketing services with controlled substitutes so tests cannot trigger irreversible actions outside the test environment.
• Set merge gates on behaviour, not phrasing Block changes when task completion, safety, or step order drops below threshold even if the final natural-language response still looks acceptable.

Key takeaways

• AI agents need trajectory-aware testing because multiple valid paths can mask serious intermediate failures.
• Production traces are not just observability data, they are the raw material for stronger regression coverage.
• As agents touch tools and state, evaluation becomes part of identity governance and release control.

Key terms

• Trajectory Evaluation: Trajectory evaluation checks the sequence of decisions an agent makes from start to finish rather than only the final answer. It is used to confirm task decomposition, tool choice, state retention, and termination logic across multi-step workflows.
• Scorer: A scorer is the rule or model that assigns a quality value to an agent run against defined criteria. It can judge deterministic checks such as schema validity or more subjective criteria such as task completion, allowing teams to threshold quality before release.
• Trace Instrumentation: Trace instrumentation records an agent's internal steps, tool inputs, tool outputs, and errors as separate spans. It gives reviewers enough evidence to reconstruct where a run failed and to convert that failure into a reusable evaluation case.
• Regression Suite: A regression suite is a stable set of test cases used to detect whether a change has degraded behaviour that previously passed. For AI agents, it should include production-derived cases, edge conditions, and known failure patterns so score drops are meaningful.

What's in the full article

Braintrust's full guide covers the operational detail this post intentionally leaves for the source:

• Step-by-step examples for scoring tool selection, argument construction, and output quality in agent workflows
• Concrete patterns for building trace-enabled evals that expose the exact tool call or schema failure
• Implementation guidance for repeated trials, threshold selection, and CI regression gating
• Production monitoring patterns that turn low-scoring traces into new evaluation cases

👉 Braintrust's full guide covers trajectory scoring, trace debugging, and CI regression patterns in more operational detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It is designed for practitioners who need to connect identity controls to agentic and workload-driven access decisions.