Topic starter
18/08/2026 2:13 pm
TL;DR: AI agents can reach the same valid outcome through different tool paths, so exact-match assertions miss real failures in tool choice, argument construction, execution order, and state changes, according to Braintrust. Reliable evaluation needs trace-based scoring, repeated trials, regression suites, and production feedback loops that turn observed failures into new cases.
Editorial analysis by NHI Mgmt Group, based on content published by Braintrust: “How to test AI agents: A practical guide”.
Key questions
Q: What should teams do when an AI agent can reach the right outcome through different tool paths?
A: They should stop using exact-match checks as the main pass or fail signal and score the agent on tool choice, argument quality, dependency order, and final state.
Q: Why do AI agents need trajectory testing instead of only output testing?
A: Output testing only checks the final response, but agent failures often begin earlier in the run.
Q: How can security teams tell whether an AI agent test suite is good enough?
A: A useful suite covers happy paths, edge cases, adversarial inputs, and known failures, then runs those cases repeatedly under controlled conditions.
Practitioner guidance
- Define scored criteria for each critical agent decision Separate tool selection, argument validation, output format, and response quality into distinct scorers so a run can pass one check and fail another without losing diagnostic value.
- Trace every state-changing tool call Record tool inputs, outputs, and errors in the execution trace so the failed step can be tied back to the prompt, schema, or tool description that caused it.
- Build regression suites from reviewed production failures Promote failed real-world traces into the dataset, then rerun them in CI with blocking thresholds for release-critical behaviours such as task completion and safe tool use.
Bottom line: AI agents can be correct through different trajectories, so tests must measure decisions and state changes, not only the final message.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
18/08/2026 3:05 pm
Trajectory scoring is the right control because agentic failure is often path-dependent. The central testing mistake is to treat an agent like a traditional function with one correct output. Once tool choice, step order, and mutable state enter the flow, the same task can have several acceptable trajectories, so the control must evaluate the path as well as the result. Practitioners should treat scored traces as the primary evidence of correctness.
A few things that frame the scale:
- Gartner predicts that more than 40% of agentic AI projects will be cancelled by the end of 2027, citing rising costs, unclear value and insufficient risk controls.
A question worth separating out:
Q: What is the difference between unit tests and evals for AI agents?
A: Unit tests verify deterministic code contracts, such as whether a function rejects invalid input, while evals measure model-driven behaviour where several outputs or paths may be acceptable. Agent systems need both because a unit test cannot judge decision quality and an eval cannot prove the underlying tool or service is correct. Use each for the type of risk it can actually detect.
👉 Read our full editorial: AI agent testing needs trajectory scoring, not exact-match checks