Because evals should measure the behaviour that actually produced the outcome, not an abstract guess about what might matter. Traces show the decision path, so the evaluator can inspect the right span, score the right behaviour, and catch the failure mode that drives the final report. That makes improvement specific instead of speculative.
Why real agent traces matter for eval design
Real traces give you the actual sequence of observations, tool calls, intermediate decisions, retries, and handoffs that led to an outcome. That matters because an eval only becomes useful when it measures the behaviour that produced the result, not a simplified reconstruction of it. If you start from traces, you can define the test around the decision points that were truly in play.
For agentic systems, that usually means evaluating at the level of a task episode rather than at the level of a prompt or final answer alone. A trace shows where the agent chose a tool, ignored a signal, overrode a constraint, or recovered from a bad branch. That is the difference between a benchmark that feels plausible and one that predicts operational behaviour.
Real traces also preserve the context that makes failures legible. A final outcome might look like a simple miss, but the trace can reveal whether the root cause was bad retrieval, tool misuse, stale state, unsafe delegation, or an incorrect stop condition. When the evaluator can see that path, scoring becomes tied to the mechanism, not just the symptom.
What traces let you score that abstractions miss
Abstractions tend to flatten the task into a single success or failure label. That hides the specific span where the system went off track. Traces let you score the right segment, for example whether the agent should have asked a clarifying question, used a different tool, or declined to act because the available evidence was insufficient.
They also support more defensible rubric design. If several traces show the same class of mistake, you can turn that pattern into an eval criterion that checks for a concrete failure mode instead of a vague notion of quality. In practice, that produces better separation between systems that are merely polished and systems that are actually reliable.
For teams building or reviewing agent evaluations, the most useful trace-based questions are often, “What decision was available here?”, “What signal should have changed the path?”, and “What evidence would prove the agent saw it?” The point is to bind the score to the causal path, so the result tells you what to fix.
Why trace-derived evals improve iteration speed and control
When evals come from real traces, improvement loops become much more actionable. Engineers can replay a failing episode, inspect the exact branch that broke, and verify whether a change improved the targeted behaviour without accidentally hiding the issue elsewhere. That reduces guesswork and makes regression detection far more precise.
Trace-based evals also improve governance over agent changes. If the same episode can be replayed after a model, prompt, tool, or policy update, you get a stable comparison point for whether behaviour changed in a meaningful way. That is especially important when the system’s risk comes from how it uses tools or privileges, not just from what it says.
In mature practice, trace-backed evals become part of a broader control loop: observe the real workflow, extract the decision points that matter, turn them into tests, and keep those tests aligned as the system evolves. AI Agent Observability, Audit and Incident Response Guide is useful here because it frames the logging and attribution needed to make those traces actionable.
Risk and Threat Considerations
Without real traces, evals can miss the failure path that matters most, especially when an agent’s error only becomes visible several steps later. That creates false confidence: the system may look fine on abstract tests while still taking unsafe actions, overusing tools, or following the wrong delegation path in production.
Failure mechanism: The benchmark measures the end state instead of the path, so the evaluator cannot see the bad decision, the missed signal, or the unsafe handoff that actually caused the outcome.
Impact: Teams ship changes that appear to improve scores but do not reduce real operational risk, and they may miss regressions in tool use, authorization, or recovery behaviour.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, CSA MAESTRO and MITRE ATLAS address the attack and risk surface, while NIST AI RMF sets the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | Real traces expose unsafe delegation and privilege use in agent runs. |
| Recommendation — Score trace spans for privilege misuse and require per-action authorization checks. | ||
| CSA MAESTRO | MAESTRO | Trace-based evals support agentic threat modelling and episode replay. |
| Recommendation — Use episode traces to map failures to threat scenarios and control gaps. | ||
| NIST AI RMF | Govern | Trace-derived evals support AI risk governance and ongoing monitoring. |
| Recommendation — Use operational traces to validate AI risks, controls, and monitoring claims. | ||
| MITRE ATLAS | MITRE ATLAS | Traces help map adversarial agent behaviour to concrete attack techniques. |
| Recommendation — Map traced failures to attack techniques and prioritize detections for them. | ||
Practitioner Guidance
What to prioritise: Start by collecting traces from the workflows that carry the highest operational consequence, not the easiest-to-label examples. The best eval set is usually the one that exposes judgment under uncertainty, retries, and tool choice, because that is where agent failures tend to hide.
What to verify: Make sure every eval case can point to a concrete trace span, an observable decision point, and a scoring reason that a reviewer would agree with. If you cannot explain why that exact span is being scored, the eval is probably too abstract to be reliable.
Common mistake: Teams often build evals from synthetic scenarios or polished transcripts that look tidy but omit the messy intermediate steps. Those cases are easier to grade, but they are much worse at predicting whether the agent will behave safely when the real workflow branches or degrades.
Practitioner takeaway: The strongest evals are built from the episodes that actually happened, because only real traces show where the system made the consequential choice.
Related resources from NHI Mgmt Group
- What breaks when agent evals are built only from synthetic examples?
- How should teams investigate recurring failures in agent production traces without losing the thread across code, evals, and monitoring?
- What is the difference between ephemeral credentials and real agent governance?
- What breaks when AI agent access is not re-evaluated in real time?