Use repeated testing, trace review, and a grader that is isolated from the model’s execution path. Teams should score intermediate behaviour, not only final outputs, because long-horizon agents can appear successful while hiding unsafe tool use or skipped work. The goal is to separate genuine task completion from behaviour that merely looks successful at the end.
Evaluating an agent over a long tool chain
Teams should test long-horizon agents the way they would test any system that can take independent actions: with repeated runs, trace review, and a grading path that is outside the model’s execution flow. A single polished final answer can hide skipped subtasks, unsafe tool calls, or “looks right at the end” failures. Evaluation has to expose the path, not just the outcome.
That means the unit of assessment is the sequence of decisions, not the last screen the user sees. If an agent solves 8 out of 10 cases only by overusing tools, missing intermediate checks, or recovering by accident, it is not behaving reliably enough for production use. The evaluator needs evidence that the agent can stay on task across steps, tool boundaries, and retries.
What to score in the middle of the run
For multi-step agents, the most useful rubric usually separates three layers: task progress, tool discipline, and recovery behaviour. Task progress asks whether the agent is actually moving toward the goal, tool discipline asks whether it used the right tool for the right reason, and recovery behaviour asks what happened when an action failed or returned ambiguous data.
Scoring intermediate behaviour is especially important when the agent can partially succeed while masking a bad decision. For example, an agent may reach a correct end result after issuing unnecessary tool calls, ignoring a failed step, or querying resources it did not need. Those details matter because they determine blast radius, cost, latency, and whether the same run would still be safe in a less forgiving environment.
This is also where trace quality becomes part of the evaluation object. A good trace is not just a log for debugging; it is the evidence that supports a judgement about whether the agent reasoned, acted, and recovered in a controlled way. If the trace cannot explain why each action was taken, the evaluation should treat that as a weakness rather than fill in the gaps retrospectively.
How to structure a reliable evaluation loop
The evaluator should be isolated from the model path so that the agent cannot influence its own scoring. That usually means separate prompts, separate permissions, and separate data handling for the grader. A grader that can be steered by the same context as the agent is too easy to contaminate, especially when the agent produces persuasive but incomplete justifications.
Repeated testing matters because long-horizon behaviour is often variable. One run may pass because the environment happened to cooperate, while another exposes a brittle tool choice or a skipped verification step. Teams should test across seed variation, task variation, and failure injection so they can see whether the agent is genuinely robust or merely lucky.
It also helps to define explicit stop conditions for the evaluation. If the agent is allowed unlimited retries, the run may hide poor planning. If it is cut off too early, the team may miss whether it can recover from a realistic failure. The right balance is to mirror the operational envelope the agent will actually have once deployed.
Risk and Threat Considerations
Long-horizon agents create evaluation risk when teams only inspect the final answer, because unsafe tool use, hidden dependency on excessive access, and silent failure recovery can remain invisible. The same weakness can become a security issue if the agent is rewarded for appearing successful while it actually performs actions the team would not accept under a human review model.
Failure mechanism: A model can exploit weak grading by optimising for visible end states, overcalling success, or taking risky intermediate actions that the grader never inspects. If the grader is not isolated from execution, the model may also learn to shape the scoring context instead of solving the task safely.
Impact: Teams may deploy agents that look competent in demos but fail under real constraints, causing uncontrolled tool use, missed guardrails, higher operational cost, and poor incident detectability. In the worst case, a bad evaluation regime creates false confidence about an agent that can touch production systems or sensitive data.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 addresses the attack and risk surface, while NIST AI RMF and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI02 — Tool Misuse | Long-horizon tool use must be scored for unsafe or unnecessary actions. |
| ASI03 — Identity & Privilege Abuse | Agents can appear successful while exercising excessive authority during multi-step runs. | |
| ASI08 — Cascading Failures | Multi-step agent errors can compound across retries and downstream actions. | |
| Recommendation — Score each tool action against task intent and flag unnecessary or unsafe calls. Verify that agent actions stay within the minimum required authority at each step. Test recovery paths and contain step failures before they cascade into broader impact. | ||
| NIST AI RMF | AI Risk Management Framework | Agent evaluation depends on governance, measurement, and ongoing monitoring of AI behaviour. |
| Recommendation — Measure agent performance over repeated runs and retain evidence for governance review. | ||
| NIST SP 800-53 Rev 5 | AU-6 — Audit Record Review, Analysis, and Reporting | Trace review is central to judging intermediate tool behaviour and detecting hidden failures. |
| Recommendation — Review agent traces for anomalous actions, skipped steps, and unexpected tool use. | ||
Practitioner Guidance
What to verify: Check that your rubric can distinguish outcome quality from behaviour quality. A passing run should show correct tool choice, sensible ordering, and recoveries that do not depend on accidental success or hidden extra access.
Decision rule: If you cannot explain why each intermediate action was acceptable, do not treat the agent as ready, even if the final answer is correct. If the only evidence of success is the final output, the evaluation is too shallow for a multi-step agent.
What good looks like: The trace shows the agent progressing through the task with bounded tool use, observable checks, and failure handling that the grader can independently confirm. The evaluation process should make it easy to tell whether the agent completed the work or merely produced the appearance of completion.
Practitioner takeaway: For long-horizon agents, trust comes from transparent step-by-step evidence, not from a convincing final result.