Per-action accuracy can look strong while end-to-end safety fails. Early mistakes change the environment, which changes later observations, beliefs, and choices. In practice, a long action chain can still end in an unauthorised or unsafe state even if most steps were individually acceptable. Teams need trajectory-level testing, because a safe sequence depends on how actions interact over time.
Why per-action scoring can mislead
Per-action accuracy treats each step as if it were independent, but autonomous security agents operate inside a changing environment. A correct-seeming action can still shift the state, narrow future options, or alter what the agent believes it sees next. That is why a chain can look strong step by step and still drift into unsafe territory by the end.
For security decisions, the real unit of analysis is not the isolated action but the trajectory. A single mis-sequenced request, permission escalation, or mistaken assumption can create a cascade that later actions inherit. In other words, the agent may be “right” often enough locally while the overall mission fails globally.
This is especially visible in systems where per-action authorization is applied without checking whether the full sequence remains within policy. The issue is not just whether each action was individually allowed, but whether the combined path stayed safe, intended, and reversible.
What trajectory-level failure looks like in practice
Trajectory-level failure usually shows up as state drift. The agent may gather data, branch on partial evidence, or commit an action that changes the environment in a way later prompts cannot fully recover from. Once that happens, later “correct” actions are judged against the wrong state, so the final outcome can be unsafe even if intermediate steps look acceptable.
That is why security evaluation needs to include agent observability, audit and incident response. If teams cannot reconstruct the sequence of decisions, state changes, and approvals, they cannot tell whether the agent failed on one action or on the interaction between actions.
The same problem appears when an agent is allowed to reuse a capability across steps without re-checking scope. A chain that begins with low-risk reconnaissance can end with a privileged write, deletion, exfiltration, or configuration change that was never safe in combination with the earlier steps.
How to evaluate agents so the metric matches the mission
Security teams should test the full run, not only the step. That means scoring success by whether the final state is authorised, bounded, and recoverable, not whether most actions were individually plausible. If a model can complete a sequence only by accumulating hidden risk, the evaluation method is too weak.
Practically, this means adding trajectory tests, adversarial branching, and stateful replay to your validation process. A useful control is to pair outcome checks with action tracing, so you can see when an apparently harmless early step creates the conditions for later misuse. Zero trust for AI agents is the right mindset here: verify the request and the principal at each step, but also verify that the end-to-end path still satisfies policy.
It also helps to test for rollback and containment. If one action goes wrong, can the system still prevent the next action from compounding the issue? If not, the agent is being evaluated on a metric that understates the actual blast radius.
Risk and Threat Considerations
Per-action evaluation creates blind spots that threat actors can exploit by making each move look innocuous while the full sequence becomes malicious. The danger is compounded when the agent can accumulate permissions, change state, or act on stale assumptions, because the compromise emerges over time rather than in a single obvious event.
Failure mechanism: An early action changes the environment, trust context, or available information, and the evaluation loop fails to account for how that change alters later decisions. The agent then follows a path that is locally acceptable step by step but globally unsafe, unauthorized, or irreversible.
Impact: Teams may miss emerging unsafe behavior, approve agent workflows that create hidden privilege growth, and discover the failure only after harmful state changes, data exposure, or unauthorized actions have already occurred.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 addresses the attack and risk surface, while NIST CSF 2.0 and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | Per-step evaluation can miss privilege growth across an agent's action chain. |
| ASI08 — Cascading Failures | The question is about early mistakes compounding into unsafe end-to-end outcomes. | |
| Recommendation — Test whether an agent's full trajectory stays within intended authority and scope. Model how one action can change later agent behavior and propagate failure. | ||
| NIST CSF 2.0 | ID.RA-01 — Asset vulnerabilities are identified and documented | Trajectory testing depends on identifying where stateful failure can arise across a run. |
| PR.AA-05 — Identities and credentials are managed and verified | Agent actions remain safe only when authority is re-checked across the sequence. | |
| Recommendation — Document stateful failure points that make sequence-level evaluation necessary. Verify authority at each step and prevent implicit permission accumulation. | ||
| NIST SP 800-53 Rev 5 | AU-3 — Content of Audit Records | Trajectory-level testing needs logs that preserve action order and state changes. |
| Recommendation — Record action sequence, context, and state changes for replay and review. | ||
Practitioner Guidance
What to prioritise: Measure the whole episode, not just individual actions. Use trajectory success criteria such as final authorization state, state-change bounds, rollback viability, and whether the agent remained within its intended task scope.
What to verify: Confirm that your evaluation harness captures intermediate state, tool use, and decision context well enough to replay why the agent chose each next step. If you cannot reconstruct the path, you are likely overtrusting per-step scores.
Common mistake: Treating a high step-level pass rate as evidence of safety. For autonomous agents, that often rewards “mostly harmless” micro-decisions while missing the macro-level failure mode that matters operationally.
Practitioner takeaway: The right question is not “did the agent choose acceptable actions?” but “did the sequence of actions leave the system in a safe and authorised end state?”
