They should test the full trajectory, not just the final response. That means logging every tool call, validating intermediate decisions, and checking the environment left behind after the task completes. A stateful agent can be wrong in ways that only emerge after several steps, so evaluation must cover path quality, side effects, and end state together.
Why This Matters for Security Teams
Evaluating a stateful agent only on its final answer misses the real security risk: the sequence of actions it took to get there. Agents can read sensitive data, invoke tools, modify records, or open new attack paths even when the end result looks correct. That is why evaluation must cover trajectory, intermediate decisions, and residual state, not just output quality. Guidance in the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework both point toward governance that treats agent behaviour as an operational risk surface, not a single-shot model response.
Security teams often under-test state because traditional model evaluation was built for prompts and outputs, while agents introduce tool use, memory, and side effects. A plan that is logically plausible can still be unsafe if it escalates privilege, overwrites data, or leaves a hidden dependency behind. The practical question is whether the agent stayed within policy at every step, not whether the final text was acceptable. In practice, many security teams encounter stateful-agent failures only after a harmless-looking success has already changed systems of record, rather than through intentional trajectory testing.
How It Works in Practice
A useful evaluation method separates the agent run into three layers: what it decided, what it touched, and what it left behind. Start by capturing every prompt, tool call, retrieval event, and state transition. Then compare each step against expected policy, expected permissions, and expected business logic. This creates a trail that can be reviewed for prompt injection, tool misuse, overbroad access, or unsafe chaining of actions. The MITRE ATLAS adversarial AI threat matrix is useful here because it helps teams think about attack paths against agent workflows, not just single model responses.
For practical assessment, teams should test scenarios that force the agent to carry state across steps:
- Ask the agent to gather data, then verify whether it retained only the minimum necessary context.
- Give it a tool that can write or approve changes, then check whether approvals are scoped correctly.
- Inject misleading instructions in retrieved content and observe whether later steps inherit the compromise.
- Inspect the final environment for changed permissions, created tokens, modified records, or dangling sessions.
This is where logging and policy enforcement need to work together. Telemetry should show not only success or failure, but also which state variables changed, which tools were called, and whether any human approval gates were bypassed. If the use case involves autonomous execution, the control set should also reflect baseline hardening and monitoring expectations in NIST SP 800-53 Rev 5 Security and Privacy Controls. These controls tend to break down when agents operate across multiple systems with inconsistent logging, because no single platform can reconstruct the full trajectory.
Common Variations and Edge Cases
Tighter trajectory evaluation often increases logging, review, and test-design overhead, requiring organisations to balance visibility against operational cost. That tradeoff is real, especially when agents are used in high-volume workflows or when tool calls span internal apps, SaaS services, and external retrieval sources. Best practice is evolving, but current guidance suggests that partial observability is not enough for stateful agents, because the most important risk may be an intermediate action rather than the terminal output.
Edge cases usually appear when the agent has long-lived memory, writes to shared systems, or can act on behalf of privileged users. In those environments, the evaluation question should include whether the agent can be safely replayed, rolled back, and bounded within a defined session. The NIST AI Risk Management Framework helps frame this as governance over lifecycle risk, while the CSA MAESTRO agentic AI threat modeling framework is useful where teams need a structured way to reason about agentic workflows. The main exception is tightly sandboxed, read-only agents, where environment residue is limited and the evaluation focus can shift more heavily toward reasoning quality and retrieval integrity.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST AI RMF, NIST CSF 2.0 and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | GOVERN | Trajectory evaluation needs accountability and lifecycle governance for agent behaviour. |
| OWASP Agentic AI Top 10 | Agentic risk includes tool misuse, prompt injection, and unsafe multi-step actions. | |
| MITRE ATLAS | Adversarial AI tactics help model attacks against multi-step agent workflows. | |
| NIST CSF 2.0 | PR.AA,DE.CM,RS.AN | Stateful agent runs need monitoring, analysis, and response across the full execution path. |
| NIST SP 800-53 Rev 5 | AU-2, AC-6, SI-4 | Audit logs, least privilege, and monitoring are essential for traceable agent state changes. |
Implement detailed auditing, constrain privileges, and monitor agent activity continuously.
Related resources from NHI Mgmt Group
- How should security teams govern AI agents that move across multiple trust boundaries?
- How should security teams govern AI agents that reason across multiple data platforms?
- How should security teams audit privileged access across multiple clouds?
- How should security teams manage cloud identities across multiple applications?
