Function-style tests assume fixed inputs produce fixed outputs, but agents are path-dependent and can choose different tools, memory paths, or timings on each run. That means a model may look correct in isolation while the surrounding system fails in production. Teams need trajectory-based evaluation, not just result checking, to understand real agent risk.
Why This Matters for Security Teams
Evaluating an AI agent like a normal function creates false confidence because the thing being tested is not just a model output, but an execution path that can change with tool availability, memory state, prompt context, and timing. Security teams often focus on whether the answer is right, but agent risk also includes whether the agent invoked the wrong tool, exposed secrets, escalated privileges, or followed a malicious instruction. That is why current guidance from the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10 emphasizes governance, misuse resistance, and runtime controls rather than output correctness alone.
The practical issue is that agents operate across multiple layers of risk at once: model behavior, orchestration logic, tool permissions, memory persistence, and external dependencies. A unit test can verify that a prompt returns a plausible response, but it cannot prove the agent will resist prompt injection, remain within its task boundary, or recover safely after a partial failure. In practice, many security teams encounter agent misuse only after a tool has already been called, rather than through intentional evaluation design.
How It Works in Practice
Trajectory-based evaluation treats the agent as a sequence of decisions instead of a single response. That means testing not only the final answer, but also the intermediate steps: which tools were selected, what context was loaded, whether the agent asked for inappropriate permissions, and whether it followed policy when instructions conflicted. This approach aligns with the control logic behind MITRE ATLAS adversarial AI threat matrix because adversaries often target the path, not just the output.
Operationally, teams should evaluate agents across a representative set of trajectories:
- benign requests with normal tool calls
- prompt injection attempts that try to redirect the agent
- stateful scenarios where memory changes later decisions
- tool failure cases where the agent must recover safely
- high-risk prompts that should trigger refusal or escalation
Good practice also includes logging the reasoning trace, tool chain, and policy decisions so defenders can compare expected and actual behaviour. For higher-risk deployments, the CSA MAESTRO agentic AI threat modeling framework is useful for mapping where the agent can be manipulated, while the Anthropic AI-orchestrated cyber espionage report shows why autonomous sequencing must be treated as a security boundary. These controls tend to break down when the agent relies on hidden state, dynamic tool discovery, or loosely governed plugins because the execution path becomes non-deterministic and hard to reproduce.
Common Variations and Edge Cases
Tighter agent evaluation often increases testing cost and operational overhead, requiring organisations to balance coverage against speed, especially when agents are updated frequently. There is no universal standard for this yet, so best practice is evolving. Some teams use sandboxed replay harnesses, others simulate adversarial tool use, and more mature programs add policy assertions for each step rather than each final answer. The point is to detect unsafe trajectories before production users do.
Edge cases matter. Agents with long-term memory can appear stable in isolated tests but drift across sessions. Agents that call internal APIs may pass functional checks while still violating least-privilege expectations. Multi-agent workflows create another failure mode: one agent can behave correctly while another introduces unsafe context, a pattern reflected in the agentic attack taxonomy documented by the OWASP Top 10 for Agentic Applications 2026. The most reliable evaluations therefore combine trajectory review, permission scoping, and adversarial testing, not just pass or fail scoring on the final output.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, MITRE ATLAS and CSA MAESTRO address the attack and risk surface, while NIST AI RMF and NIST AI 600-1 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | GOV-1 | Agent evaluation needs governance, ownership, and risk accountability across the full workflow. |
| OWASP Agentic AI Top 10 | LLM08 | Prompt injection and unsafe agent actions are central agentic failure modes here. |
| MITRE ATLAS | AML.TA0001 | Adversarial manipulation targets agent trajectories, not just final responses. |
| CSA MAESTRO | Agentic threat modeling helps expose state, tool, and orchestration weaknesses. | |
| NIST AI 600-1 | GenAI controls support safer evaluation of autonomous behaviours and outputs. |
Assign owners, define risk tolerance, and govern agent evaluations beyond single-output testing.
