How should security teams evaluate agentic AI workflows that use multiple tools and maintain state across turns?

Security teams should evaluate the full trajectory, not just the final answer. That means checking tool selection, parameter validity, error handling, memory use, and whether the agent follows the intended workflow from start to finish. Multi-turn scoring and trace replay help expose failures that simple logging misses, especially when an agent can look successful while taking a messy or unsafe path.

Why This Matters for Security Teams

agentic ai workflows create risk because the security decision is no longer limited to a single model output. Once an agent can call tools, write to memory, and continue across turns, the exposure shifts to task planning, tool authorization, state persistence, and recovery from failure. That makes evaluation closer to a control review than a prompt test. The NIST AI Risk Management Framework is useful here because it pushes teams to assess system-level risks, not just model quality.

Practitioners often miss that an agent can appear successful while taking an unsafe route, such as using the wrong tool, over-collecting data, or persisting a bad assumption into later turns. That is why security teams should review the full chain of action, including what the agent tried, what it selected, what it stored, and what it recovered from after an error. Evaluation should also cover whether the workflow is resilient when tool responses are delayed, partial, ambiguous, or maliciously shaped.

In practice, many security teams encounter unsafe agent behaviour only after a tool misuse or memory leak has already affected downstream systems, rather than through intentional evaluation of the full workflow.

How It Works in Practice

Security evaluation should combine functional testing, adversarial testing, and trace review. The right question is not only whether the agent completed the task, but whether it did so with authorised actions, correct tool arguments, and bounded state. That is the practical difference between a chatbot and an agentic workflow. Guidance from the OWASP Top 10 for Agentic Applications 2026 is especially relevant because it frames tool misuse, excessive agency, and unsafe memory patterns as concrete risk areas.

A strong evaluation process usually includes:

• Tool allowlisting and deny-by-default rules for actions that change data, send messages, or invoke external systems.
• Replayable traces that capture prompts, tool calls, parameters, outputs, and state transitions across turns.
• Assertions for each step, such as expected tool choice, parameter bounds, and required human approval points.
• Tests for prompt injection, poisoned retrieval content, stale memory, and privilege escalation through chained actions.
• Scoring that separates task success from policy compliance, so a correct end result does not hide unsafe intermediate behaviour.

Where possible, teams should align these tests with adversarial techniques from the MITRE ATLAS adversarial AI threat matrix and agent threat modelling approaches such as the CSA MAESTRO agentic AI threat modeling framework. That helps connect evaluation to realistic abuse paths rather than abstract model behaviour. Current guidance suggests treating stateful agents as systems with an attack surface that grows over time, not as isolated inference calls. These controls tend to break down when the agent has unrestricted tool access across multiple tenants because trace fidelity and containment both weaken.

Common Variations and Edge Cases

Tighter agent oversight often increases latency and operational overhead, requiring organisations to balance safety against workflow speed and user experience. That tradeoff matters because not every agent needs the same depth of review. High-risk workflows, such as those touching secrets, financial actions, customer records, or production systems, should get the most stringent step-level evaluation, while lower-risk assistants may be assessed with lighter sampling and policy checks.

Best practice is evolving for state management, because there is no universal standard yet for how much memory an agent should retain, how long it should keep it, or when it should forget it. In some environments, persistent memory is useful for continuity but dangerous if it stores unverified assumptions, sensitive data, or attacker-controlled instructions. Security teams should test for memory poisoning, cross-session leakage, and whether the agent can recover safely after corrupted state.

Where agentic systems interact with regulated or security-sensitive operations, it is also worth mapping evaluation to control families in the NIST SP 800-53 Rev 5 Security and Privacy Controls. That gives governance teams a clearer way to evidence access restrictions, logging, and response handling. The main exception is highly dynamic environments with rapidly changing tools or permissions, where static test suites become stale quickly and continuous red-team style evaluation becomes more valuable than periodic review.

Related resources from NHI Mgmt Group

• How should security teams govern AI workflows that use multiple tools and data sources?
• How should teams govern agentic AI when the model can act across multiple tools and services?
• How should security teams govern AI use cases across multiple business units?
• How should security teams evaluate AI pentesting tools for enterprise use?