How should security teams evaluate AI agents that make tool calls and update systems across multiple steps?

Security teams should evaluate agents at the workflow level, not just the response level. Test the final outcome, the tool choices, the parameter quality, and each intermediate decision that leads to system changes. Use repeatable tasks, multiple trials, and controlled data so failures can be traced to the exact step that caused them.

Why This Matters for Security Teams

Agents that call tools and mutate systems are not being judged on a single answer; they are being judged on whether a sequence of decisions stays safe under real operational pressure. That changes the test surface from “is the text correct?” to “does the workflow preserve trust, authorization, and data integrity at every step?” Current guidance from the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework both point toward runtime evaluation, not static trust in the model.

This is especially important because agents can chain tools, reuse context, and propagate an early mistake into a later system change. A harmless-looking retrieval step can become a bad parameter in a write operation, and a bad write can become an operational incident. NHI Management Group has documented the real-world impact of compromised AI identities in LLMjacking: How Attackers Hijack AI Using Compromised NHIs, where exposed credentials were abused quickly enough to show how short the window for safe evaluation can be.

In practice, many security teams discover unsafe agent behavior only after a workflow has already updated a production system or exposed a sensitive record.

How It Works in Practice

Evaluate the agent as a workflow with observable checkpoints. That means tracing the task from intent, to tool selection, to parameter construction, to execution, to post-action verification. The right question is not just “did the agent answer correctly?” but “did it choose the correct tool, supply safe arguments, limit scope, and stop when the task was complete?”

A practical test plan should include repeatable scenarios with controlled data, multiple trials, and explicit expected outcomes. Security teams should log each intermediate decision so failures can be tied to the exact step that introduced risk. This is where policy-based controls matter: runtime authorization, step-level approvals for sensitive actions, and context-aware checks aligned to the CSA MAESTRO agentic AI threat modeling framework and OWASP NHI Top 10.

• Test final outcome and every intermediate tool call.
• Validate parameters, not just tool names.
• Use short-lived credentials and scoped permissions for each workflow.
• Record prompts, tool outputs, policy decisions, and system deltas.
• Re-run the same task under varied inputs to expose brittle behavior.

For higher-risk systems, align these evaluations with MITRE ATLAS adversarial AI threat matrix so abuse paths such as lateral tool chaining and privilege escalation are explicitly exercised. These controls tend to break down when agents have unrestricted write access to production systems because one flawed step can immediately become an irreversible change.

Common Variations and Edge Cases

Tighter evaluation often increases test overhead, requiring organisations to balance coverage against release speed. That tradeoff is real, but current guidance suggests the cost is lower than investigating a bad agent action after it has touched a live system.

There is no universal standard for agent evaluation yet, so teams need to choose the level of rigor based on blast radius. Low-risk drafting agents may only need output review and limited tool simulation. Agents that can create tickets, move funds, delete records, or change infrastructure should be tested like privileged workloads, with approval gates and deterministic rollback paths. The Replit AI Tool Database Deletion incident illustrates why destructive actions need stricter evidence than normal QA gives.

Teams should also treat multi-agent pipelines differently from single-agent tasks. One agent may pass an action to another with altered intent, which makes source-of-truth logging and handoff validation essential. For this reason, the Analysis of Claude Code Security and broader agentic findings from NHI Management Group are useful reminders that the workflow boundary, not the prompt boundary, is where risk accumulates.

Best practice is evolving, but the principle is stable: evaluate what the agent did, why it did it, and whether each step remained within policy.

Related resources from NHI Mgmt Group

• How should security teams limit the risk from AI agents that have access to production systems?
• How should security teams govern AI agents that can access enterprise systems?
• How should security teams evaluate agents that change state across multiple steps?
• How should security teams manage permissions for AI agents?