AI agent red teaming must test actions, not just model output

At a glance

What this is: This is a guide to enterprise AI agent red teaming, with the central finding that agent security has to test actions, workflows, tools, and memory rather than only model responses.

Why it matters: It matters because AI agents can cross the boundary from content generation into privileged execution, so IAM, PAM, and NHI teams need controls that match multi-step behaviour, not just prompt safety.

By the numbers:

• AI-enabled adversary activity rose 89% year over year in 2025, according to CrowdStrike's 2026 Global Threat Report.
• Security researchers filed more than 30 CVEs against MCP servers, clients, and related infrastructure in the first two months of 2026 alone, according to Akto.
• 2, survey of over 2,600 implementations found most carried some form of path traversal or command injection risk, according to Akto.
• Enterprise programs should assume AI agent risk is already material: 80% of organisations report their AI agents have performed actions beyond intended scope, according to SailPoint.

👉 Read Akto's complete guide to enterprise AI agent red teaming

Context

AI agent red teaming exists because conventional LLM testing stops at the model boundary, while agents can invoke tools, write data, and trigger business processes. Once an AI agent can act on behalf of a user or system, the security question changes from whether a prompt is unsafe to whether the system can be manipulated into taking an unsafe action.

That matters for identity governance as much as for AI security. Agents increasingly operate as non-human identities, with permissions, memory, and delegation paths that can expand blast radius if they are not continuously tested and constrained. Enterprise programs that still treat agent behaviour as a chatbot problem will miss the operational risk that appears when tools, access, and decisioning converge.

Key questions

Q: How should security teams test AI agents that can call tools and APIs?

A: They should test the agent's actual execution paths, not just its text responses. That means covering tool calls, protocol handling, approvals, and state transitions with adversarial scenarios that mirror real misuse. If the test cannot reach the access point where the agent acts, it does not meaningfully assess the security risk.

Q: Why do AI agents create a different access problem from human developers?

A: AI agents create a different access problem because they can parallelise work, retrieve context on demand, and initiate actions without the pauses that human workflows naturally create. That changes the control objective from managing interactive users to governing high-volume machine actions. Identity teams need to account for speed, chaining, and fan-out, not just login events.

Q: What do security teams get wrong about agent red teaming?

A: The biggest mistake is assuming a one-time prompt test gives meaningful coverage. Agent risk is stateful, and new tools or memory changes can reopen old weaknesses. Teams also underweight business impact, even though the real question is whether an attacker can trigger an unauthorized transaction, data exposure, or production change.

Q: When should organisations treat an AI system as a non-human identity?

A: Treat an AI system as an NHI when it can authenticate, request tools, or perform actions without direct human supervision. At that point it needs inventory, lifecycle, least privilege, monitoring, and revocation controls just like other machine identities.

Technical breakdown

Why agentic AI changes the red teaming target

Traditional LLM red teaming measures whether a model can be pushed into producing harmful, biased, or policy-violating text. Agentic systems require a different lens because the model sits inside an execution environment with tools, memory, and permissions. The relevant failure is no longer only unsafe output. It is unsafe action, such as calling an API, modifying a record, or progressing a workflow the user never authorised. That is why agent red teaming must evaluate the full chain from prompt to tool call to business effect, not just the response text.

Practical implication: test what the agent can do, not only what it says.

MCP servers, tool trust, and privilege boundaries

The Model Context Protocol gives agents a standard way to discover and use external tools, but every connected MCP server becomes a trust boundary. If authentication, tool descriptions, or token handling are weak, an attacker can turn a legitimate integration into an execution path. This is where tool poisoning, unauthorized tool invocation, and privilege creep appear. In governance terms, the agent is only as constrained as the narrowest tool permission it can reach, which makes delegated access design part of the security model rather than an implementation detail.

Practical implication: inventory every MCP connection and validate its access scope before release.

Multi-step workflows create hidden attack paths

Single-turn prompt tests miss the most damaging agentic failures because the exploit often emerges over several steps. A benign-looking instruction can shape the agent's memory, alter its goal, and then steer a later tool call toward exfiltration or fraud. The same issue appears in multi-agent systems, where one compromised agent can pass bad data or spoofed messages downstream. Red teaming therefore needs workflow replay, stateful testing, and cross-agent abuse cases, because the control gap is usually in the chain, not in one isolated response.

Practical implication: build tests that replay complete workflows and shared-context transitions.

Threat narrative

Attacker objective: The attacker wants the AI agent to carry out a harmful business action while appearing to act within normal workflow boundaries.

1. Entry begins with direct or indirect prompt injection, or with a compromised tool or MCP integration that lets an attacker influence the agent's context.
2. Escalation follows when the agent accepts poisoned instructions, expands tool use, or reaches beyond its intended permissions through weak scope enforcement.
3. Impact occurs when the agent performs an unauthorized action such as data exfiltration, fraudulent transaction processing, or production record modification.

NHI Mgmt Group analysis

AI agent red teaming is becoming a governance control, not just a security test. Once an agent can call tools, remember context, and act across sessions, the risk moves from model behaviour to delegated execution. That makes red teaming part of identity governance, because the core question becomes what the agent is allowed to do, for whom, and under what conditions. Practitioners should treat agent testing as an access-control validation discipline, not a one-off model review.

Agent identity sprawl is the real control problem. Enterprises are not deploying one agent, they are deploying many agents, many tools, and many release paths. That creates a non-human identity surface where permissions drift faster than manual review can track, especially when MCP servers and shared memory are involved. Agent identity sprawl: the rapid multiplication of tool-enabled agent identities, each with its own access path and failure mode. The practical conclusion is that lifecycle control matters more than isolated prompt safety.

Static red teaming produces false confidence when systems are stateful. A prompt library can tell you whether a model rejects a bad request, but it cannot show how a workflow behaves after memory is updated, a tool is chained, or a previous step changes the decision path. The article reinforces a broader industry pattern: the dangerous failure is often delayed until the third or fifth action. Security teams should therefore judge coverage by workflow depth and state transitions, not by test count alone.

Tool trust is the new privileged access problem. In agentic systems, privilege is expressed through tools, APIs, and contextual reach rather than only human login sessions. That means least privilege, approval boundaries, and authentication of tool endpoints matter in the same way PAM matters for human administrators. The practitioner takeaway is clear: if the agent can reach production systems, its tool chain needs the same governance discipline as any other high-risk access path.

Continuous agent red teaming will increasingly map to OWASP and NIST governance models. The article aligns naturally with OWASP Agentic Applications guidance and the NIST AI Risk Management Framework because both frameworks emphasise measurable controls, monitoring, and accountability. That matters for board reporting and audit evidence as much as for engineering. Teams should expect red team findings to become part of compliance and assurance workflows, not just security backlogs.

What this signals

Agentic AI security is now moving into the same governance lane as IAM and PAM because the failure mode is no longer limited to unsafe content. When an AI system can call tools and persist state, it needs lifecycle oversight, approval boundaries, and revocation logic that look closer to non-human identity governance than to prompt filtering. The practical signal for security leaders is that agent inventory, ownership, and auditability are becoming board-relevant controls, not experimental add-ons.

Agent identity sprawl: enterprises will need to manage a rapidly expanding population of tool-enabled AI identities across development, operations, and business workflows. That will push teams toward explicit policy for access scope, memory isolation, and release gating, with the OWASP Agentic Applications Top 10 and the NIST AI Risk Management Framework giving a defensible structure for that work. The programmes that act early will be the ones that can prove control coverage when the next agent workflow goes live.

For practitioners

• Inventory agents, tools, and MCP servers continuously Build a live inventory of every AI agent, MCP server, connected API, and data source. Include shadow deployments and homegrown workflows, then map which identities, secrets, and permissions each system can reach. Without that map, red teaming coverage will always be incomplete.
• Test workflow chains, not single prompts Design probes that replay multi-turn, stateful workflows and measure whether the agent can be pushed from harmless input to harmful action. Include prompt injection, tool misuse, memory poisoning, and cross-agent abuse paths in the same test library.
• Treat tool permissions as privileged access Review every tool call path for scope creep, default trust, and weak authentication. If an agent can write to production systems, trigger payments, or access confidential records, apply explicit approval, least privilege, and revocation controls.
• Integrate red team findings into release gates Make agent red teaming part of CI/CD so new prompts, tools, memory changes, and MCP integrations are tested before deployment. Turn confirmed exploits into permanent regression tests so the same failure does not return in a later release.
• Align testing outputs to governance frameworks Map findings to OWASP Agentic Applications, OWASP NHI Top 10, MITRE ATLAS, and the NIST AI Risk Management Framework. That gives security, risk, and audit teams a common way to prioritise findings and evidence control coverage.

Key takeaways

• AI agent red teaming has to measure harmful action, not just harmful text, because the control failure appears when an agent can execute across tools and workflows.
• The real risk sits in the identity and privilege layer, where agent permissions, MCP trust boundaries, and memory persistence can turn a small prompt issue into a business-impacting event.
• Continuous workflow-based testing, mapped to OWASP and NIST frameworks, is now the practical way to govern AI agents at enterprise scale.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• MCP Server: An MCP server is a tool endpoint that connects an AI agent to external systems and data sources through Model Context Protocol. Because it extends what the agent can reach, it becomes part of the identity and access surface and must be reviewed like any other privileged connector.
• Agent Identity: An agent identity is the set of attributes, credentials and permissions assigned to an autonomous software entity. It is treated as a non-human identity because it can authenticate, act on systems and accumulate access over time, which creates governance, audit and lifecycle obligations similar to other production identities.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.

What's in the full article

Akto's full guide covers the operational detail this post intentionally leaves for the source:

• Step-by-step probe design for tool misuse, memory poisoning, and cross-agent attacks
• Coverage guidance for MCP-specific testing and workflow replay at release time
• Implementation detail on mapping findings to OWASP Agentic Applications and MITRE ATLAS
• Practical guidance for connecting red team results to runtime guardrails and CI/CD gates

👉 The full Akto guide covers probe libraries, MCP attack paths, and continuous testing workflows in more implementation detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and secrets management. It helps security practitioners translate identity control principles into programmes that can govern agents, workloads, and privileged access consistently.