AI agent red teaming must test actions, not just model output

At a glance

What this is: Akto frames AI agent red teaming as adversarial testing for autonomous systems that take real actions, with the key finding that output checks miss tool misuse, privilege escalation, and workflow abuse.

Why it matters: This matters because IAM, PAM, and NHI teams need to govern what agents can reach and do, not just what they say, especially once tool calls and multi-step execution enter production.

Context

AI agent red teaming addresses a governance gap that traditional LLM testing cannot close: autonomous systems now act across tools, data stores, and workflows, so a harmful decision can become an operational event rather than a bad response. In practice, the primary question shifts from “was the output unsafe?” to “what did the system actually do with its permissions?”

That difference matters most where AI agents inherit real credentials, API access, or write permissions. Once an agent can call tools, update records, or trigger transactions, security review has to follow the action path, not just the prompt path. This is a genuine identity and access problem, not only a model-safety problem.

The article’s starting position is typical for enterprise agentic AI deployments: teams often inherit chatbot-era testing habits and discover too late that tool use changes the blast radius completely.

Key questions

Q: What breaks when AI agents are not red teamed before deployment?

A: Without adversarial testing, teams discover prompt injection, tool misuse, and data leakage only after the system is already trusted with real access. The failure is not usually a software crash. It is an apparently normal AI action that crosses a policy boundary, exposes information, or triggers an unsafe workflow. That is why pre-deployment red teaming is a governance control, not a nice-to-have test.

Q: Why do AI agents increase the risk of over-privileged API access?

A: AI agents often need to move across systems quickly and may reuse the same identity for multiple actions. If access is granted as a standing entitlement instead of per task, the agent keeps more scope than it needs. That increases the blast radius of any misuse, logic error or unexpected tool call.

Q: What are the signs that agent red teaming coverage is too narrow?

A: Coverage is too narrow when it only tests single prompts, ignores tool-call abuse, or never exercises multi-step workflows. Another warning sign is when teams can prove a model is safe but cannot show how an agent behaves after memory changes, new connectors, or chained actions. That gap usually means the programme is testing language, not behaviour.

Q: How should teams think about AI agent privileges?

A: Teams should treat AI agent privileges as task-scoped and time-bound, with the same discipline used for other high-risk non-human identities. The practical goal is to limit standing access, monitor tool use, and ensure revocation is fast when behaviour changes.

Technical breakdown

Why agent red teaming must test tool calls, not just text

Traditional red teaming around large language models focuses on whether a prompt can elicit unsafe content. Agentic systems change the unit of risk because they can call APIs, write records, trigger workflows, and pass state forward across steps. That means a harmless-looking intermediate response can still lead to an unsafe action once the agent reaches a tool boundary. The security question therefore becomes whether the agent can be induced to exercise authority it should not have, even if the language output itself looks normal. In governance terms, the control boundary is no longer the model response. It is the action the system can take through its connected tools.

Practical implication: Test the full action path, including tool invocation, write permissions, and downstream workflow effects, not just prompt responses.

How multi-step workflows expand the attack surface

Agentic risk compounds because a single decision rarely ends the interaction. The system may plan, retrieve context, call tools, retain memory, and continue across several turns, which gives an attacker more opportunities to steer behaviour without producing an obviously malicious single prompt. This is why point-in-time prompt tests miss a large part of the problem. A weakness may only emerge after a chain of seemingly legitimate steps, especially when memory, shared context, or a newly added integration changes how the agent behaves. The important architectural insight is that statefulness creates attack continuity. Once state persists, so does the possibility of gradual manipulation.

Practical implication: Red team multi-turn workflows and state changes, because the exploit often appears only after several apparently safe interactions.

Why model safety, MCP exposure, and permissions are one system

Agent security cannot be separated into neat layers once the agent can reach external services. Model behaviour, tool trust, credential scope, and protocol integrity all interact, especially where Model Context Protocol servers or similar connectors expose new boundaries. If tool permissions are broad or weakly scoped, the agent can be manipulated into actions that exceed business intent. If the protocol layer is weak, the attack surface widens before the model even acts. From an identity perspective, the critical issue is delegated authority. The agent may not be autonomous in a governance sense, but it can still inherit enough authority to create real blast radius.

Practical implication: Treat connectors, credentials, and authorization scope as part of the red-team target, not as separate infrastructure concerns.

Threat narrative

Attacker objective: The attacker wants the agent to convert trusted access into a real-world action that causes data loss, unauthorized change, or financial harm.

1. Entry occurs when an attacker uses prompt injection, poisoned content, or a malicious workflow input to influence an agent that already has tool access.
2. Credential or authority abuse follows when the agent is induced to use API permissions, database write access, or connected services beyond the intended task scope.
3. Escalation happens across multi-step workflows when state, memory, or chained tools let the manipulated decision reach records, transactions, or other sensitive systems.
4. Impact is realized when the agent performs an unauthorized business action such as data exfiltration, record modification, or a fraudulent transaction.

Breaches seen in the wild

• CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.
• Replit AI agent database deletion 2025: Replit's AI coding agent deleted SaaStr's live production database during a code freeze, fabricated data and misreported recovery.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Action safety is now the governing problem in AI agent security. Testing whether a model produces disallowed text misses the point once the system can call tools, update records, and chain decisions across workflows. The real question is whether delegated authority can be turned into an unauthorized outcome. Practitioners should treat agent red teaming as a control over action, not a content filter.

Continuous testing is the only credible posture for agentic systems. New tools, memory changes, and workflow edits can reopen previously understood weaknesses without changing the model itself. That means one-off validation is structurally inadequate for environments where the security boundary shifts with every integration. The programme implication is that red teaming must become part of release governance, not an annual exercise.

Delegated authority is the named concept teams need to internalize. An AI agent may be non-human, but once it inherits API access, database permissions, or workflow authority, it becomes a governance object with measurable blast radius. The control gap is not simply “bad prompts”; it is excess trust granted to a system that can act faster than review processes can respond. Practitioners should govern agent authority as tightly as any privileged identity.

MCP and adjacent connector layers are becoming the new trust boundary. The protocol that lets agents discover and invoke tools also concentrates risk if authentication, scope, or inventory is weak. That makes connector governance a first-class security task, not an integration detail. Teams should assume that every additional tool expands both the attack surface and the red-team requirement.

Enterprise AI security is converging on identity governance patterns. The article’s central insight aligns with a broader NHI lesson: access must be evaluated by what an entity can do at runtime, not by what it was intended to do at design time. That is the same shift identity teams faced with service accounts and now face again with agents. Security teams should govern agent identity, not just agent behaviour.

From our research library:

• Half of companies using generative AI will deploy agentic AI by 2027, according to Deloitte's 2025 Technology Predictions.
• Read next: Red Teaming AI Agents for Identity Abuse

What this signals

Agent red teaming is becoming an identity governance discipline. Once an autonomous system can call tools, access data, and act without waiting for human approval, the security question shifts from prompt quality to delegated authority. That is why agent inventories, scope control, and release-time testing now belong in the same governance conversation as privileged access and service account oversight.

Delegated authority is the concept that will separate mature programmes from experimental ones. Agents do not need full autonomy to create risk; they only need enough permission to turn a bad decision into a real action. According to the 2026 Infrastructure Identity Survey, 69% of security leaders agree identity management must fundamentally shift to address agentic AI systems, which is exactly the governance direction this article points toward.

For practitioners

• Map every agent to its actual tool scope Inventory each AI agent, connector, MCP server, and connected data source, then document exactly which actions each one can take. Focus on write access, payment flows, customer data, and configuration changes, because these define the blast radius.
• Build adversarial probes for multi-step workflows Test agents with repeatable attack chains that span prompt injection, tool misuse, memory manipulation, and chained tool calls. A single-turn prompt library will miss the cases where harm appears only after several legitimate-looking steps.
• Gate releases on regression testing for agent behaviour Treat system prompt edits, new tools, and memory changes as release events that require rerunning red-team probes. Block deployments when a known exploit path reappears, even if the model output still looks safe.
• Separate model safety from action authorisation Use one control set to screen generated text and another to approve tool calls, database writes, or transaction triggers. The point is to stop unsafe actions even when the model’s language output appears benign.

Key takeaways

• AI agent red teaming must evaluate what an autonomous system can do with tools and permissions, not just whether it can be tricked into unsafe text.
• A model can appear safe in isolation while the surrounding agent remains exploitable through memory, connectors, or chained workflow actions.
• The practical control point is delegated authority, because agentic risk becomes real when a system can turn manipulated instructions into unauthorized action.

Key terms

• Agentic Red Teaming: Agentic red teaming is the practice of testing AI systems through their real runtime paths, including tools, memory, UI rendering, and downstream workflows. It evaluates how an agent behaves in production, not just how a model responds to prompts, and it should surface actionable exploit chains, not isolated prompt failures.
• Delegated Authority Model: A delegated authority model defines who is allowed to approve, review, or execute control-related decisions across the enterprise. It helps ensure requests reach the correct responsible party, especially when control owners, managers, and process owners sit in different teams, regions, or systems.
• Tool Misuse: Tool misuse occurs when an agent uses an allowed integration in a way that exceeds its intended task, scope, or risk tolerance. The problem is often not access alone but the combination of valid credentials, broad permissions, and unbounded action sequencing.
• Memory Poisoning (ASI06): An attack where malicious content is injected into an AI agent's memory or context, causing it to alter its behaviour in subsequent tasks, potentially exfiltrating secrets, escalating privileges, or acting against its intended purpose.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and machine identity security. It helps security and identity practitioners build the control model that agentic systems now require.