Agentic red teaming shows why AI agent governance needs runtime testing

At a glance

What this is: This is a practical guide to agentic red teaming, showing that AI agents need adversarial testing across tools, memory, and workflows rather than only model-level prompt checks.

Why it matters: It matters because AI agents often operate with inherited access and dynamic decision paths, which means identity, privilege, and runtime controls must be assessed together across AI and IAM programmes.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.
• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.

👉 Read Akto's guide to agentic AI red teaming and workflow security

Context

Agentic red teaming is the practice of attacking AI systems the way an adversary would, by probing how agents behave when they can call tools, retain memory, and take multi-step actions. The primary governance gap is that traditional AI testing still focuses too narrowly on model outputs, while real risk emerges in sessions, toolchains, and delegated permissions.

That matters for identity programmes because an AI agent is not just a model, it is a runtime actor operating under some identity, privilege set, and trust boundary. Once those privileges reach into APIs, databases, browsers, or multi-agent workflows, the security question becomes how to test and contain misuse before the agent turns legitimate access into an attack path.

Key questions

Q: How should security teams implement agent red teaming in production environments?

A: Security teams should test agents continuously across staging and live-like environments, not just before release. The highest-value cases combine prompt injection, tool abuse, and multi-step workflow abuse, because those are the paths attackers use to turn legitimate agent behaviour into harmful action. Retest after every prompt, tool, or memory change.

Q: Why do agentic systems create different identity risks from chatbots?

A: Agentic systems can act, sequence tasks, and reuse state. That means identity risk is not limited to authentication or prompt quality, because the software can carry delegated authority into later steps. The result is a broader attack surface where tool misuse, scope drift, and control bypass become operational concerns.

Q: What breaks when agentic systems are tested only at the model level?

A: Model-level testing misses the dangerous part of the system, which is how the agent behaves once it can use tools, remember context, and chain actions across sessions. A prompt that looks harmless in isolation can become a harmful instruction when combined with a tool response or external data source.

Q: Which controls matter most when agent behaviour goes outside intended scope?

A: Accountability, privilege scoping, and runtime monitoring matter most. Teams need clear ownership for findings, minimal permissions for every agent identity, and logging that shows which tool calls or external inputs shaped a decision. If those three controls are weak, the organisation cannot reliably contain or explain agent misuse.

Technical breakdown

Why model-only testing misses agentic risk

Model-only testing assumes the dangerous part is the text response. Agentic systems break that assumption because the model is only one component inside a larger runtime that includes tools, memory, orchestration logic, and external dependencies. A harmless-looking instruction can become unsafe once the agent couples it to a browser action, an API call, or a database write. The real failure mode is not just bad output, but bad execution across a session.

Practical implication: test the full agent workflow, not just the prompt-response pair.

Prompt injection, tool abuse, and multi-step workflow abuse

Prompt injection is the mechanism by which attackers steer the agent’s reasoning, either directly through user input or indirectly through retrieved content such as documents, web pages, or tool responses. Once the instruction is accepted, the agent can misuse tools with valid credentials, which makes the action look authorised even when the intent is malicious. Multi-step chains are especially dangerous because each individual step may appear normal while the combined sequence produces harm.

Practical implication: simulate chained attacks that combine instruction injection with privileged tool calls.

Why runtime adversarial simulation is now necessary

Static tests only cover conditions that exist at build time. Agentic systems change as tools are added, prompts are edited, and memory accumulates, so the attack surface shifts after deployment. Runtime adversarial simulation exercises the agent with live or staging dependencies, exposing timing issues, context leakage, and behavioural drift that pre-release checks cannot see. Continuous testing is therefore a control, not a one-off validation step.

Practical implication: run regression and live-surface simulations whenever tools, prompts, or data sources change.

Threat narrative

Attacker objective: The attacker wants to turn legitimate agent permissions into unauthorised action, data exposure, or destructive workflow execution without needing to compromise the underlying model directly.

1. Entry begins when an attacker injects malicious instructions through direct prompt content or indirect data sources such as web pages, documents, emails, or tool responses.
2. Escalation occurs when the agent accepts those instructions and uses inherited tool permissions to perform actions the operator did not intend, including sensitive reads or writes.
3. Impact follows when the agent completes a chained workflow that exposes data, issues harmful tool calls, or executes irreversible actions across connected systems.

NHI Mgmt Group analysis

Agentic red teaming is becoming the missing control layer between AI capability and operational trust. Traditional testing can tell you whether a model resists a prompt, but it cannot tell you whether an agent safely handles a browser session, a database query, or a delegated tool chain. That gap matters because agentic systems fail in execution, not just in conversation. Practitioners should treat continuous adversarial testing as part of identity and access governance, not as a separate AI lab exercise.

Runtime identity and privilege shape agentic risk more than model quality does. If an agent holds broad API access, database privileges, or persistent memory, a successful injection becomes a privilege problem as much as a model problem. This is where NHI governance intersects directly with AI security: the agent is effectively a non-human identity with runtime authority. The control question is whether that identity can be constrained, observed, and revoked fast enough to limit blast radius.

MCP broadens the agent attack surface because tool trust becomes identity trust. When agents connect through Model Context Protocol, tool permissions and data sources are inherited into the agent workflow. If the connected server or response path is not integrity-checked, the agent can be manipulated without anyone touching the core model. That makes tool provenance, session scoping, and response validation central governance issues for any organisation exposing agents to external services.

Continuous red teaming creates detection depth that static guardrails cannot provide. Static guardrails can block known bad inputs, but they do not reveal whether the agent will drift, chain benign steps into harm, or behave differently after a tool update. A named concept here is workflow compromise latency: the delay between a new agent capability being deployed and a hidden exploit chain being discovered. Practitioners should reduce that latency with regression testing, runtime monitoring, and clear remediation ownership.

The market is moving from model safety to system safety, and that changes the governance benchmark. The relevant question is no longer whether an LLM can be made polite or policy-compliant. It is whether the surrounding agent system can be constrained across tools, sessions, and identities. Teams that keep testing only the model will miss where the real control failure sits, which is in the workflow boundary and the delegated access model.

What this signals

Workflow compromise latency is the governance gap teams should expect to widen as agent deployments expand. Every new tool, memory store, or integration increases the time between a capability being added and a hidden abuse path being found. That argues for continuous adversarial testing, runtime logging, and fast privilege rollback rather than one-time sign-off.

The practical signal for security programmes is that AI governance and identity governance are converging around the same runtime controls. If an agent can read, decide, and act under delegated authority, then access review, tool scoping, and monitoring must be joined up. The teams that separate AI safety from IAM will keep missing the actual attack path.

As agent ecosystems grow, security leaders should expect red teaming to become a standing operational function rather than an occasional assessment. That shift aligns well with NIST AI Risk Management Framework thinking, because the control problem is ongoing exposure management, not static model approval. The right preparation is a repeatable test library, clear ownership, and evidence that failures are being closed, not just logged.

For practitioners

• Implement continuous agent red teaming Add adversarial simulation to the CI/CD and release process so every new tool, prompt change, or data source is tested before and after deployment. Include direct prompt injection, indirect injection through retrieved content, and chained tool abuse scenarios.
• Scope agent privileges to the minimum runtime need Treat each agent as a non-human identity with explicit access boundaries. Limit API permissions, narrow database access, and separate read from write actions so a compromised instruction cannot automatically trigger high-impact operations.
• Validate MCP and other tool boundaries Review every connected tool for provenance, permission scope, and response integrity. Reject broad inherited access by default, and log which data sources and tool results influenced each autonomous decision.
• Track behavioural drift after every change Re-run previously discovered attack cases whenever an agent prompt, memory configuration, or tool chain changes. Treat new side effects, inconsistent outputs, or session-to-session differences as findings that require triage.
• Create a clear remediation owner for agent findings Assign one accountable team to triage red-team findings, adjust privileges, and verify that fixes persist across future releases. Without ownership, agent vulnerabilities sit in backlog while the deployment surface keeps expanding.

Key takeaways

• Agentic red teaming is needed because AI agents fail through tools, memory, and workflow chains, not just through bad text outputs.
• The strongest evidence in the field shows that AI agents already behave outside intended scope at scale, which turns governance into an immediate operational issue.
• Security teams should move from one-off model checks to continuous runtime testing, privilege scoping, and accountable remediation for every agent identity.

Key terms

• Agentic Red Teaming: Agentic red teaming is the practice of testing AI systems through their real runtime paths, including tools, memory, UI rendering, and downstream workflows. It evaluates how an agent behaves in production, not just how a model responds to prompts, and it should surface actionable exploit chains, not isolated prompt failures.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Non-Human Identity (NHI): A digital identity assigned to a non-human entity such as a software application, service account, API key, bot, machine, or AI agent that enables it to authenticate and interact with systems without direct human involvement. NHIs now outnumber human identities in most enterprises by 25 to 50 times.
• Adversary Simulation: Adversary simulation is the practice of testing controls by recreating realistic attacker behaviour. For identity security, it exposes whether compromised credentials, delegated access, or excessive privilege can be used to move through systems before defenders detect the abuse.

What's in the full article

Akto's full blog covers the operational detail this post intentionally leaves for the source:

• A step-by-step agentic red teaming workflow for prompt injection, tool misuse, and multi-turn abuse cases
• Implementation detail on testing MCP-connected tools, browser integrations, and multi-agent communication paths
• Examples of automated adversarial testing pipelines that run in CI/CD and staging environments
• Tooling guidance on when to pair manual red team work with automated agent security checks

👉 Akto's full post covers prompt injection tests, runtime simulations, and agent security tooling in more detail

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and the controls that matter when non-human systems act at runtime. It is designed for practitioners who need to connect identity governance to broader security and access risk.