Agentic red teaming shows why AI agent governance needs runtime testing

At a glance

What this is: Akto shows that agentic red teaming has to probe AI agents at runtime across tools, memory, and workflows because static prompt testing misses the way autonomous systems fail in chains.

Why it matters: This matters because IAM-style assumptions about bounded, reviewable access break down when agents can act, persist context, and misuse tools across multiple steps and sessions.

Context

Agentic AI changes the security problem because the system no longer stops at model output. Once an AI system can browse, call APIs, write to databases, or coordinate with other agents, the control question becomes how it behaves under adversarial conditions while actually executing work. That is a governance issue as much as a testing issue, and it touches identity because the agent inherits permissions, tool access, and trust relationships that must be constrained.

Traditional AI testing focuses on prompts and outputs, but that approach assumes a largely deterministic system. Agentic systems are dynamic, stateful, and dependent on live tools and memory, which means the risk surface changes every time a workflow, data source, or permission set changes. The article's starting position is typical of real agent deployments: capability arrives faster than runtime governance.

Agentic red teaming is the practice of adversarially testing those moving parts before attackers do. It is especially relevant for identity and access programmes because the agent is not just a model, but a decision-making system operating through non-human permissions, inherited entitlements, and delegated trust.

Key questions

Q: What breaks when client-facing AI is only tested with single prompts?

A: Single-prompt tests miss behavioural drift, which is when a model stays compliant at the start of a conversation but becomes unsafe after several exchanges. That matters in regulated workflows because the model can gradually weaken disclaimers, overstep its role, or produce transactional language that would not appear in an isolated test.

Q: Why do autonomous AI agents create more access risk than task bots?

A: Autonomous AI agents create more access risk because they can combine persistence, reactivity, and initiative. A task bot usually executes a narrow command, but an agent can preserve context, request more access, and act on changing conditions. That wider behavioural range increases the chance of overreach, unintended data exposure, and difficult-to-trace actions.

Q: What are the signs that an AI agent workflow is failing governance or operating outside its intended scope?

A: Common warning signs include agents issuing queries that touch data they do not need, calling tools outside the approved workflow, producing outputs that do not match the user request, or making repeated autonomous actions without review. Missing traces, weak audit logs, and unclear ownership are also red flags. If teams cannot explain what the agent accessed and why, governance is already failing.

Q: How should security teams respond when an agent's tool or memory chain is compromised?

A: Contain the agent's delegated access, invalidate the affected workflow path, and rerun adversarial tests against the exact sequence that succeeded. Then review whether the compromise came from indirect prompt injection, over-broad tool permissions, or persistent memory. The remediation target is the chain of trust, not just the bad input.

Technical breakdown

Why static prompt tests miss agentic risk

Static prompt tests evaluate isolated model responses, but agentic systems behave across a full session with tools, memory, and external dependencies in the loop. That means a harmless-looking prompt can become dangerous only after a tool response, a memory lookup, or a second turn changes the context. The architectural issue is state and sequence, not just content. If you only test the model, you miss the trust boundary between the model and the runtime environment that turns one decision into a chain of actions.

Practical implication: test the full agent session, not just single prompts or model outputs.

How tool access turns an agent into a delegated identity

When an agent can call APIs, browsers, plugins, or internal services, it inherits the permissions attached to those tools and the identity under which it operates. That creates a delegated-access problem: the attacker does not need to break the tool itself if they can steer the agent into using it wrongly. This is where prompt injection and tool abuse converge. From an identity perspective, the critical question is whether the agent's access is scoped, observable, and revocable at runtime, not merely approved at deployment.

Practical implication: scope each agent's tool entitlements and treat them as governed NHI-style access.

Why multi-step workflows create chained failure modes

Agentic attacks often work because no single step looks malicious. An attacker can influence retrieval, steer a tool call, and trigger an irreversible action only after the agent has combined those steps into one workflow. Memory persistence makes this worse because instructions can survive across sessions and reappear later in a different context. That is why runtime adversarial simulation matters: it reveals whether the workflow can be manipulated gradually, not just whether one instruction is obviously hostile.

Practical implication: red team the workflow chain and memory state, not just the first interaction.

Threat narrative

Attacker objective: The attacker wants to manipulate the agent into misusing its delegated access so the system itself performs the harmful action.

1. Entry occurs when an attacker plants malicious instructions directly in a prompt or indirectly in retrieved content, web pages, emails, documents, or tool responses.
2. Credential or control abuse follows when the agent trusts that content and uses its own tool access, API permissions, or plugin rights to act on the injected instruction.
3. Escalation happens across a multi-turn or multi-session workflow when the agent carries forward manipulated goals or memory state and executes a harmful sequence of actions.
4. Impact is data exposure, unauthorized system action, or irreversible operational harm caused by a trusted AI system acting on attacker-shaped instructions.

Breaches seen in the wild

• Gemini CLI prompt injection flaw 2025: Tracebit showed a poisoned README could make Gemini CLI run hidden commands and exfiltrate developer secrets; Google fixed it in 0.1.14.
• Gemini AI Breach — Google Calendar Prompt Injection: Gemini AI assistant prompt injection attack leaks sensitive Google Calendar data.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Runtime testing is now the control plane for agent governance: static approval models cannot keep pace with agents that change behaviour as tools, prompts, and memory evolve. Security teams need to treat runtime adversarial simulation as the place where trust is actually validated, not assumed. That shifts governance from design-time paperwork to live behavioural assurance, which is the only credible posture for autonomous workflows.

Agentic AI creates a delegated-identity problem, not just a model-risk problem: once an agent can browse, query databases, or act through plugins, it operates through non-human permissions that need scope, ownership, and revocation. The identity bridge is explicit here: if the agent's access is broader than a human operator would receive, the blast radius grows faster than conventional access review can contain. Practitioners should govern agent entitlements as production identities.

Multi-step manipulation is the named failure mode this topic exposes: the weakness is not a single malicious prompt, but the assumption that each step can be assessed in isolation. Agentic red teaming shows how benign intermediate actions can compose into a harmful outcome, which means governance has to account for chain integrity, state persistence, and tool trust. The practitioner conclusion is straightforward: test the chain, not the component.

MCP and similar tool layers widen the trust boundary faster than most teams realise: once external data sources and tool servers can influence the agent, they become part of the security perimeter. That perimeter is dynamic, because every added tool changes the attack surface and every new workflow changes the reachable actions. Teams that do not continuously retest after tool changes are governing yesterday's system.

Agentic red teaming should feed policy, not sit in a security silo: findings from runtime simulation need to drive access scoping, tool allowlisting, logging, and escalation thresholds across the programme. Otherwise, testing becomes theatre and the organisation keeps deploying agents that have not been challenged under realistic adversarial conditions. The field is moving toward continuous, evidence-based governance, and the control point is runtime.

From our research library:

• Gartner predicts that AI systems will initiate 50% of all service requests by 2030, driven largely by agentic AI.
• Read next: Agentic AI Security Guide

What this signals

Multi-step manipulation is the control gap that matters most: once an attacker can influence one turn, the security problem is whether that influence survives long enough to change later tool calls, memory, or downstream decisions. Agent governance has to move upstream to issuance, authorization, and runtime monitoring, because post-hoc review cannot reconstruct every chained choice.

AI agents need identity controls designed for non-human execution: the practical failure is granting broader access to an agent than the same task would justify for a human operator. According to the 2026 Infrastructure Identity Survey, 70% of organisations grant AI systems more access than they would give a human employee performing the exact same job. That gap is where blast radius grows.

Runtime testing will become the default assurance model for agentic systems: teams that keep relying on static reviews will miss the failures that only appear when live tools, live data, and live state interact. Security programmes should expect red-team findings to drive policy updates, not just test-case libraries.

For practitioners

• Map every agent tool and decision path Inventory browsers, APIs, plugins, databases, and any downstream agents the system can reach. Treat each reachable action as part of the attack surface and assign ownership for the identity under which it executes.
• Run runtime adversarial simulations before production Test prompt injection, indirect prompt injection, tool abuse, and chained workflows in staging and live-like environments. Re-run the suite whenever prompts, tools, or data sources change.
• Scope delegated permissions to the minimum viable task Limit agent entitlements to the narrowest actions needed for the task, then separate high-risk actions from low-risk ones. Revisit whether the agent should hold persistent access at all.
• Instrument memory and context for security review Log when the agent retains instructions, reuses prior context, or carries state between sessions. Use those logs to spot injected content that survives long enough to affect a later decision.
• Turn red-team findings into regression tests Convert every validated exploit path into a recurring test case so the same manipulation cannot reappear after a prompt, tool, or model update. Keep the test library tied to agent ownership and remediation deadlines.

Key takeaways

• AI agent governance fails when teams treat model testing as sufficient and ignore the runtime behaviour created by tools, memory, and chained decisions.
• Agentic red teaming is most valuable because it exposes manipulation paths that only appear when autonomous systems act across multiple steps and sessions.
• Practitioners need to manage agent permissions, workflow state, and continuous simulation together, or the control surface will stay behind the attack surface.

Key terms

• Agentic Red Teaming: Agentic red teaming is the practice of testing AI systems through their real runtime paths, including tools, memory, UI rendering, and downstream workflows. It evaluates how an agent behaves in production, not just how a model responds to prompts, and it should surface actionable exploit chains, not isolated prompt failures.
• Adversary Simulation: Adversary simulation is the practice of testing controls by recreating realistic attacker behaviour. For identity security, it exposes whether compromised credentials, delegated access, or excessive privilege can be used to move through systems before defenders detect the abuse.
• Indirect Prompt Injection: Indirect prompt injection is an attack where malicious instructions are hidden inside content that an AI system reads later. The model may treat that content as context rather than as hostile input, which can influence tool use, data access, or workflow actions if controls are weak.
• Delegated Access: Delegated access is permission granted to one identity to act on behalf of another user, service, or system. In NHI environments, this usually appears in OAuth-connected apps and automation tooling. It is powerful, but it must be tightly scoped and reviewed because it can persist long after the original business need ends.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Build a stronger governance baseline if your programme now includes autonomous systems, delegated access, or machine identities.