Agentic AI red teaming exposes gaps in identity governance

At a glance

What this is: This is a TrojAI webinar-derived blog on agentic AI red teaming that argues single-turn prompt tests miss the real risk because agents chain tools, retain state, and act across workflows.

Why it matters: It matters because IAM, PAM, and identity governance teams must treat agent permissions, tool access, and runtime oversight as a live control plane rather than a static approval exercise.

Context

Agentic AI security is the problem space here, not chatbot prompting. When an AI system can reason, choose tools, and execute tasks across multiple steps, the identity question shifts from who can ask a question to what authority the system can exercise while it works.

Traditional IAM models assume access is granted, observed, and reviewed in discrete states. In agentic workflows, memory, delegated permissions, and tool chaining can change the security posture mid-session, so the governance model has to account for runtime behaviour as well as provisioning.

The article positions this as a red-teaming problem because failures often emerge only after several turns or task transitions. That is typical for emerging agentic AI programmes, where teams are still adapting human-era control patterns to machine-driven execution.

Key questions

Q: What breaks when agentic AI red teaming stays at single prompts?

A: Single-prompt testing misses stateful failures that emerge only after tool calls, memory updates, and follow-on decisions interact. In agentic systems, the risky behaviour often appears in the second or third step, not the first prompt. Teams need scenario-based testing that follows the entire workflow, because the security issue is the sequence, not only the input.

Q: Why does delegated access increase risk in agentic AI workflows?

A: Because an agent with broad or standing access can turn one task into many unintended actions across connected systems. The risk grows when permissions are not tightly tied to a specific workflow, because the agent can act faster than human review cycles and can cross boundaries that were never intended for that task.

Q: What are the signs that an AI agent has gone out of scope?

A: Common signs include attempts to use unapproved tools, unexpected access to production data, spawning additional agents without a clear mandate, and repeated requests that expand beyond the original task. The key indicator is deviation from the declared lane, especially when the action is technically possible but operationally out of policy.

Q: Should organisations treat agentic AI like a normal automation platform?

A: No. Normal automation follows predetermined rules, while agentic AI can reason, sequence actions, and choose tools at runtime. That means governance must focus on delegated authority, state integrity, and action review, not only on the initial workflow design or static policy checks.

Technical breakdown

Why single-turn testing misses agentic AI risk

Single-turn testing evaluates one prompt and one response, which is useful for baseline safety but not for systems that plan, call tools, and carry state forward. Agentic workflows introduce sequence dependence: an apparently safe first step can set up a later action that crosses a permission boundary. That is why multi-turn red teaming is necessary. The control failure is not just model output quality, but the interaction between model reasoning, retained context, and external side effects across time.

Practical implication: test full task sequences, not isolated prompts, when validating agentic AI behaviour.

How delegated authority expands the identity attack surface

Agentic systems need fine-grained delegation because broad standing access lets one task become many unintended ones. In identity terms, the agent is acting as a non-human identity with tool access, and the risk comes from overbroad authorisation rather than only from bad model output. If a workflow has access to HR, finance, or production tools, the agent can amplify a small mistake into a real operational change. The technical problem is authority scope, not just authentication.

Practical implication: bind each agent to narrowly scoped permissions that match the specific workflow and time window.

How memory poisoning and planning manipulation work across steps

Memory poisoning corrupts retained context so later decisions drift toward attacker goals without an obvious single malicious prompt. Planning manipulation changes how the agent sequences actions, which can reroute tool use, alter task order, or trigger unsafe follow-on behaviour. These are multi-step failures, so they are harder to catch with static rules. The important technical point is that the agent’s decision path becomes the attack surface, not just the text it receives.

Practical implication: instrument state transitions, plan changes, and tool-call sequences for anomalous behaviour.

Threat narrative

Attacker objective: The attacker aims to steer an agent into taking harmful actions that look like normal task execution while bypassing human review.

1. Entry begins with a benign-looking multi-turn prompt or trusted tool interaction that gains the agent’s attention and state.
2. Escalation occurs when retained memory or planning logic is influenced so the agent selects a broader or riskier tool path than intended.
3. Impact follows when chained actions cross into production systems, causing unauthorized workflow execution, access misuse, or cascading errors.

Breaches seen in the wild

• Langflow Flodrix botnet 2025: Attackers used Langflow's unauthenticated code endpoint to dump AI servers' environment variables and enlist them in the Flodrix botnet.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Access review logic was designed for stable privilege states, and that assumption breaks under agentic execution. Review models presume access persists long enough to be observed, certified, and revoked in a later cycle. When an AI agent can acquire, use, and move through permissions inside a workflow, the review window no longer matches the control window. The implication is that governance has to be evaluated at issuance and decision time, not only at review time.

Multi-turn red teaming should be treated as an identity governance control, not just a model safety exercise. The article is really describing a runtime trust problem where state, delegation, and tool access interact. That moves the discipline from prompt testing into identity-bound workflow assurance. Practitioners should read this as a signal that agent behaviour and authorisation design are now inseparable.

Fine-grained delegation is the core control concept for agentic AI identity, but it only works if workflow boundaries are explicit. A task-scoped agent with narrow permissions is materially different from a broadly enabled automation layer. The named concept here is runtime delegation gap: the distance between what the agent is authorised to do at provisioning time and what it can actually do once it begins chaining tools and decisions. That gap is where governance fails.

Memory and planning are now part of the security perimeter for AI agents. Traditional IAM does not model retained context or mutable plans as protected assets, yet these are exactly the mechanisms adversaries can influence. This is where OWASP-AGENTIC, NIST AI RMF, and CSA MAESTRO become relevant: they help teams describe the control problem in terms of agent behaviour, not just application access. Practitioners should treat state integrity as a first-class governance concern.

Agentic AI security is converging with NHI governance faster than many programmes realise. Once an agent can call tools and act autonomously, the identity question becomes about non-human authority, not conversational quality. That means existing IAM, PAM, and audit structures will be judged by whether they can constrain machine actions in real time. Teams should assume the governance baseline is now shared across workloads, service identities, and agents.

From our research library:

• 53% of security leaders expect AI to run major portions of their infrastructure autonomously within the next three years, according to the 2026 Infrastructure Identity Survey.
• 67% of organisations still rely heavily on static credentials despite the risks they pose to agentic AI deployments, according to the 2026 Infrastructure Identity Survey.

What this signals

Runtime delegation gap: the control problem is no longer just whether an AI agent was approved, but whether its permissions still match the actions it can chain together after state changes. That forces teams to move governance closer to execution, where tool use and context interact.

Security programmes that already separate identity proofing, authorisation, and audit trails have an advantage here, but only if they extend those controls to machine actions. Agentic systems blur the line between access and execution, so monitoring has to cover the workflow, not just the login or the prompt.

OWASP-AGENTIC and NIST AI RMF are useful here because they push teams to describe risk in terms of agent behaviour, not just model output. That is the right lens for programmes that need to govern tool misuse, memory manipulation, and cascading decisions before they reach production.

For practitioners

• Define task-scoped agent privileges Map each agent to the smallest viable tool set, data scope, and action boundary for one workflow. Do not let a general-purpose agent inherit broad platform permissions just because it can complete multiple tasks.
• Red team complete agent workflows Test benign starts, mid-session state changes, tool chaining, and follow-on actions as one scenario so failures appear in the same sequence users will run in production.
• Instrument state and plan changes Log memory updates, tool selections, and task sequencing so security teams can detect when an agent’s decision path diverges from the approved workflow.
• Add supervisory checks before critical actions Require validation for external side effects such as account changes, ticket closures, or production writes before the agent completes the task.

Key takeaways

• Agentic AI red teaming is really a test of whether identity governance can keep up with runtime delegation, multi-step reasoning, and tool use.
• The article shows that the most dangerous failures appear across workflows, not in isolated prompts, which makes state and sequence part of the attack surface.
• Practitioners should move controls closer to execution, because agent permissions, context integrity, and approval boundaries now fail together.

Key terms

• Agentic AI Red Teaming: Agentic AI red teaming is the practice of testing autonomous AI systems for unsafe behavior, misuse, and control failures. It uses adversarial scenarios to probe how an agent plans, acts, calls tools, handles prompts, and responds to manipulation, with attention to identity, authorization, data exposure, and unintended actions.
• Fine-grained delegation: The practice of giving an identity only the specific rights needed for a narrow task and a limited duration. For agents, this has to be enforced at runtime, because broad or persistent delegation increases the chance that the software will reach beyond the intended workflow.
• Memory Poisoning (ASI06): An attack where malicious content is injected into an AI agent's memory or context, causing it to alter its behaviour in subsequent tasks, potentially exfiltrating secrets, escalating privileges, or acting against its intended purpose.
• Planning Manipulation: Planning manipulation is the alteration of how an AI agent sequences and prioritises actions during task execution. It matters because the control failure is not only the output the model produces, but the route it takes through tools, decisions, and dependencies.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.