Agent-led AI red teaming exposes how autonomous tests break controls

At a glance

What this is: TrojAI argues that agent-led AI red teaming replaces static prompt checks with coordinated autonomous testing that can follow multi-turn attack chains and map findings to major security frameworks.

Why it matters: This matters because practitioners now need testing methods that reflect how autonomous AI systems actually fail, especially when controls must support governance, prioritisation, and remediation across AI agents, applications, and models.

Context

AI security testing is moving beyond one-shot prompt probes toward coordinated attack simulations that can adapt during execution. In this article, the vendor frames that shift around agent-led red teaming for AI agents, applications, and models, with results mapped to established governance frameworks.

The governance gap is straightforward: static scripts can expose obvious flaws, but they struggle to represent the interaction patterns, backtracking, and escalation paths that emerge in live AI systems. For identity and access teams, that means AI testing now sits closer to runtime control validation than to traditional app security sampling.

Key questions

Q: How should security teams test AI systems for multi-turn failure paths?

A: Security teams should test AI systems with scenarios that evolve across several exchanges, not just single prompts. Multi-turn paths reveal conditioning, context drift, tool abuse, and gradual boundary failure that one-shot checks miss. The goal is to observe how the system behaves once trust, memory, and escalation are in play, then map those failures to a remediation owner.

Q: Why do AI red-team findings need framework mapping?

A: Framework mapping gives AI red-team results a common language for prioritisation and accountability. Without it, findings are just technical observations that are hard to compare, budget, or track. Mapping to OWASP, MITRE, or NIST helps security, compliance, and leadership understand which controls failed and what action needs to follow.

Q: What are the signs that an AI system is failing under adaptive testing?

A: Common signs include refusal patterns changing over time, safety boundaries weakening after repeated context, and different attack variants producing inconsistent outcomes. If the model only looks safe in the first exchange, the test is incomplete. Adaptive testing should expose whether control strength degrades when the conversation lengthens or the tactic changes.

Q: How do AI agents change the way red teams should work?

A: AI agents let red teams simulate coordinated, adaptive adversaries at a scale that manual scripts struggle to match. That matters because the testing process itself becomes more realistic, with strategy selection, backtracking, and parallel execution. Teams should use that capability to validate the system’s resilience under realistic interaction pressure, not to replace human judgment.

Technical breakdown

How coordinated agents change AI red teaming

Coordinated red teaming uses multiple test agents with different roles, rather than a single scripted probe, to explore how an AI system behaves under pressure. One agent can select strategy, another can execute a turn, and another can evaluate whether the target resisted, complied, or drifted. That structure matters because modern AI failures are often conversational and cumulative, not immediate. The article’s architecture shows how orchestration, evaluation, and adaptive feedback can be layered into one assessment flow so the test evolves as the target evolves. That is materially different from static prompt fuzzing, which assumes the first response tells the whole story.

Practical implication: Treat AI red teaming as a multi-turn control test, not a one-off prompt check.

Why multi-turn attack chains reveal different failure modes

Multi-turn attacks matter because many AI weaknesses only emerge after context has been built, trust has been established, or the system has been nudged into a different operating state. A single prompt may miss prompt-injection variants, tool misuse, or gradual scope drift that appears after repeated exchanges. In the article, the vendor’s dynamic attack chains are designed to mimic that progression by changing tactics during the test. For practitioners, the point is not just coverage. It is whether the AI system can hold its boundary when an interaction becomes longer, noisier, and more adversarial.

Practical implication: Include long-form, adaptive scenarios in AI assurance plans, not just single-turn checks.

Why framework mapping matters for AI governance

Framework mapping turns test output into something governance teams can use. Raw findings are hard to prioritise because they are often technical, repetitive, or isolated from risk language. Mapping the same finding to OWASP, MITRE, and NIST creates a shared reference point for security engineering, compliance, and leadership. That does not make the risk smaller, but it makes it governable. For AI programmes, the value is that red-team evidence can feed control discussions instead of sitting in a specialist report with no decision path.

Practical implication: Use framework-mapped findings to connect AI testing results to governance, risk, and remediation workflows.

Threat narrative

Attacker objective: The objective is to prove how an AI system can be driven into unsafe, unreliable, or governability-breaking behaviour through adaptive interaction.

1. Entry begins when the target AI system is engaged through prompt content, tool context, or conversational setup that the red team wants to stress.
2. Escalation occurs as coordinated agents change tactics mid-test, build memory, and push the model into more permissive or confused states.
3. Impact is the exposure of design flaws, unsafe behaviours, or boundary failures that would be difficult to surface with single-step testing.

Breaches seen in the wild

• CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.
• Langflow Flodrix botnet 2025: Attackers used Langflow's unauthenticated code endpoint to dump AI servers' environment variables and enlist them in the Flodrix botnet.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Agent-led red teaming is becoming the closest thing to runtime assurance for AI systems: static evaluation cannot represent the conversational, adaptive, and chained behaviours now seen in production AI. When a system can change state across turns, the meaningful control question is whether the test itself can adapt with it. Practitioners should treat dynamic red teaming as a core validation layer, not a specialist add-on.

Multi-turn interaction is the real fault line in AI security: many controls are still built around the assumption that one prompt or one request can reveal a weakness. That assumption breaks when a target can be conditioned, redirected, or gradually destabilised over several exchanges. The implication is that AI governance has to judge boundary durability, not just initial refusal behaviour.

Framework mapping is what turns AI findings into governance evidence: security teams do not need more isolated findings, they need findings that can be ranked, assigned, and tracked through existing control language. OWASP, MITRE, and NIST mapping gives AI red-team output a decision structure that boards and operators can act on. The practitioner conclusion is clear: if findings cannot be mapped, they are much less likely to change behaviour.

Autonomous test agents create a new assurance problem for AI programmes: the testing system is no longer a passive script, it is an active participant that selects strategies, backtracks, and learns during the run. That changes the assurance model from deterministic validation to adaptive control testing. Security leaders should expect red teaming itself to become part of the control surface for AI governance.

Adaptive red teaming exposes the boundary between AI assistance and AI autonomy: once a test agent can retain memory, revise tactics, and coordinate with other agents, the assessment begins to resemble the behaviour of the systems it is testing. That is useful because it reveals where AI programmes lack durable control boundaries. The implication is that identity, authorisation, and safety review for AI must be designed for interaction, not just deployment.

From our research library:

• According to PwC, 79% of organizations have already adopted AI agents to some degree.
• Read next: Red Teaming AI Agents for Identity Abuse

What this signals

Adaptive AI testing will become a governance expectation, not a specialist curiosity: programmes that only validate first-response behaviour will miss the failures that emerge after memory, context, and repeated interaction accumulate. The practical shift is toward assurance that can follow the system through the full conversation, not just the first turn.

AI security teams should expect their control evidence to become more narrative-driven: isolated alerts are less useful when a model’s real weakness only appears through a chain of exchanges. A red-team workflow that preserves sequence, context, and outcome will be easier to defend in risk reviews and post-incident analysis.

According to the 2026 Infrastructure Identity Survey, 79% of organizations have already adopted AI agents to some degree, so testing discipline now needs to scale with usage rather than trail it. That makes adaptive red teaming a programme design issue, not a lab exercise.

For practitioners

• Test for multi-turn failure paths Build red-team scenarios that extend beyond one prompt so you can observe how an AI system behaves after context accumulates and tactics shift.
• Map findings to control language Require each AI test result to map to a recognised framework label so governance, engineering, and compliance teams can triage the same evidence.
• Evaluate adaptive behaviour under pressure Include backtracking, role changes, and dynamic prompts in assessment runs so the test can expose boundary drift rather than only obvious refusals.
• Correlate results into one report Consolidate outputs from different test agents into a single remediation view so duplicated findings do not fragment ownership or delay fixes.

Key takeaways

• Agent-led AI red teaming matters because AI failures often appear only after context, memory, and strategy shifts accumulate across multiple turns.
• Framework mapping turns red-team output into governance evidence that security, compliance, and leadership can actually use.
• Adaptive testing gives practitioners a better way to see boundary drift, unsafe tool use, and other failures that static scripts miss.

Key terms

• Agentic Red Teaming: Agentic red teaming is the practice of testing AI systems through their real runtime paths, including tools, memory, UI rendering, and downstream workflows. It evaluates how an agent behaves in production, not just how a model responds to prompts, and it should surface actionable exploit chains, not isolated prompt failures.
• Multi-Stage Attack Chain: A multi-stage attack chain is a connected sequence of actions that turns individual weaknesses into a complete compromise path. One low-severity issue may not matter alone, but when combined with reusable access, pivot opportunities, and internal trust relationships, it can produce a much more serious breach outcome.
• Framework Mapping: Framework mapping is the process of linking a single control or evidence source to multiple standards or regulations. It helps teams see where requirements overlap, where they diverge, and where extra sector-specific safeguards are still needed to stay compliant.
• Adaptive Testing: A testing approach that adjusts execution, prioritisation, or maintenance based on application change and operational context. It is designed for environments where static scripts break quickly, and where validation must stay aligned with release velocity, infrastructure variability, and production-like conditions.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an identity security programme, it is worth exploring.