AI red teaming is becoming the control plane for agentic AI security

At a glance

What this is: LEVO’s article says AI red teaming is becoming the core control for agentic AI because autonomy creates behavioral failures that traditional security testing misses.

Why it matters: That matters because IAM, NHI, and AI governance teams now have to test how delegated systems behave under attack, not just whether they authenticate and inherit permissions correctly.

By the numbers:

• 88% of organizations now use AI in at least one business function, up from 78% a year earlier.
• 62% are already experimenting with autonomous AI agents, and nearly a quarter have begun scaling agentic systems in production.
• AI incidents are increasing by roughly 50% in just 6 months.
• Early adopters report 40 to 50% productivity improvements in functions like IT and HR.

Context

AI red teaming is a controlled adversarial test of an AI system’s behavior. It matters because autonomous and semi-autonomous systems do not fail only through code defects; they also fail through manipulated goals, tool abuse, data leakage, and unsafe decisions. In identity terms, that means governance has to follow the delegated authority given to AI systems, not just the model prompt or interface.

As enterprises move from copilots to agents that can plan and act, the security problem shifts from model quality to control of execution. Traditional access reviews, static testing, and one-time validation do not capture how an AI system behaves when context changes or malicious inputs reshape its decisions. That makes red teaming relevant to AI governance, NHI controls, and broader security assurance.

LEVO frames the current adoption curve as typical of an early scaling phase, not an edge case. The article’s examples show that the failure mode is already operational, which is why adversarial testing is becoming part of the control stack rather than a post-hoc assurance activity.

Key questions

Q: What breaks when AI agents are not red teamed before deployment?

A: Without adversarial testing, teams discover prompt injection, tool misuse, and data leakage only after the system is already trusted with real access. The failure is not usually a software crash. It is an apparently normal AI action that crosses a policy boundary, exposes information, or triggers an unsafe workflow. That is why pre-deployment red teaming is a governance control, not a nice-to-have test.

Q: Why do AI agents create more risk than traditional automation?

A: AI agents create more risk because they can interpret context, choose actions, and invoke tools autonomously. Traditional automation follows fixed rules, but an agent can be manipulated into using its own authority in unintended ways. That makes permission scope, tool boundaries, and monitoring more important than model accuracy alone.

Q: How do security teams know if AI red teaming is working?

A: AI red teaming is working when testing finds real prompt injection paths, over-scoped integrations, and policy gaps before attackers do, and when fixes are re-tested successfully after model or workflow changes. The strongest signal is repeatable reduction in exposed authority, not a lower number of red-team findings on its own.

Q: How should security teams govern AI models that can call tools and access data?

A: Security teams should govern AI models as non-human identities with named owners, limited scope, short-lived credentials, and continuous authorization. The critical shift is to treat every tool call, data read, and update path as a privileged action that can be logged, revalidated, and revoked. Without that discipline, model risk becomes identity risk.

Technical breakdown

Why non-deterministic AI systems need adversarial testing

AI systems can produce different outputs from the same input, which means security cannot rely on one-time validation or static test cases. A model or agent may appear safe in normal conditions and still fail under prompt injection, indirect instruction, or adversarial context. That is especially important when the system can call tools, move data, or take actions across workflows. Red teaming creates repeated hostile scenarios so teams can observe how behavior changes under pressure, where guardrails fail, and which conditions trigger unsafe execution. It is less about finding a single bug and more about mapping the system’s failure envelope.

Practical implication: Treat safety testing as repeated behavioral verification, not a one-off launch gate.

How tool access turns AI into an identity and privilege problem

Once an AI system can access APIs, files, tickets, or cloud actions, it is no longer just a model. It is a delegated actor operating through credentials, permissions, and workflow context. That makes identity and privilege the control surface, not only prompt content. Red teams therefore test whether an agent can be induced to exceed its intended scope, misuse tools, or leak data through legitimate interfaces. In practice, this is where AI security intersects with NHI governance: the system may not have a human account, but it still acts through non-human identity, and the authorization model has to reflect that reality.

Practical implication: Assign and test AI-specific permissions as if the system were a high-risk non-human identity.

What multi-step agent workflows change about attack paths

Single-prompt abuse is only part of the picture. Agentic systems often chain planning, retrieval, memory, tool calls, and downstream actions, which means a safe step can become unsafe once combined with others. A red team therefore looks for emergent failures such as poisoned context, hidden dependency abuse, or unsafe delegation across agents. The security issue is not just whether one component is robust, but whether the entire workflow preserves intent when the system passes state from one step to another. That makes workflow design, monitoring, and containment as important as model alignment.

Practical implication: Test the full agent chain, because control failure often emerges at the handoff points.

Threat narrative

Attacker objective: The attacker’s objective is to coerce the AI system into exposing data, misusing tools, or taking unsafe actions that have real enterprise impact.

1. Entry begins when an attacker uses prompt injection, malicious email content, or other adversarial inputs to influence an AI system’s instructions or context.
2. Credential access or abuse follows when the manipulated system uses legitimate tool permissions, tokens, or delegated access to retrieve data or perform actions it should not take.
3. Escalation occurs as the AI agent propagates the malicious instruction through chained workflows, allowing unsafe decisions, data leakage, or unauthorized changes across connected systems.
4. Impact is operational and governance-level, including leaked confidential data, altered downstream decisions, or production failures caused by an AI system acting beyond intent.

Breaches seen in the wild

• Spain's first AI agent data breach 2026: Spain's AEPD logged its first breach notification attributed to an attacker's AI agent, which altered personal data and accessed invoices.
• CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI red teaming is becoming the practical control plane for agentic AI governance. The central issue is not whether models can answer questions, but whether delegated systems can be trusted to behave under hostile pressure. Once agents can act across tools and data, security must test behavior before production, not merely document it after deployment. The practitioner conclusion is that adversarial testing has to sit alongside authorization and monitoring as a core control.

Agentic AI creates a new kind of identity security problem because the system acts through delegated authority. A prompt is not an identity, but an agent operating through tokens, tools, and workflow permissions behaves like a non-human actor. That means NHI governance, privilege scoping, and offboarding logic all become relevant to AI systems that were never designed as users. The practitioner conclusion is that identity controls must move from account-centric thinking to action-centric control of machine behavior.

AI red teaming exposes the gap between theoretical policy and actual runtime behavior. Many organisations can describe what an agent should never do, but few can prove it will resist prompt injection, tool misuse, or multi-step manipulation in production conditions. That is why the named concept here is behavioural assurance gap: the distance between written guardrails and observed execution. The practitioner conclusion is that assurance needs evidence from attack simulation, not just policy statements.

Continuous red teaming is becoming a lifecycle discipline, not a project milestone. Agent prompts, tools, memory, and integrations change too quickly for point-in-time testing to remain reliable. The article’s regulatory references reinforce that adversarial testing is moving into formal governance expectations for high-risk AI systems. The practitioner conclusion is that security teams should expect repeated retesting whenever an agent’s authority, context, or toolchain changes.

The market signal is clear: AI security is converging on control of delegation, not just model quality. The article’s examples show that the highest-risk failures are operational, where autonomous behavior meets enterprise permissions. That means AI security teams, IAM leaders, and GRC stakeholders now share responsibility for proving that delegated systems stay inside intended boundaries. The practitioner conclusion is to treat agent oversight as a governance function, not a model-only concern.

From our research library:

• 70% of organisations grant AI systems more access than they would give a human employee performing the exact same job, according to the 2026 Infrastructure Identity Survey.
• Only 13% of organisations feel extremely prepared for the reality of agentic AI despite the majority racing toward autonomous adoption, according to the 2026 Infrastructure Identity Survey.
• Read next: Agentic AI Identity Guide

What this signals

Behavioural assurance gap: the most important AI security failure is often the distance between policy and observed runtime behavior. As agents gain tool access, organisations need evidence that controls hold under attack, not just in design reviews. That is why adversarial testing, authorisation, and monitoring have to work as one programme.

The strongest signal for practitioners is that AI governance is becoming lifecycle governance. Every meaningful change to prompts, tools, memory, or delegated authority should trigger new tests, because agent behaviour changes with context. Security teams that keep red teaming separate from operations will miss the point of continuous assurance.

For practitioners

• Define adversarial test cases for delegated AI workflows Cover prompt injection, indirect instruction, tool misuse, data leakage, and unsafe action scenarios across the agent’s real task paths.
• Scope AI permissions to the minimum executable task Limit tokens, API access, and data reach so the agent can complete only the intended workflow and nothing adjacent.
• Retest agents after every material change Re-run red team scenarios whenever prompts, tools, memory, or downstream integrations change, because behavior shifts with context.
• Feed red team findings into runtime controls Use the results to tighten monitoring, escalation rules, and alerting so malicious or unsafe behavior is caught after deployment.

Key takeaways

• AI red teaming is shifting from a specialist check to a core governance control because agentic systems can be manipulated into unsafe actions under realistic pressure.
• The real failure surface is behavioral, not just technical, which is why static scans and one-time validation do not prove that an AI system will stay within intent.
• Practitioners need continuous adversarial testing, scoped delegated access, and runtime monitoring if they want to scale autonomous AI without losing control.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Agentic AI: Autonomous AI systems capable of planning, deciding, and taking actions, including calling APIs, writing code, and orchestrating other agents, with minimal human oversight. Agentic AI introduces new NHI risks as agents must authenticate to external services.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Non-Human Identity (NHI): A digital identity assigned to a non-human entity such as a software application, service account, API key, bot, machine, or AI agent that enables it to authenticate and interact with systems without direct human involvement. NHIs now outnumber human identities in most enterprises by 25 to 50 times.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and secrets management. It helps practitioners connect delegated access, authorization, and lifecycle control across identity and security programmes.