AI red teaming is becoming the control plane for agentic AI security

At a glance

What this is: This article argues that AI red teaming has become a core control for finding behavioral failures in non-deterministic AI systems before they become operational incidents.

Why it matters: It matters to IAM, PAM, and security teams because AI agents increasingly act with delegated access, so testing their boundaries is now part of identity and access governance, not just AI assurance.

By the numbers:

• 88% of organisations now use AI in at least one business function, up from 78% a year earlier.
• 62% are already experimenting with autonomous AI agents, and nearly a quarter have begun scaling agentic systems in production.
• AI incidents are rising sharply, increasing by roughly 50% in just 6 months according to OECD tracking.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope.

👉 Read LEVO's analysis of AI red teaming for autonomous AI security

Context

AI red teaming is a way to test whether AI systems can be manipulated into unsafe behaviour before they are trusted in production. The governance gap is that non-deterministic systems do not fail like ordinary applications, so traditional testing and access review methods often miss the conditions that trigger harmful outcomes. In identity terms, this becomes a delegated-access problem once copilots or agents can use tools, data, and credentials on behalf of users.

The article’s core point is that scaling AI without adversarial testing creates a control deficit, not just a model-risk issue. That is why the identity angle matters: when an AI system can access files, APIs, or workflows, red teaming becomes a way to validate whether those privileges stay within intent. The starting position described here is increasingly typical for enterprise AI programmes, not an edge case.

Key questions

Q: What breaks when AI agents are not red teamed before deployment?

A: Without adversarial testing, teams discover prompt injection, tool misuse, and data leakage only after the system is already trusted with real access. The failure is not usually a software crash. It is an apparently normal AI action that crosses a policy boundary, exposes information, or triggers an unsafe workflow. That is why pre-deployment red teaming is a governance control, not a nice-to-have test.

Q: Why do delegated credentials increase risk when AI agents and users are not clearly separated?

A: Delegated credentials can blur the boundary between user intent and agent autonomy. If the agent inherits the user’s identity, it may execute actions that are technically authorized but operationally unsafe, such as deleting production resources or exposing sensitive data. Clear identity separation allows additional policy checks, runtime controls, and confirmation steps before high-risk actions proceed.

Q: How do security teams know if AI red teaming is working?

A: AI red teaming is working when testing finds real prompt injection paths, over-scoped integrations, and policy gaps before attackers do, and when fixes are re-tested successfully after model or workflow changes. The strongest signal is repeatable reduction in exposed authority, not a lower number of red-team findings on its own.

Q: How should security teams govern AI agents that can access enterprise systems?

A: Security teams should govern AI agents as non-human identities with explicit ownership, scoped privileges, and continuous monitoring. The control set should include inventory, task-bound credentials, audit trails, and revocation paths. If an agent can call tools or touch production systems, it belongs in the same governance model as service accounts and other machine identities.

Technical breakdown

How AI red teaming finds prompt injection and instruction hijacking

AI red teaming deliberately presents malicious or misleading prompts, indirect instructions, and context poisoning to see whether a model or agent follows attacker intent instead of system intent. Unlike conventional application testing, the failure is often behavioural rather than a crash or obvious error. The system may still produce a polished answer while silently obeying a hidden instruction, leaking data, or selecting the wrong tool. That makes transcript review, repeated runs, and scenario variation essential. In multi-step workflows, the risk compounds because one compromised input can shape later reasoning, tool calls, and downstream outputs.

Practical implication: test prompts, memories, and tool instructions as a single attack surface, not as separate control domains.

Why tool misuse and delegated access change the threat model

Once an AI agent can call APIs, access documents, or execute actions, it effectively becomes a delegated identity with runtime privileges. The important question is no longer whether the model is accurate, but whether it can be coerced into using legitimate access for illegitimate purposes. Red teaming therefore probes privilege boundaries, authorization checks, and whether sensitive actions require meaningful confirmation. This is where AI security intersects with IAM and PAM. The issue is not just model safety, but whether the surrounding access model assumes the agent will always act in good faith, which attackers are trying to disprove.

Practical implication: treat agent permissions as privileged access and validate the approval path for every high-risk action.

Why continuous testing beats one-time validation for non-deterministic systems

AI systems change frequently through prompt updates, model swaps, retrieval changes, tool additions, and policy edits. Because outcomes are non-deterministic, a test that passes once may fail later under slightly different wording, context, or data. That is why the article frames red teaming as a continuous control rather than a pre-launch event. Continuous testing is also the only practical way to catch drift in multi-agent workflows, where one component change can alter the full behaviour chain. For governance teams, the real issue is evidence of ongoing control, not a single compliance checkpoint.

Practical implication: build red teaming into release, change, and revalidation cycles so security keeps pace with agent drift.

Threat narrative

Attacker objective: The attacker wants to turn a trusted AI workflow into a channel for data theft, policy bypass, or unauthorised action without needing to break the underlying infrastructure.

1. Entry occurs when a malicious prompt, email, or indirect instruction is introduced into an AI workflow and the system accepts it as valid context.
2. Escalation follows when the agent applies that instruction to its toolchain, data sources, or delegated permissions instead of treating it as untrusted input.
3. Impact is reached when the agent leaks data, alters decisions, or performs unsafe actions that look legitimate because they were executed through authorised systems.

NHI Mgmt Group analysis

AI red teaming is now an identity control problem as much as an AI assurance problem. When an agent can read data, call tools, or trigger workflows, it functions as a delegated non-human identity with runtime authority. That changes the governance question from model quality to privilege containment, approval boundaries, and post-issuance monitoring. The practical conclusion is that AI red team results should inform IAM and PAM decisions, not sit only in an AI risk register.

Instruction hijacking exposes a verification trust gap in enterprise AI programmes. Organisations often assume that a system which passes initial validation will remain safe under new prompts and new context, but adversarial testing shows that assumption is fragile. A model can appear stable while still being one prompt away from disclosing data or misusing a tool. The field needs to treat behavioural verification as an ongoing control objective, not a one-off assurance exercise.

Prompt injection and tool misuse should be treated as a shared control domain across security, privacy, and governance. The article shows that failures happen across model logic, retrieval, orchestration, and downstream systems. That means AI governance cannot be isolated from access governance, logging, and incident response. The practitioner takeaway is to define ownership for adversarial testing, remediation, and retesting before agent deployments scale.

Continuous red teaming is the named concept this article makes unavoidable. It is the practice of repeatedly attacking AI systems as they evolve, rather than validating them once at launch. That matters because model updates, tool expansion, and retrieval changes can reopen previously closed pathways. The conclusion for practitioners is simple: if the system can change, the testing regime must change with it.

AI agent autonomy increases the blast radius of weak access design. The article’s examples show that unsafe behaviour becomes materially worse when an agent can reach sensitive systems or data through live credentials. That aligns with OWASP NHI and NIST AI RMF concerns about control boundaries, traceability, and monitored authority. The field should treat delegated access to agents as a governed privilege, not a convenience feature.

What this signals

Continuous red teaming is becoming a baseline control for AI programmes that expose tools, memory, or retrieval layers. As systems evolve after launch, assurance cannot remain point-in-time. The practical shift is toward lifecycle testing that links model changes, prompt changes, and connector changes to revalidation before new authority is granted.

AI governance teams should expect red teaming evidence to become part of broader security governance packs. Boards and regulators are increasingly asking not only whether AI was tested, but whether failures were remediated and re-tested. That pushes AI security toward the same evidentiary expectations that already shape IAM, PAM, and incident response programmes.

Delegated AI access is now a measurable control boundary, not a theoretical one. When an agent can access files, systems, or workflows, the control question becomes whether that access is observable, bounded, and revocable. Teams should align agent review processes with the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10.

For practitioners

• Define red team scope around delegated AI privileges Map every tool, API, connector, and data source an AI system can reach, then test whether malicious prompts can drive those privileges beyond intended use. Prioritise high-impact workflows where a single bad action could expose data or trigger transactions.
• Test for indirect prompt injection in real workflows Use emails, documents, retrieved content, and chat messages as attack carriers, then verify whether the agent follows hidden instructions embedded in ordinary business content. Include multi-step scenarios where one poisoned input influences later tool calls.
• Treat AI outputs and tool calls as audit evidence Log prompts, retrieved context, model responses, tool actions, and approval events so red team findings can be reproduced and investigated. Preserve transcripts in a form that supports root-cause analysis and control retesting.
• Tie red team findings to IAM and PAM controls Require remediation when an agent can reach sensitive data without adequate approval, step-up validation, or least-privilege scoping. Where the agent acts as a delegated identity, validate that high-risk actions cannot proceed on standing access alone.

Key takeaways

• AI red teaming is a behavioural control, not a model-quality exercise, because the failures that matter are unsafe actions, data exposure, and tool misuse.
• The scale of enterprise AI adoption means adversarial testing is now a governance requirement, especially where agents can act with delegated access.
• Practitioners should connect red team findings to IAM, PAM, logging, and retesting so AI authority stays bounded as systems change.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Governed AI access: Governed AI access is the approved use of AI services through defined identities, policy, and logging. It gives security and compliance teams a reviewable path for who may use which tools, what data they may submit, and how the resulting interactions are retained and monitored.
• Non-Deterministic Behaviour: Non-deterministic behaviour is software behaviour that does not produce the same outcome every time under similar inputs. In AI systems, this makes traditional testing and code review incomplete unless teams also capture and analyse production outcomes.

What's in the full article

LEVO's full article covers the operational detail this post intentionally leaves for the source:

• Detailed breakdown of red teaming methods for prompt injection, jailbreaks, and indirect instruction attacks
• Examples of multi-step testing against tool-using agents, including data leakage and unsafe action scenarios
• Operational challenges in continuous testing for non-deterministic and multi-agent workflows
• LEVO's view of how monitoring and runtime enforcement complement red teaming across the AI lifecycle

👉 LEVO's full article covers testing methods, operational challenges, and the role of continuous AI security controls.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management in modern enterprise environments. It helps security practitioners connect delegated access, runtime authority, and lifecycle controls across identity programmes.