AI red teaming is becoming core to AI risk management

At a glance

What this is: This is an analysis of AI red teaming tools and why adversarial testing is becoming part of core AI risk management as production systems face prompt injection, jailbreaks, data poisoning, and agent misuse.

Why it matters: It matters because IAM, PAM, and governance teams now need to treat AI systems as controlled production surfaces, not just models to test once, and connect findings to runtime policy, monitoring, and enforcement.

Context

AI red teaming is the practice of simulating adversarial behavior against AI models, applications, and agents to expose weaknesses before they are exploited in production. In this article, the governance gap is not testing itself but the gap between test results and runtime control.

For identity and access teams, the key issue is that AI systems now make decisions, invoke tools, and move data through production workflows that need governable access boundaries. That turns red teaming into part of AI risk management, not a standalone security exercise.

The article also frames the market around whether organisations need point tools for adversarial simulation or broader platforms that connect testing to monitoring, policy enforcement, and compliance evidence. That distinction is becoming material for enterprise AI programmes.

Key questions

Q: What breaks when AI red teaming stops at a report?

A: If red teaming ends with a report, the organisation has evidence of weakness but no guarantee that the weakness is removed or contained in production. The gap is between discovery and enforcement. Effective programmes connect test results to runtime policy, monitoring, and remediation so findings change live behavior instead of sitting in a document.

Q: When should organisations prioritize runtime controls over more scanning?

A: Organisations should prioritize runtime controls when data changes hands frequently, when AI agents or automations touch sensitive content, or when third-party handoffs are common. In those cases, the highest risk is not unscanned storage. It is uncontrolled movement by identities that already have legitimate access.

Q: What are the signs that AI-powered red teaming is not providing trustworthy results?

A: Warning signs include outputs that are hard to justify, inconsistent attack paths across similar conditions, overreliance on opaque logic, and findings that do not map to real exposure. If teams cannot trace why an attack path was selected or validate the result against actual conditions, the red team output should be treated as advisory, not authoritative.

Q: Should AI red teaming be governed like application security or like identity control?

A: It should be governed as both, because AI red teaming tests behavior, but the risk is often in what the system can access or execute. Application security checks exposure, while identity control defines authority. In AI systems, those controls overlap, so practitioners need one governance model that covers both.

Technical breakdown

How AI red teaming exposes prompt injection and jailbreak paths

AI red teaming uses adversarial inputs to probe how models, applications, and agents respond under hostile conditions. Prompt injection tries to override system instructions, while jailbreaks aim to bypass safety constraints and elicit disallowed behavior. The important technical point is that these tests are not just about the model’s output quality. They reveal whether instruction hierarchy, context handling, and tool-calling boundaries hold under adversarial pressure. In agentic environments, a successful attack can move from bad output to unsafe action if the agent can call tools or trigger workflows. Practical implication: treat adversarial testing as a control validation step for runtime behavior, not a one-time content test.

Practical implication: validate whether instruction boundaries and tool permissions still hold under adversarial prompts before agents reach production.

Why agentic AI support changes the red teaming problem

Agentic AI adds a different attack surface because the system can select tools, chain actions, and operate across multiple steps. That means a red team is no longer only testing model responses but also testing the decision path that leads from a prompt to an external action. Tool misuse, unsafe delegation, and chained workflows become governance problems because the harmful outcome may be produced by legitimate capabilities used in the wrong sequence. This is where identity and authorization matter: the system must know what the agent is allowed to do, when, and under what policy. Practical implication: assess agentic systems for tool authorization, action scope, and escalation paths, not just prompt safety.

Practical implication: define and test tool authorization boundaries for agents before granting them production access to actions or data.

Why red teaming alone does not close AI risk

Red teaming finds weaknesses, but it does not itself enforce protection after the test ends. That is why the article draws a clear line between simulation and operational control. Enterprise AI risk management needs monitoring, policy enforcement, and auditability so findings translate into reduced exposure in live environments. Without that bridge, organisations end up with reports that document failure modes but do not change system behavior. This is particularly relevant where AI touches sensitive data, regulated workflows, or agent-driven operations. Practical implication: evaluate whether test findings can drive runtime policy, logging, and response workflows before choosing a red teaming approach.

Practical implication: require a path from adversarial findings to policy enforcement, logging, and runtime response in the same operating model.

Threat narrative

Attacker objective: The objective is to make AI systems reveal sensitive information, ignore safety policy, or execute unintended actions in production.

1. Entry occurs when adversarial prompts, poisoned inputs, or crafted instructions reach an AI model, application, or agent and begin shaping its behavior.
2. Credentialed action or tool misuse follows when the system accepts hostile instructions and invokes tools, returns sensitive data, or bypasses intended guardrails.
3. Escalation occurs when an agentic workflow turns a single unsafe response into downstream actions across chained tools, contexts, or models.
4. Impact is the disclosure of data, unsafe actions, or production behavior that violates governance and compliance expectations.

Breaches seen in the wild

• EchoLeak (Microsoft 365 Copilot) 2025: A crafted email could make Microsoft 365 Copilot leak data from its context with no click, a zero-click prompt injection fixed as CVE-2025-32711.
• ForcedLeak (Salesforce Agentforce) 2025: A critical Agentforce flaw let attackers steer an AI agent through a Web-to-Lead form to leak CRM data via an expired allowlisted domain.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI red teaming is no longer a specialist test activity, it is a governance control for production AI. The article shows that prompt injection, jailbreaks, data poisoning, and agent misuse are already operational risks, which means testing now sits inside the AI risk management lifecycle. The field is moving from proving that models can fail to proving that organisations can govern failure modes after discovery. For practitioners, red teaming is becoming evidence for control design, not just a penetration exercise.

Runtime enforcement, not test coverage alone, is the real maturity signal. A red team finding only matters when it changes policy, visibility, or response behavior in production. That puts the burden on control planes that can apply intent-based policy, logging, and action boundaries across models and agents. The practitioner conclusion is simple: if findings stop at a report, governance is incomplete.

AI agents turn access control into behavior control. Traditional IAM assumes access can be granted, observed, and later reviewed. That assumption weakens when the system can select tools and execute actions dynamically through a session. The implication is that AI governance must move from static permission assignment to runtime authorization and action-scoped policy enforcement.

Adversarial breadth now matters as much as attack depth. Teams evaluating AI red teaming need coverage that includes agents, multimodal inputs, CI/CD integration, and reporting that can support compliance evidence. Narrow jailbreak checks miss the operational reality of production AI, where model, application, and workflow boundaries all matter. Practitioners should measure whether a platform can validate the whole AI path, not just the model prompt.

AI security and identity governance are converging around one question: who or what is allowed to act. As AI systems move deeper into enterprise workflows, the decisive control is no longer only content safety. It is whether a model, application, or agent can be constrained to approved actions, approved data, and approved timing. That is the same governance problem identity teams already manage, only now with machine decisioning in the loop.

What this signals

Runtime AI governance is becoming the differentiator. Enterprises will increasingly judge AI security programmes by whether adversarial findings change production behavior, not by how many tests were run. That pushes governance teams toward controls that can enforce policy at the point of action, especially where agents can invoke tools or move data.

AI red teaming also exposes a familiar identity lesson: access without runtime constraint is not governance. Once an AI system can act inside business workflows, teams need to ask what it is permitted to do, when it is permitted to do it, and how those permissions are enforced after deployment.

For practitioners

• Map red team findings to runtime controls Require every high-severity AI red team finding to map to a live control, such as policy enforcement, tool restriction, or logging, before the issue is considered closed.
• Test agent tool authorization paths Include tool-calling chains, delegated actions, and multi-step workflows in scope so the assessment covers what the agent can actually do, not only what it says.
• Evaluate reporting for audit readiness Check whether findings can be exported into compliance evidence, risk registers, and remediation workflows without manual reformatting or separate documentation work.
• Differentiate model testing from production governance Separate adversarial simulation from ongoing monitoring, because a system that passes a test can still drift or be abused once it is live.

Key takeaways

• AI red teaming is moving into the center of AI risk management because adversarial techniques already reach production systems and agentic workflows.
• The limiting factor is not finding weaknesses, but translating findings into runtime control, policy enforcement, and governance evidence.
• Practitioners should assess red teaming programmes by whether they reduce live risk, not only by whether they surface attack paths.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Agentic AI: Autonomous AI systems capable of planning, deciding, and taking actions, including calling APIs, writing code, and orchestrating other agents, with minimal human oversight. Agentic AI introduces new NHI risks as agents must authenticate to external services.
• Runtime Protection: Runtime protection is a control model that observes application behavior while software is running and blocks unsafe actions as they occur. In Java estates, it helps distinguish active exploit paths from dormant vulnerable code, which is essential when patching is delayed or impossible.
• Intent-Based Policy: Intent-based policy evaluates why an AI interaction is happening, not just what text it contains. In regulated environments this matters because the same data can be benign or risky depending on purpose, context, role, and downstream action, making intent a core control variable.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.