AI red teaming is becoming central to GenAI governance

At a glance

What this is: This is an independent analysis of AI red teaming and its role in exposing GenAI, LLM, and AI system weaknesses before they become operational risks.

Why it matters: It matters because IAM, security, and governance teams need testing approaches that account for AI-specific abuse paths, not just traditional software vulnerabilities.

👉 Read WitnessAI's analysis of AI red teaming for GenAI security and compliance

Context

AI red teaming is structured adversarial testing for AI systems. It probes how models, prompts, APIs, and surrounding workflows behave when a malicious or careless user tries to push them beyond intended boundaries, which makes it relevant to AI governance as well as security assurance.

For identity and access teams, the key issue is that AI systems can expose data, make risky decisions, or amplify misuse even when the underlying infrastructure is hardened. That is why red teaming now sits alongside access control, lifecycle governance, and data protection in modern AI programmes.

Key questions

Q: How should security teams run AI red teaming for GenAI systems?

A: Start with the system’s trust boundaries, then test prompts, retrieval sources, tool calls, and output handling together. Good AI red teaming does not stop at finding bad answers. It checks whether the system can be pushed into revealing data, bypassing policy, or taking unsafe actions through connected interfaces.

Q: Why do AI systems need red teaming beyond traditional penetration testing?

A: Because many AI failures are behavioural rather than exploit-based. A model can be manipulated through prompt injection, poisoned context, or unsafe tool routing without any infrastructure flaw. Traditional penetration testing may miss these runtime trust failures, while red teaming is designed to expose them.

Q: When does AI red teaming become a governance requirement instead of a nice-to-have?

A: It becomes a governance requirement when the AI system handles sensitive data, makes user-facing decisions, or connects to tools that can move or expose information. At that point, red teaming is evidence of control validation, not an optional technical exercise.

Q: How do organisations know if AI red teaming is actually working?

A: Look for repeatable findings, clear ownership, and measurable reductions in the same failure modes after remediation. If the same prompt injection, leakage, or unsafe-action cases keep reappearing, the programme is producing noise rather than control assurance.

Technical breakdown

Prompt injection and jailbreak testing

Prompt injection is an attack pattern where an adversary manipulates an AI system through crafted input so it follows hidden instructions, reveals data, or ignores guardrails. Jailbreaks are similar, but usually aim to bypass content or policy constraints. In LLM environments, these attacks matter because the model may blend user prompts, system instructions, retrieved context, and tool output into a single response path. That makes the failure mode less like a classic exploit and more like trust boundary confusion across the prompt stack.

Practical implication: test where user-controlled input can override policy, context, or tool behaviour before the system is allowed into production.

Data poisoning and privacy leakage in AI systems

Data poisoning alters training or retrieval inputs so the model learns, surfaces, or favours attacker-chosen behaviour. Privacy leakage occurs when a model memorises or reconstructs sensitive data, then reproduces it in output under the right prompt conditions. These are not theoretical issues in GenAI deployments because many systems ingest documents, chats, logs, and knowledge bases from multiple sources. Once those sources are mixed, the risk is no longer just model accuracy. It becomes a governance problem involving data provenance, sensitive data boundaries, and access to retrieval layers.

Practical implication: verify what data the model can see, where it comes from, and whether sensitive content can reappear in output or logs.

Automated red teaming and continuous evaluation

Automated red teaming uses tools to generate repeated adversarial inputs at scale, so AI systems can be tested continuously rather than only during a point-in-time review. That matters because modern AI deployments are dynamic: prompts change, models are updated, connectors expand, and tools are added. Continuous evaluation is therefore closer to identity monitoring than one-off security testing. The objective is to catch behavioural drift, unsafe tool use, and content leakage as the system evolves, not after a major incident forces a retroactive review.

Practical implication: treat red teaming as an ongoing control that tracks drift, not as a pre-launch checkbox.

Threat narrative

Attacker objective: The objective is to make the AI system behave outside its intended boundaries so it reveals sensitive information, bypasses safeguards, or produces unsafe actions.

1. entry: An attacker or tester starts with normal prompts, malicious inputs, or crafted retrieval content that reaches the AI system through its approved interfaces.
2. escalation: The prompt or context manipulates the model into revealing hidden instructions, ignoring constraints, exposing data, or taking unsafe actions through connected tools.
3. impact: The final outcome is data leakage, unsafe output, policy bypass, or misleading system behaviour that undermines trust in the AI service.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI red teaming is becoming an identity governance discipline, not just a model-testing exercise. The article treats red teaming as a way to evaluate resilience, but the deeper issue is whether the AI system can be trusted to stay inside authorised bounds when exposed to adversarial input. That moves the conversation from software testing into access governance, decision control, and data boundary enforcement. Practitioners should treat AI red teaming as part of AI identity assurance, not a separate security ritual.

Prompt injection is a trust-boundary failure, not just a content problem. When an AI system blends user input, hidden instructions, retrieval content, and tool output, the control failure is usually about whose instructions are allowed to dominate at runtime. That is why red teaming exposes a governance gap that classic testing misses. The implication is that control design must account for instruction precedence, not only model accuracy.

Context-bound AI behaviour: the system appears controlled until attacker-shaped input changes what it is allowed to reveal or do. That concept is useful because it captures the gap between static policy and runtime behaviour in GenAI environments. If the system can be redirected by prompt content or tool context, then the governance model is already weaker than the approval model suggests. Practitioners should map where context becomes authority.

Automated red teaming validates that AI governance now needs continuous assurance. The article’s point about ongoing simulation is important because AI systems change too quickly for quarterly or annual testing to be enough. Model updates, new connectors, and altered prompts can all introduce new failure paths after deployment. Security teams should expect red team coverage to become part of normal operations rather than a project milestone.

Regulatory pressure is turning AI testing into an operational requirement. The article points to the EU AI Act, White House guidance, and the NIST AI RMF, which means red teaming is no longer only a technical preference. The practical consequence is that evidence of testing, issue handling, and control validation will increasingly matter to auditors and risk owners. Practitioners should prepare for repeatable evidence, not just one-off findings.

From our research:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%), according to AI Agents: The New Attack Surface report.
• Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation, according to the same research.
• For a broader NHI lens on governance pressure, see The 52 NHI breaches Report, which helps frame how identity failures become repeatable attack patterns.

What this signals

Context-bound AI behaviour: red teaming is increasingly the only practical way to see when a model’s apparent compliance disappears under adversarial input. With 80% of organisations reporting AI agents acting beyond intended scope, according to AI Agents: The New Attack Surface report, the governance problem is no longer theoretical.

For identity and security programmes, the next step is to connect red team findings to access control, data handling, and lifecycle ownership. If a model can be redirected by prompt content, then the surrounding governance model must account for runtime behaviour as well as static permissions.

Teams that already align AI testing to the NIST AI 600-1 Generative AI Profile will have an easier path to repeatable evidence. The signal to watch is whether findings turn into durable control changes rather than one-off fixes.

For practitioners

• Map AI trust boundaries before testing begins List where prompts, retrieved data, system instructions, and tool calls intersect, then define which inputs can influence model decisions. This gives red teamers a clear boundary map and helps owners see where prompt injection or unsafe tool use is most likely to succeed.
• Test for data exposure across retrieval and output paths Probe whether the model can reproduce sensitive information from training content, embedded documents, or connected knowledge sources. Validate both the response channel and any logging or export path that may retain the same data.
• Include unsafe tool-use scenarios in every evaluation cycle Simulate cases where the model is nudged to call tools, fetch records, or act on context it should ignore. This is especially important where agents, copilots, or workflow assistants can change execution paths at runtime.
• Treat red team findings as control ownership issues Assign each finding to a named owner for policy, prompt design, data governance, or access control. Findings without ownership tend to reappear when the model, prompt set, or connected app changes.

Key takeaways

• AI red teaming is an essential control for GenAI because many failures emerge through behaviour, not code exploitation.
• The most relevant risks are prompt injection, data leakage, jailbreaks, and unsafe tool use, all of which can bypass conventional assumptions about control.
• Organisations should tie red team findings to ownership, data boundaries, and continuous evaluation so the same failure modes do not keep returning.

Key terms

• AI Red Teaming: AI red teaming is adversarial testing that tries to make an AI system fail in realistic ways. It focuses on model behaviour, data exposure, unsafe actions, and policy bypass, not just software defects. In practice, it tests whether the surrounding governance can withstand malicious or careless use.
• Prompt Injection: Prompt injection is an attack technique that manipulates an AI system through crafted input so it follows attacker instructions instead of intended policy. It matters because the system may treat untrusted text as operational context. In AI governance, it is a trust-boundary failure that can change runtime behaviour.
• Data Poisoning: Data poisoning is the deliberate contamination of training or retrieval data so an AI system learns or surfaces attacker-chosen behaviour. It can degrade accuracy, bias outputs, or create data leakage paths. The control problem is provenance, source trust, and what content the model is allowed to absorb.
• Automated Red Teaming: Automated red teaming uses tools to generate adversarial tests at scale and repeat them continuously. It is useful where AI systems change often, because point-in-time testing quickly becomes stale. The goal is to detect behavioural drift, unsafe tool use, and output leakage before users do.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or governance maturity in your organisation, it is worth exploring.

This post draws on content published by WitnessAI: AI red teaming for GenAI security and compliance. Read the original.