AI red teaming is becoming core to AI security governance

At a glance

What this is: This is an analysis of AI red teaming and its role in AI security governance, with the key finding that adversarial testing must become a repeatable control rather than a one-off review.

Why it matters: It matters because AI and IAM teams now have to govern model behaviour, tool use, and data exposure with the same discipline used for other high-risk security controls.

Context

AI red teaming is a structured way to test how AI systems fail under adversarial pressure. It looks for behavioural weaknesses such as prompt injection, jailbreaks, data leakage, unsafe outputs, and tool-use abuse rather than treating the model as a static application.

For identity and security teams, the governance gap is that probabilistic systems can behave safely in one context and fail in the next. That makes model evaluation, monitoring, and regression testing part of the control surface, not an optional assurance layer.

TROJ.AI positions this as a lifecycle discipline rather than a point-in-time event, which is the right framing for programmes that need to keep pace with changing models, policies, and integrations.

Key questions

Q: How should security teams run AI red teaming for GenAI systems?

A: Start with the system’s trust boundaries, then test prompts, retrieval sources, tool calls, and output handling together. Good AI red teaming does not stop at finding bad answers. It checks whether the system can be pushed into revealing data, bypassing policy, or taking unsafe actions through connected interfaces.

Q: Why does AI red teaming need to be repeated after each model or policy change?

A: Because AI behaviour is probabilistic and context dependent, yesterday’s fix can fail after a model update, prompt change, data refresh, or policy adjustment. Repetition is what turns red teaming into a control rather than a one-time finding. Without regression, organisations cannot show that risk really fell, only that it was reduced once.

Q: What are the signs that a generative AI red teaming program is missing important risks?

A: A weak program usually focuses only on obvious prompt attacks and misses business harm, policy violations, and runtime behavior. Another warning sign is testing too late in the SDLC, or only once before release. If teams cannot show what changed after each exercise, or cannot connect findings to guardrails, the program is not producing durable security value.

Q: What should boards ask for after an AI red teaming exercise?

A: Boards should ask for trendlines, not anecdotes: what changed in exploit success rate, how quickly unsafe behaviour was detected and mitigated, whether privacy or policy failures recurred, and which high-risk integrations were exercised. That gives leadership a view of residual risk and whether governance is keeping pace with model change.

Technical breakdown

Why AI red teaming is different from pentesting

AI red teaming is an adversarial assessment of model behaviour and the controls around it. Pentesting usually targets infrastructure or application flaws, while red teaming for AI probes misuse paths, emergent behaviour, and socio-technical failure modes that appear when people, prompts, retrieval, tools, and policy interact. That difference matters because the same model can be safe in one conversation and unsafe in another. The objective is not only to break the system, but to measure how it fails, how often it fails, and whether the failure can be repeated under realistic pressure.

Practical implication: Treat AI red teaming as a control family with its own scope, metrics, and remediation loop, not as a variant of infrastructure testing.

How behavioural weaknesses show up in generative AI

Generative AI expands the attack surface because the input space is conversational, contextual, and easy to manipulate. Prompt injection can redirect behaviour, jailbreaks can bypass policy constraints, data leakage can expose secrets or personal data, and tool-use chains can amplify a bad instruction into a wider system failure. These are not just model issues. They are control issues across retrieval, moderation, tool permissions, and output handling. When the model is connected to external systems, the impact can move from unsafe text to unsafe action.

Practical implication: Map red-team scenarios to the full AI stack, including retrieval, tools, connectors, and policy enforcement points.

Why repeatability and regression matter

A single successful test does not prove the system is safe, and a single fix does not prove the problem is gone. AI red teaming has to be repeatable because model updates, prompt changes, policy adjustments, and data refreshes can reintroduce previously closed failures. That is why the article emphasises attack success rate, time to detect, time to mitigate, and regression tracking. These measures turn red teaming into an operational assurance process rather than a one-time demonstration.

Practical implication: Build a regression harness so every confirmed failure becomes a test case before the next release or policy change.

Threat narrative

Attacker objective: The attacker wants the AI system to ignore its intended constraints and produce harmful, secret-bearing, or policy-violating output at scale.

1. Entry begins when an adversary supplies a crafted prompt, injected instruction, or manipulated context that reaches the model through chat, retrieval, files, or tools.
2. Credential or control abuse occurs when the model follows malicious instructions far enough to expose secrets, cross policy boundaries, or act through connected tools it should not trust.
3. Escalation happens when the unsafe behaviour is repeated, generalised, or chained across multiple turns, tools, or modalities, increasing the scope of harm.
4. Impact is the release of sensitive data, policy-violating content, or unsafe actions that demonstrate a broken trust boundary between the model and its operating environment.

Breaches seen in the wild

• Microsoft Azure OpenAI abuse by Storm-2139: Storm-2139 used API keys leaked by Microsoft customers to hijack Azure OpenAI, bypass safety guardrails and resell access to generate harmful content.
• 12,000 secrets in LLM training data: Truffle Security found 11,908 live API keys and passwords hard-coded in web pages captured by Common Crawl, a dataset used to train LLMs.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI red teaming is now a governance control, not a specialist side activity. The article correctly frames adversarial testing as part of the AI control stack because behavioural failure is a security outcome, not just a model-quality issue. That shifts responsibility onto security, product, legal, and AI governance teams together. The practitioner conclusion is simple: if a model changes, the control must be re-run.

Behavioural risk creates a distinct kind of assurance problem. Traditional assurance assumes the system behaves consistently enough for static review, but generative AI produces variable outputs under changing context. That means prompt injection, jailbreaks, and data leakage are not edge cases, they are the expected failure classes that red teams must pressure-test. The practitioner conclusion is to measure behaviour under stress, not just benchmark accuracy.

Repeatability is the difference between evidence and theatre. The strongest point in the article is its lifecycle model, because scoping, testing, triage, mitigation, and regression only have value when they recur after every meaningful model or policy change. That makes attack success rate and regression rate governance metrics, not technical vanity metrics. The practitioner conclusion is to tie release approval to repeatable adversarial evidence.

Human creativity still matters because automation cannot discover every meaningful failure path. The article is right to argue for hybrid testing, since automated generation widens coverage but expert testers still find the highest-value chains, especially where domain context or multi-turn reasoning matters. That has direct implications for programme design: AI red teaming teams need both scale and judgment. The practitioner conclusion is to staff for depth, not just tooling throughput.

Adversarial testing is becoming part of the AI accountability model. As regulation, executive guidance, and customer assurance expectations converge, organisations will need to prove they are testing unsafe behaviour before deployment and after change. That aligns AI red teaming with NIST AI RMF style governance more than with classic point-in-time assurance. The practitioner conclusion is to treat evidence production as part of the control itself.

What this signals

AI red teaming is becoming a standing control for AI governance. The important programme shift is from testing models as if they were static software to testing them as living systems that change with prompts, data, tools, and policy. That means release gates, post-change regression, and evidence collection now belong inside the governance model, not alongside it.

Model behaviour has become a security boundary. Once AI systems can leak data, ignore instructions, or act through connected tools, governance has to focus on operational behaviour rather than abstract model quality. For practitioners, that means the real question is not whether the model performs well in general, but whether it remains safe under adversarial pressure across its full operating environment.

For practitioners

• Define the system under test Specify the model version, data sources, tools, user roles, and harm categories before any adversarial run so results are attributable and repeatable.
• Instrument full telemetry Capture prompts, context windows, tool calls, outputs, and classifier verdicts in a controlled environment so investigators can reconstruct each failure path.
• Turn failures into regressions Convert every confirmed unsafe behaviour into a test case, then gate the next release on whether the same behaviour reappears under updated prompts, data, or policy.
• Use hybrid red-teaming methods Combine automated adversarial generation for breadth with human testers for scenario depth, triage, and mitigation design where the impact is highest.
• Report governance metrics Track attack success rate, time to detect, time to mitigate, residual risk, and regression rate so leadership can see whether the control is reducing exposure over time.

Key takeaways

• AI red teaming addresses a different class of risk from conventional testing because the failure mode is behavioural, contextual, and often tied to how people and tools interact with the model.
• The article’s core evidence is the need to track exploit success, mitigation speed, and regression over time, which makes red teaming measurable rather than subjective.
• The practical implication is to run adversarial testing as a recurring control tied to release changes, with hybrid testing and regression harnesses built into the programme.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Attack Success Rate: Attack success rate is the percentage of test attempts in which an injected adversarial scenario achieved its intended effect. For agentic systems, it helps distinguish a one-off anomaly from a repeatable workflow weakness that requires governance action.
• Runtime Harness: The combination of environment, permissions, tools, and lifecycle controls that determines what an AI agent can actually do. In practice, the harness is part of the control surface, because it shapes execution boundaries, failure modes, and the evidence available for review.
• Context Injection: A failure mode where malicious or misleading content enters an agent's decision path and affects what it does next. In MCP environments, the injected context can alter tool selection, broaden data exposure, or trigger unsafe follow-on actions across multiple systems.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an identity or security programme, it is worth exploring.