AI red teaming is becoming core to AI security governance

At a glance

What this is: AI red teaming is structured adversarial testing for AI systems that evaluates behavioural failures, misuse paths, and control gaps under realistic attack pressure.

Why it matters: It matters because AI, NHI, and human governance programmes now need evidence that model behaviour, tool access, and policy controls hold up under adversarial conditions.

👉 Read TROJ.AI's analysis of AI red teaming as a board-level security control

Context

AI red teaming is the disciplined practice of attacking AI systems the way a real adversary would, then using the results to improve governance, safety, and monitoring. The primary issue is not whether the model performs well in ideal conditions, but whether its behaviour stays within policy when prompts, retrieval content, tools, and users are all part of the attack surface.

For identity teams, the connection is practical: AI systems increasingly sit on top of credentials, APIs, retrieval layers, and delegated access, which means model behaviour can affect secrets exposure and policy enforcement. That makes red teaming part of identity governance as much as model assurance, especially where non-human identities and delegated tool use are involved.

Key questions

Q: How should security teams run AI red teaming against systems with tool access?

A: Security teams should test the full system, not just the model. That means exercising prompts, retrieval, files, plugins, and API calls in controlled conditions, then measuring whether the AI can be coerced into leaking data or taking unsafe actions. The most useful programmes turn confirmed failures into regression tests and tie them to change control.

Q: When does AI red teaming become more important than normal model evaluation?

A: It becomes more important when the AI can access data, tools, or workflows that matter to the business. Standard evaluation measures performance under normal conditions, but red teaming tests misuse, coercion, and context-dependent failure. If the system can reveal secrets, trigger actions, or influence decisions, adversarial testing should be part of the release process.

Q: What do organisations get wrong about AI red teaming?

A: The common mistake is treating it as a one-time assessment or a list of prompts. That misses the fact that AI behaviour changes with context, model updates, and surrounding controls. A useful red team programme is iterative, evidence-based, and tied to remediation, rather than a standalone exercise that produces a report and stops.

Q: Who should own AI red teaming when identity and security controls are involved?

A: Ownership should be shared across security, product, legal, and the teams that manage access and integrations. When AI systems use credentials, APIs, or delegated permissions, identity owners need to understand the failure modes as clearly as the model team does. Without that shared ownership, findings are hard to triage and even harder to fix.

Technical breakdown

AI red teaming vs penetration testing

AI red teaming is not a renamed pentest. Penetration testing focuses on exploiting infrastructure or applications within a bounded scope, while red teaming targets behaviour, instruction handling, retrieval, and tool-use paths that can produce unsafe outcomes without a classic exploit. The key unit of analysis is the socio-technical system, not only the model. That includes prompts, context windows, guardrails, plugins, and the business process around the AI. This is why adversarial testing often finds failures that benchmark evaluation misses, because the model can be statistically accurate yet operationally unsafe under pressure.

Practical implication: test the full AI system, including tools and retrieval, rather than treating the model as an isolated asset.

Prompt injection, jailbreaks, and context poisoning

Prompt injection exploits the model’s tendency to treat untrusted instructions as if they were legitimate context, especially when those instructions arrive through retrieved documents, files, tool outputs, or embedded content. Jailbreaks try to bypass policy boundaries directly, while context poisoning works indirectly by seeding malicious instructions into data the model will later trust. These are behavioural weaknesses, not infrastructure vulnerabilities, which is why simple filtering rarely solves them. Once an AI system can read, summarise, decide, or act on external content, adversaries can steer it by shaping what it believes is relevant.

Practical implication: sanitise retrieval and tool inputs, and assume any external content can become an instruction channel.

Metrics that prove red teaming is reducing risk

A useful red team programme measures change, not just findings. Attack success rate shows how often adversarial attempts bypass controls, time to detect and time to mitigate show whether the organisation can respond quickly enough, and regression rate shows whether fixes hold after model or policy changes. Coverage matters too, because a programme that only tests one language, one modality, or one tool chain creates false confidence. The strongest governance signal is trend data over time, paired with repeatable scenarios that can be rerun after each release.

Practical implication: build a regression harness and track trendlines so each model update can be compared against the last.

NHI Mgmt Group analysis

AI red teaming exposes a governance gap, not just a testing gap: organisations still treat model assurance, application security, and identity governance as separate disciplines, but adversarial AI testing cuts across all three. The article is right to frame red teaming as a lifecycle because the failure modes recur whenever models, prompts, retrieval sources, or tool permissions change. That makes the control problem continuous rather than episodic. The practitioner conclusion is that AI assurance must be operationalised as part of identity and access governance, not bolted on after deployment.

The behavioural risk surface is now bigger than the model itself: prompt injection, jailbreaks, and data leakage only become operationally dangerous when the model has reach into tools, retrieval, or credentials. That is why NHI governance matters here even when the article talks in AI terms. The real issue is delegated access with behavioural uncertainty, which is a classic boundary problem for identity teams. The practitioner conclusion is that AI red teaming must include the identities, tokens, and permissions that extend model capability.

Attack success rate is the right kind of evidence for AI governance maturity: traditional pass-fail reporting hides the fact that AI systems fail probabilistically. A red team programme that reports exploit success rate, time to detect, time to mitigate, and regression rate is much closer to how risk is actually experienced. That aligns with NIST Cybersecurity Framework 2.0 style measurement thinking and makes AI security reportable to boards. The practitioner conclusion is to use repeatable metrics that show whether behaviour is improving, not just whether issues were found.

AI red teaming should be treated as a standing control family: the article’s lifecycle framing is the right one because release cycles, policy updates, and tool changes all reshape the attack surface. The named concept here is runtime behavioural assurance, meaning the organisation must continuously prove that AI actions remain within expected bounds after each change. That is a governance posture, not a one-off exercise. The practitioner conclusion is to tie adversarial testing to change management, incident reviews, and release gates.

Board oversight becomes necessary once AI systems can produce policy-relevant harm at scale: safety, privacy, and compliance failures are no longer purely technical incidents when they can generate illegal instructions, bias, or confidential disclosure. The governance question shifts from whether a model is accurate to whether the organisation can evidence adversarial resilience. The practitioner conclusion is to treat red team trendlines as board material, especially for higher-risk systems and regulated environments.

From our research:

• 85% of organisations lack full visibility into third-party vendors connected via OAuth apps, according to The State of Non-Human Identity Security.
• Only 1.5 out of 10 organisations are highly confident in their ability to secure NHIs, which shows the control gap is already visible before AI systems add more delegated access.
• For a deeper operating model, see Ultimate Guide to NHIs , Lifecycle Processes for Managing NHIs for how provisioning, rotation, and offboarding change when access must be continuously governed.

What this signals

Runtime behavioural assurance: AI security programmes are moving from point-in-time evaluation to continuous evidence of control performance. That shift matters because models, prompts, data sources, and toolchains all change faster than annual review cycles, and the board will eventually ask for trend data rather than anecdotes.

For identity teams, the immediate pressure point is delegated access. With 85% of organisations lacking full visibility into third-party vendors connected via OAuth apps, according to The State of Non-Human Identity Security, AI systems that can call tools or retrieve data inherit the same blind spots if governance is not redesigned.

Practitioners should expect red teaming to converge with release engineering, access review, and incident response. The organisations that will reduce risk fastest are the ones that can rerun adversarial cases after every material change and prove that the failure rate is trending down.

For practitioners

• Map the full AI attack surface Inventory prompts, retrieval sources, tool connections, output destinations, and the identities that let the model act. Treat the model, surrounding code, and delegated access as one control plane rather than separate projects.
• Build adversarial scenarios from real misuse paths Create test cases for prompt injection, jailbreaks, data leakage, multilingual coercion, and tool misuse. Use both seeded prompts and dynamic mutations so the programme does not ossify around a static list.
• Turn each confirmed failure into a regression test Re-run the same scenario after any model, policy, data, or tool change. Gate releases on whether previously observed failures reappear and require owners to explain any increase in attack success rate.
• Tie AI testing to governance cadence Schedule red team campaigns before major model updates, policy changes, or new integrations, and report the results alongside risk metrics already used for security and compliance reviews.

Key takeaways

• AI red teaming is a behavioural control, not a model-only evaluation, because the real risk emerges when prompts, retrieval, tools, and credentials interact.
• The scale of the problem is governance visibility, not just technical weakness, and organisations already show large blind spots in delegated access paths.
• The practical response is a repeatable testing lifecycle with measurable regression, tied directly to change management and identity governance.

Key terms

• AI Red Teaming: A structured adversarial assessment of an AI system’s behaviour under misuse, coercion, and control pressure. It tests whether the system can be manipulated into unsafe outputs or actions, and whether surrounding guardrails, tools, and governance processes can withstand that pressure.
• Prompt Injection: A technique that hides malicious instructions inside content an AI system is likely to trust, such as documents, web pages, or tool output. The risk is not only bad text generation, but also the model treating attacker-controlled content as if it were legitimate direction.
• Regression Test: A repeatable test used to confirm that a previously discovered failure has not returned after a model, policy, data, or tool change. In AI governance, regression tests are crucial because the same system can pass one day and fail the next when context changes.
• Behavioural Assurance: Evidence that an AI system continues to act within expected bounds under realistic pressure. For autonomous or tool-connected systems, behavioural assurance has to cover decision-making, tool use, and timing, not just output quality in clean test conditions.

What's in the full article

TROJ.AI's full article covers the operational detail this post intentionally leaves for the source:

• Step-by-step red team workflow for scoping, execution, triage, mitigation, and regression.
• Examples of harmful AI scenarios across prompt injection, jailbreaks, privacy leakage, and tool abuse.
• Metric definitions and board-reporting signals such as attack success rate, time to detect, and time to mitigate.
• Guidance on building a hybrid human-plus-automation programme for higher-coverage testing.

👉 The full TROJ.AI article covers the red teaming lifecycle, metric framework, and governance guidance in more detail.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.