AI red teaming is becoming core to LLM security governance

At a glance

What this is: AI red teaming is presented as a governance requirement for LLM security because it exposes prompt injection, leakage, and configuration weaknesses that ordinary controls miss.

Why it matters: For IAM, NHI, and AI security teams, the key issue is that access, context, and tool use now need to be tested as part of the control plane, not only after deployment.

Context

AI red teaming is the controlled simulation of adversarial behaviour against an LLM system so teams can find weaknesses before users or attackers do. In this article's framing, the security gap is not the model alone but the full application stack around it, including prompts, context, integrations, access paths, and release workflows.

That matters because traditional application and infrastructure security controls were not designed for multi-turn manipulation, prompt-based abuse, or context injection. Where AI systems touch identities, permissions, or tool access, red teaming becomes a governance mechanism for both AI security and identity control.

The article also treats openness and usability as adoption drivers, which is an operational point rather than a product claim. If red teaming is too hard to run, it will stay outside normal engineering and security practice, which is atypical for the kind of exposure AI systems now create.

Key questions

Q: What breaks when red teaming is only done before release?

A: A pre-release-only approach misses the changes that happen after deployment, including tool updates, new integrations, memory changes, and behaviour drift. An agent that passed testing on day one may operate differently weeks later. Continuous red teaming is needed because the operational surface keeps moving after the release gate closes.

Q: Why do LLMs create risk even when the underlying code is secure?

A: Because the failure point is often the interaction layer, not the codebase. Prompt injection, manipulated context, unsafe outputs, and delegated actions can all produce harm without a classic exploit. The risk comes from runtime trust decisions that code scanning and static controls do not fully model.

Q: What are the signs that AI red teaming is missing important attack paths?

A: The main warning signs are tests that only cover single prompts, no multi-turn scenarios, little variation in input style, and reports that focus on model quality rather than abuse outcomes. If the same test suite never changes, it is probably not keeping up with how attackers actually adapt.

Q: Should security teams treat AI red teaming as part of NHI governance?

A: Yes, when an AI system can access data or trigger tools on behalf of users, it behaves like a non-human actor with delegated privilege. Governance should therefore cover context, scope, and revocation in the same programme that manages machine identities and other high-risk access paths.

Technical breakdown

Why multi-turn prompt injection evades standard controls

Prompt injection is not just a single malicious prompt. It is often a sequence of crafted inputs, formatting tricks, or context shifts that persuade the model to ignore its intended instructions. Traditional controls such as content filters or static tests often assume one-shot abuse, but LLMs can be manipulated across turns, which means the attack surface is conversational and stateful. That statefulness also makes the system harder to secure with perimeter-style controls alone, because the dangerous behavior can emerge from accumulated context rather than one obvious request.

Practical implication: Test for multi-turn abuse paths, not only single-prompt jailbreaks.

Why LLM security depends on the surrounding access and tool layer

The article treats the model as only one part of the system. Real risk often sits in the surrounding architecture, where the LLM can see sensitive context, call tools, or interact with services under delegated permissions. That is where identity and privilege questions matter most: if an AI workflow can reach data or tools that were not intended for the user, the model becomes an enforcement gap. In practice, LLM security governance has to track what the model can observe, what it can ask for, and what it can do once a tool call is made.

Practical implication: Constrain model-visible context and delegated tool permissions as tightly as any other privileged workflow.

How scoring and compliance mapping turn red teaming into governance

Red teaming only changes practice when results can be compared, reported, and acted on. Scoring gives teams a way to prioritise findings, while compliance mapping links test outcomes to governance frameworks and release decisions. The article's point is that AI security cannot remain an ad hoc exercise if organisations want repeatable assurance. Automated red teaming makes the testing cycle operational, but human review still matters because business impact, legal exposure, and policy exceptions cannot be reduced to a single score.

Practical implication: Use repeatable scoring and policy mapping so red teaming informs go or no-go decisions.

Threat narrative

Attacker objective: The objective is to manipulate the LLM into exposing information, bypassing policy, or misusing connected tools and workflows.

1. Entry occurs through prompt injection, formatting tricks, or other crafted inputs that change the model's interpretation of the conversation.
2. Credential or context abuse follows when the model is steered to reveal sensitive data or operate on information it should not expose.
3. Escalation happens when the AI workflow has access to tools, integrations, or delegated permissions that expand the effect of the manipulated interaction.
4. Impact is data leakage, policy violation, or unsafe downstream action caused by the model acting on compromised context.

Breaches seen in the wild

• ForcedLeak (Salesforce Agentforce) 2025: A critical Agentforce flaw let attackers steer an AI agent through a Web-to-Lead form to leak CRM data via an expired allowlisted domain.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI red teaming has become a governance control, not a niche testing activity. The article correctly frames red teaming as a way to find failure modes before production use, but the deeper point is that AI systems now need adversarial assurance in the same way identity systems need lifecycle control. When prompts, context, and tools all influence outcomes, security cannot rely on ordinary application testing alone. Practitioners should treat red teaming as part of the release gate, not a periodic exercise.

The governance gap is context blindness. Traditional controls are built to secure systems that behave predictably under known inputs, while LLMs can be reshaped by conversation state, injected context, and indirect instructions. That makes multi-turn testing especially important because the failure often appears only after the model has accepted a poisoned frame. In identity terms, the issue is not just access, but what the system is allowed to infer and execute from that access context. Practitioners should test the whole interaction path, not the prompt in isolation.

Identity and privilege now sit inside AI security design. The article notes role-based access controls and sensitive data protection as core priorities, which is the right direction but still too narrow if teams only think about model behaviour. If an AI system can use tools, retrieve data, or act across applications, it is effectively operating as a non-human identity with delegated authority. That means governance has to cover scope, context, and revocation as well as model quality. Practitioners should align AI red teaming with NHI and access governance, not leave it in a separate AI silo.

Open source is an adoption strategy, not just a licensing choice. The article makes a valid case that usability, transparency, and cost shape whether AI security testing becomes routine. For security teams, that matters because inaccessible tooling tends to stay outside engineering workflows and never becomes enforceable practice. The market signal is clear: the winning control model is the one developers will actually run continuously. Practitioners should value operational fit as much as technical coverage when selecting AI security tooling.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• 73% of vaults are misconfigured, leading to unauthorised access and exposure of sensitive data, according to the Ultimate Guide to NHIs.
• Read next: Agentic AI Security Guide

What this signals

Context-aware testing will become the default control expectation for AI systems. As LLMs move deeper into business workflows, teams will need to test the whole interaction layer, not just the model endpoint. That shifts assurance from static evaluation to continuous adversarial validation across prompts, retrieval, and tool calls.

Non-human privilege is now part of AI security design. Once an AI workflow can read data or invoke services, it needs the same scrutiny as any other delegated actor. The practical implication is that access scope, approval boundaries, and revocation controls should be built into AI governance from the start.

For practitioners

• Define red-team scope around the full AI interaction path Include prompts, multi-turn conversation state, retrieved context, connected tools, and post-model actions in every test plan so you do not validate the model in isolation.
• Map delegated AI permissions to business-risk boundaries Review what the model can read, what it can call, and what it can trigger, then remove any access that is not required for the intended use case.
• Build repeatable attack suites for prompt injection and leakage Maintain a living library of adversarial prompts, formatting variations, and context-injection patterns so regressions are caught before release.
• Use scoring to prioritise findings for release decisions Tie red-team results to severity thresholds, compliance evidence, and approval gates so teams can distinguish nuisance failures from production blockers.
• Run manual reviews alongside automation Have security and product owners inspect high-risk failures, because business context, policy exceptions, and legal exposure cannot be reduced to a single score.

Key takeaways

• AI red teaming is increasingly the control that exposes how LLM systems fail under realistic abuse, especially when attackers manipulate context rather than simply the prompt.
• The real exposure sits in the surrounding workflow, where delegated tools, retrieved data, and multi-turn conversations can turn a model into an unsafe actor.
• Security teams should treat red teaming as a release-gating practice that informs access boundaries, testing depth, and governance decisions.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Context Injection: A failure mode where malicious or misleading content enters an agent's decision path and affects what it does next. In MCP environments, the injected context can alter tool selection, broaden data exposure, or trigger unsafe follow-on actions across multiple systems.
• Delegated AI Authority: Delegated AI authority is the permission a person gives an assistant to act inside business systems on their behalf. It turns a conversational tool into an execution layer, which means security teams must govern scope, auditability, and revocation with the same seriousness they apply to privileged access.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is designed for teams that need to connect access governance with the rise of software agents and machine identities.