AI red teaming exposes where LLMs leak data and break policy

At a glance

What this is: This article explains how AI red teaming reveals where LLMs leak sensitive data or break policy through prompt-based abuse, retrieval poisoning, and unsafe function use.

Why it matters: It matters because identity, access, and data controls that work for deterministic systems often fail when AI systems infer, synthesize, or overshare information across changing contexts.

Context

AI red teaming is a security testing approach for model-driven systems, but the control problem is broader than prompt abuse. Large language models can leak information through retrieval, context, and function execution in ways that traditional application testing does not expose, which makes AI governance a runtime issue rather than a release checkpoint. Where enterprise AI touches sensitive data, the boundary between model behaviour and access control becomes a real identity and policy concern.

The article is most relevant to teams governing enterprise GenAI, copilots, and retrieval-augmented workflows because the model can surface information that users were never meant to see. That shifts the question from whether the model is accurate to whether it can be trusted to respect data boundaries under adversarial prompting, poisoned context, or misused tool permissions.

Key questions

Q: What breaks when an LLM is not red teamed for prompt injection and leakage?

A: The failure is usually not a code crash but a policy collapse. The model can be steered into revealing sensitive context, following hostile instructions, or exposing data through retrieval and tool use. Without red teaming, teams often discover the problem only after users, logs, or downstream systems have already seen the unsafe output.

Q: Why do LLMs create more risk than ordinary application workloads?

A: LLMs do not just process input, they interpret natural language, retrieve context, and may trigger actions based on that interpretation. That means a malicious prompt can become an execution path, especially when the model has access to data stores, tools, or business systems. Identity scope and runtime policy therefore matter more than static perimeter controls.

Q: What are the signs that AI security controls are failing in production?

A: Common warning signs include unapproved model behavior, unexpected data access, prompt leakage, suspicious outbound calls, and runtime actions that do not match the workload’s intended function. If teams also see weak visibility into pipelines, inconsistent policy enforcement, or repeated attempts to access sensitive datasets, the control layer is not keeping pace with the AI threat surface.

Q: Should AI red teaming be governed like application security or like identity control?

A: It should be governed as both, because AI red teaming tests behavior, but the risk is often in what the system can access or execute. Application security checks exposure, while identity control defines authority. In AI systems, those controls overlap, so practitioners need one governance model that covers both.

Technical breakdown

Why LLM red teaming differs from classic penetration testing

Traditional penetration testing looks for deterministic flaws in code, network paths, or configuration. LLM red teaming instead tests a probabilistic system whose outputs vary with prompt wording, conversation state, retrieved context, and tool access. That means the attack surface includes semantics, not just syntax. Small changes in instructions can alter whether the model discloses data, follows policy, or calls a function it should not. The security question is therefore not only whether the system is reachable, but whether the model can be steered into unsafe reasoning or disclosure under realistic adversarial pressure.

Practical implication: Treat AI red team results as governance evidence for model behaviour, not just as bug reports for developers.

How prompt injection and vector-store poisoning create leakage paths

Prompt injection works by inserting instructions that override or confuse the model’s intended behaviour. Vector-store poisoning takes advantage of retrieval systems by seeding content that the model later treats as trusted context. In both cases, the model may reveal sensitive material, follow attacker-chosen instructions, or produce outputs that violate policy even when the underlying application code is intact. This is why RAG systems need security review at the retrieval layer, not just at the prompt layer. A poisoned document can become a durable control failure if its trust status is never challenged.

Practical implication: Review retrieval sources, ranking logic, and context assembly as part of AI access control.

Why function misuse turns an LLM into an access-risk amplifier

When an LLM can invoke tools, APIs, or internal functions, it is no longer only generating text. It is participating in execution paths that may touch sensitive data, privileged workflows, or downstream systems. Function misuse occurs when the model selects an action it should not, requests permissions it does not need, or calls an endpoint in a way that breaches policy. This is especially dangerous in enterprise environments where access decisions are implicit in the tool chain. The risk is not just leakage, but unauthorised action taken under the appearance of normal model behaviour.

Practical implication: Constrain tool permissions and validate model-to-tool requests as if they were privileged access events.

Threat narrative

Attacker objective: The attacker aims to make the model reveal protected information or execute a privileged action while appearing to operate normally.

1. Entry begins with crafted prompts, poisoned retrieval content, or malicious context designed to steer the model away from its intended policy boundary.
2. Credential access or abuse occurs when the model is manipulated into surfacing sensitive snippets, using trusted context incorrectly, or invoking functions with excessive privilege.
3. Impact follows when confidential data, policy exceptions, or unsafe actions are exposed to users, logs, or downstream workflows.

Breaches seen in the wild

• EchoLeak (Microsoft 365 Copilot) 2025: A crafted email could make Microsoft 365 Copilot leak data from its context with no click, a zero-click prompt injection fixed as CVE-2025-32711.
• ForcedLeak (Salesforce Agentforce) 2025: A critical Agentforce flaw let attackers steer an AI agent through a Web-to-Lead form to leak CRM data via an expired allowlisted domain.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI red teaming has become an access-control discipline, not just a testing practice. Once models can retrieve, infer, and act across systems, the question is whether they respect policy boundaries under pressure. That makes red teaming a governance control for data exposure, not a specialist exercise that sits outside IAM or data security.

Inference-based leakage is the named failure mode that traditional controls miss. The model does not need a direct file read to reveal protected information if it can reconstruct or connect enough context from safe-looking inputs. That means policy enforcement has to account for synthesis risk, not only direct access paths, and practitioners should treat oversharing as a control gap, not a content issue.

Prompt injection is only one part of the problem; trust in retrieved context is the larger issue. RAG pipelines and connected tools often inherit trust from their surrounding workflow without a strong enough validation layer. For identity and access teams, that means data exposure now depends on whether the model can be made to act on content it should not trust.

Function misuse turns AI into a privilege broker unless the tool boundary is explicit. If a model can request actions without a human-style approval checkpoint, then least privilege must move into the tool layer itself. The practical result is that AI security and NHI governance now overlap at the point where models call systems on behalf of users or workflows.

Continuous simulation is the right operating model for GenAI governance. The article shows why one-time validation fails once models, connectors, and permissions keep changing. Practitioners should treat leakage testing as an ongoing control, because the boundary between acceptable output and policy breach is unstable by design.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• 43% of security professionals are concerned about AI systems learning and reproducing sensitive information patterns from codebases, according to the State of Secrets in AppSec.
• Read next: Enterprise AI Copilot Security Guide

What this signals

Inference-based leakage is the core governance gap in enterprise AI. Teams can block obvious secret disclosure and still fail when the model reconstructs sensitive meaning from safe-looking inputs. That shifts the control objective from content filtering to runtime behaviour management, which is why red teaming has to recur whenever the model, connectors, or data sources change.

AI red teaming should be treated as part of identity and access governance. Once a model can read, retrieve, and invoke tools on behalf of users, the relevant question is whether those privileges are still bounded by policy. Oversharing drift: the failure mode where a model becomes progressively less aligned with need-to-know boundaries as context, permissions, and data sources evolve. Practitioners should expect that drift and design for it explicitly.

The broader market signal is that AI-related credential leaks surged 81.5% year-over-year in 2025, according to the State of Secrets Sprawl 2026, which reinforces how quickly AI infrastructure can become an exposure surface. Security teams should assume that model-adjacent systems, not just the model itself, now need continuous inspection and control.

For practitioners

• Map AI data exposure paths Identify which prompts, retrieval sources, plugins, and connected systems can surface sensitive content before red teaming begins. Focus on the data domains, user roles, and inherited permissions that give the model access to information it should never disclose.
• Test for inference-based leakage Run adversarial prompt sets that try to reconstruct sensitive information from safe-looking context, not just direct requests for secrets. Include repeated and mutated prompts so you can see whether the model becomes more permissive as context changes.
• Constrain tool permissions and function calls Apply least privilege to every tool the model can invoke, and validate each action as if it were a privileged request. Remove unnecessary API access, tighten approval logic, and make tool execution observable in logs.
• Treat retrieval poisoning as a governance risk Review vector stores, document sources, and embedded knowledge for untrusted or stale content that can influence answers. Put ownership and review around the content the model reads, especially where business-sensitive data is reused across assistants.
• Retest after every model or permission change Repeat red-team scenarios whenever prompts, connectors, model versions, or access rules change so exposure drift is detected early. Use the retest to confirm that the same attack no longer produces the same leak or action.

Key takeaways

• LLM red teaming exposes failures that conventional penetration testing often misses because the model can leak information through inference, retrieval, and tool use.
• The most important risk is not only direct prompt abuse but unstable policy enforcement across changing context, connectors, and permissions.
• Continuous retesting after model or access changes is the control pattern that turns red-team findings into lasting governance.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Vector Store Poisoning: The tampering of retrieval content or embeddings so a RAG system retrieves attacker-influenced context during normal operation. It turns the retrieval layer into a trust boundary that must be protected like any other sensitive data store.
• Inference-Based Leak: An inference-based leak happens when an AI system reconstructs sensitive information from harmless fragments rather than reading a single protected file. This matters because traditional DLP can miss the exposure path. The risk grows when prompts, retrievals, and outputs are combined across multiple repositories and access boundaries.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, IAM, and secrets management. It helps security and identity practitioners connect runtime access decisions to governance outcomes across modern environments.