AI red teaming exposes where LLMs leak data and break policy

At a glance

What this is: AI red teaming is a structured way to simulate attacks against LLMs and AI agents to find leakage, prompt-injection, and misuse failures before they reach production.

Why it matters: It matters because IAM, NHI, and AI governance teams need to control what models, agents, and retrieval layers can see and do, not just what human users are allowed to access.

By the numbers:

• A 2025 study found indirect prompt injection succeeded in 92% of assistant-role contexts and 86% of system-role contexts, compared with 52% in standard user-role prompts.
• A 2024 Cornell University study found that just five poisoned documents could steer a retrieval-augmented model to attacker-chosen responses 90% of the time.
• A survey of 3,892 method calls found general LLMs misused APIs in roughly 35% of cases, including inappropriate permission requests and dangerous endpoint calls.
• A 2025 report found attack success rates of 71% and 70% against AgentDojo and VWA-adv, showing how often fuzzy prompt attacks can defeat LLM-agent defenses.

👉 Read Knostic's analysis of AI red teaming, LLM leakage, and governance

Context

AI red teaming tests how language models and AI agents behave under adversarial pressure, especially when the risk is not code execution but unwanted disclosure, unsafe retrieval, or policy bypass. For identity and access teams, the issue is the same one that drives NHI governance: if a system can see sensitive data, it can also reveal it, reuse it, or infer more than intended.

The operational gap is that enterprise AI is dynamic. Prompts change, connectors change, vector stores change, and permissions drift over time, so a single security review rarely holds. That makes red teaming relevant not just to AI security teams, but to IAM, PAM, and NHI owners who must decide which identities, data sources, and tool permissions an AI system should be trusted to touch.

Key questions

Q: How should security teams implement AI red teaming in enterprise environments?

A: Start by scoping the prompts, data sources, retrieval systems, and tool permissions the model can reach. Then test for prompt injection, vector poisoning, unsafe function calls, and inference-based leakage. The most useful programmes pair adversarial testing with documented remediation and retesting so findings become enforceable control changes, not isolated security reports.

Q: Why do AI agents complicate IAM and NHI governance?

A: AI agents can read data, choose actions, and invoke tools, so their effective privileges are distributed across identities, connectors, and content. That makes them behave like non-human operators whose access must be scoped, reviewed, and monitored. Traditional IAM around human logins is not enough when the system itself can trigger sensitive workflows.

Q: What breaks when retrieval-augmented generation is not governed tightly enough?

A: When retrieval scope is too broad, the model can surface sensitive material that users should not have seen, even if no one explicitly requested a protected file. Poisoned or overshared content can also shape outputs in ways that mislead users or expose secrets. The failure is usually policy mismatch across content, permissions, and model context.

Q: Who is accountable when an AI system leaks data through a prompt or tool call?

A: Accountability should sit with the team that owns the model, the retrieval layer, and the connected data sources, because the leak usually spans all three. Governance frameworks should require audit logs, access reviews, and change tracking so security, privacy, and application owners can trace how the exposure happened and prove remediation.

Technical breakdown

Why LLM red teaming is different from classic penetration testing

Traditional penetration testing targets deterministic systems, where the same input should produce the same result. LLM red teaming instead tests probabilistic systems, where context, prompt order, retrieval data, and model state can change the output even when the prompt looks identical. That means security validation must look for leakage, hallucinated access, indirect instructions, and unsafe tool invocation rather than only obvious exploit chains. In practice, the test surface is the language layer plus the connected data and action layer.

Practical implication: Treat model, prompt, retrieval, and tool access as one attack surface when defining test scope.

Prompt injection, vector-store poisoning, and function misuse

Prompt injection manipulates model behaviour through crafted instructions hidden in user content, retrieved documents, or system-adjacent context. Vector-store poisoning plants attacker-controlled material into retrieval systems so the model retrieves and repeats harmful or misleading content. Function misuse happens when the model calls internal tools or APIs in ways the enterprise did not intend, which is especially dangerous when the agent can reach sensitive workflows. These are not abstract AI issues; they are identity and authorisation failures expressed through language.

Practical implication: Limit what the model can retrieve and which functions it can call, then test both paths continuously.

Why continuous simulation matters for AI governance

AI systems evolve faster than static controls. New documents appear, permissions shift, prompts are modified, and vendors update model behaviour, which creates exposure drift even when no one changes a formal policy. Continuous simulation turns red-team findings into a repeatable governance loop: test, document, remediate, and retest. For enterprises using copilots or RAG systems, this is the difference between a point-in-time review and a control that can keep pace with operational change.

Practical implication: Build recurring red-team cycles into release and access-change workflows, not annual assurance reviews.

Threat narrative

Attacker objective: The attacker aims to make the model reveal sensitive information or perform unsafe actions while appearing to behave normally.

1. Entry begins when an attacker injects malicious instructions into prompts, retrieved content, or shared documents that an LLM will ingest during normal use.
2. Escalation occurs when the model follows the injected instruction chain, reaches overshared context, or invokes an internal function outside its intended boundary.
3. Impact follows when the model leaks sensitive data, misuses APIs, or returns attacker-chosen outputs that corrupt decisions or expose regulated information.

NHI Mgmt Group analysis

AI red teaming is becoming the practical control plane for LLM governance. Static review does not tell you how a model behaves when prompts, retrieval, and tool permissions collide under adversarial pressure. The article’s core evidence shows that leakage and misuse emerge from interactions, not just from code defects. Practitioners should treat red teaming as a standing governance function, not a one-off security exercise.

Inference-based leakage is the defining risk for enterprise AI, and IAM controls alone do not explain it. The model may reveal information it was never explicitly authorised to “access” in the traditional sense, which means policy has to cover context, retrieval scope, and tool invocation together. This is where NHI and agentic AI governance intersects with AI security: the model behaves like a non-human operator, but its privileges are distributed across identities, connectors, and content. Teams should define trust boundaries around the entire AI execution path.

Oversharing is a governance failure, not just a model quality issue. The article’s discussion of Copilot and Gemini-style exposure shows that sensitive data becomes reachable when content, permissions, and search paths are not aligned. That is the same structural problem seen in weak NHI governance: access exists somewhere in the stack, even if no human intended it to be used that way. Practitioners should align least privilege, data classification, and retrieval scoping so the model cannot surface what users should not see.

Continuous simulation is the only credible way to keep AI controls current. Red-team findings age quickly because models, connectors, and permissions all drift. The important shift is from proving a single system is safe to proving the governance loop still works after change. Practitioners should make repeat testing part of change management, access review, and AI rollout approvals.

Agentic AI security is now an identity problem as much as a model problem. Once an LLM can call tools, retrieve data, and chain actions, its effective privileges become operationally similar to a service account with broad access. That means identity governance, access review, and privilege scoping must extend to machine decision-makers. The practical conclusion is that AI security and NHI governance can no longer be managed as separate programmes.

What this signals

Oversharing drift is now a programme management problem, not just a model-risk finding. The issue changes as documents, permissions, prompts, and connectors change, so security teams need recurring assurance rather than one-time certification. For teams building AI governance, that means red-team evidence should feed access reviews, data classification decisions, and release gates, not just a backlog.

AI identity needs its own control boundary. Once an agent can retrieve data and call tools, it should be governed like a machine identity with narrowly scoped privileges and explicit ownership. That aligns directly with Top 10 NHI Issues thinking, because the risk is not just what the model knows, but what it is authorised to do with that knowledge.

Red-team findings also give security leaders a measurable way to justify AI rollout decisions. The most meaningful metrics are exposure drift, policy violations caught, and retest pass rates after permission changes, which are more operationally useful than simple model accuracy scores.

For practitioners

• Scope the full AI attack surface Inventory prompts, plugins, retrieval connectors, shared documents, and tool APIs before you test. Include the data domains the model can see and the identities it can act through, because the exposure often sits across layers rather than in one component.
• Test for inference-based leakage Red-team models for information that can be inferred, assembled, or resurfaced from benign-looking context, not just direct file access. This is where oversharing becomes visible, especially in copilots and RAG pipelines that combine search with generated output.
• Constrain model function calls Restrict which internal APIs, workflows, and permissions an agent can invoke, and verify those controls with adversarial prompts. If a model can call sensitive functions, treat that path like a privileged automation identity and monitor it accordingly.
• Make retesting part of change control Re-run red-team scenarios after model updates, permission changes, new data sources, or retrieval changes. Document the failure mode, remediation, and retest result so AI assurance can survive drift instead of resetting with every release.

Key takeaways

• AI red teaming exposes a governance gap that conventional penetration testing cannot see, because the model’s risk lies in what it reveals, infers, and does across connected systems.
• The evidence is stark: prompt injection, poisoned retrieval, and function misuse can succeed at high rates, which means AI assurance must be continuous rather than occasional.
• For IAM, NHI, and AI governance teams, the control objective is to constrain the model’s effective privileges, then prove those constraints still hold after every change.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating adversarial interactions against models, prompts, retrieval layers, and tools to uncover unsafe behaviour before attackers do. It focuses on leakage, manipulation, and misuse in probabilistic systems, where the same input can produce different results depending on context and surrounding data.
• Prompt Injection: Prompt injection is a technique that uses crafted text to override or steer an LLM’s intended behaviour. It can appear in user input, retrieved documents, or tool outputs, and it becomes especially risky when the model is connected to sensitive data or privileged functions.
• Vector-Store Poisoning: Vector-store poisoning is the insertion of malicious or misleading content into retrieval systems that feed an AI model. The aim is to influence what the model surfaces or believes, turning the knowledge layer into an attack surface that can distort answers or leak sensitive information.
• Function Misuse: Function misuse occurs when a model calls an API, workflow, or internal tool in a way the organisation did not intend. In enterprise AI, this is often an authorisation problem disguised as model behaviour, because the tool call can expose data or trigger privileged actions.

What's in the full article

Knostic's full analysis covers the operational detail this post intentionally leaves for the source:

• Step-by-step red-team workflow for scoping prompts, retrieval connectors, and tool APIs in enterprise AI.
• Detailed examples of prompt, retrieval, and function misuse tests that uncover specific leakage paths.
• Operational remediation guidance for tightening permissions, sensitivity labels, and retrieval controls.
• How the vendor maps red-team findings into governance workflows for Copilot and similar AI tools.

👉 Knostic's full post covers the attack vectors, workflow stages, and remediation patterns in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management for practitioners building access controls around AI systems. It helps security and identity teams align governance, review, and privilege management across human and non-human identities.