LLM red teaming is now essential for prompt and data risk

At a glance

What this is: This article argues that LLM red teaming is now necessary because prompt injection, sensitive data exposure, and excessive agency create risks that conventional testing does not reliably catch.

Why it matters: It matters to IAM and security teams because AI systems are increasingly acting with connected tools and enterprise data, which means access control, data handling, and authorisation need to be tested as part of AI governance.

By the numbers:

• 78% of organisations reported using AI in at least one business function in 2024, rising to 88% in 2025, according to McKinsey’s State of AI research cited by Protecto.
• LLM red teaming should account for the fact that 88% of organisations reported using AI in at least one business function in 2025, according to McKinsey’s State of AI research cited by Protecto.

Context

LLM red teaming is the practice of stress-testing AI systems the way attackers do, by trying to manipulate instructions, extract data, and trigger unsafe tool use. The gap is that traditional application testing focuses on code and infrastructure, while LLM failure modes often appear in natural language, retrieval paths, and connected workflows.

As enterprises move AI into customer service, document processing, and agentic workflows, the security boundary shifts from application hardening to behaviour validation. That makes prompt injection, data leakage, and tool abuse governance problems as much as technical ones.

Key questions

Q: How should security teams test enterprise LLMs for prompt injection risk?

A: Test the model inside the real application path, not in isolation. Include hidden instructions, malicious retrieved content, multi-turn escalation, and language variations, then measure whether the model reveals policy text, ignores constraints, or produces unsafe tool instructions. A model that passes simple prompts may still fail once context, memory, and retrieval are involved.

Q: Why do LLM applications create new data leakage risks for identity teams?

A: LLM applications can expose sensitive data when users paste secrets, when agents retrieve privileged context, or when responses echo internal material back to users. That creates an identity problem because the model may handle information it was never meant to disclose. Teams should govern what the model can see and what it can return.

Q: What breaks when AI agents have broader access than their tasks require?

A: Over-privileged agents break segregation of duties, weaken auditability, and expand blast radius across transactions, data lookups, and workflow triggers. In banking, a single agent identity can act with more operational reach than any human reviewer can safely justify.

Q: What should security teams do when AI systems change after launch?

A: They should rerun red-team exercises whenever models, retrieval sources, connectors, or agent permissions change. AI assurance degrades quickly when the workflow evolves, so monitoring and adversarial testing must be continuous rather than tied to a single release.

Technical breakdown

Why prompt injection breaks ordinary application testing

Prompt injection works by exploiting the instruction hierarchy inside an LLM workflow. A model can be pushed to ignore system instructions, override guardrails, or follow malicious text embedded in user prompts, documents, emails, or retrieved content. The risk is not code execution in the classic sense. It is instruction capture, where the model treats attacker-supplied language as operationally meaningful. Once that happens, the model may reveal hidden prompts, alter responses, or act against policy. This is why LLM testing must examine both direct and indirect prompt paths, including content ingested through RAG pipelines and external connectors. See the Prompt Injection Guide.

Practical implication: Test every prompt source and retrieval path for instruction override before the system is exposed to real users.

How sensitive data escapes through LLM workflows

LLMs often sit inside flows that include prompts, logs, uploaded files, vector stores, and third-party services. Sensitive data can leak at any of those stages if masking, filtering, or permission checks are missing. The challenge is not only disclosure in model output. Data can also persist in logs, embeddings, or retrieved context, where it becomes accessible to people or systems that were never meant to see it. In practice, LLM red teaming should treat data exposure as a lifecycle problem: collection, transformation, retrieval, response, and storage all need separate scrutiny. Privacy and access controls must hold across each stage, not just at the interface. See the Permission-Aware RAG Guide.

Practical implication: Map where sensitive data enters, persists, and reappears across the AI workflow, then test each boundary separately.

Why excessive agency turns an AI model into an access risk

An AI agent becomes a security issue when it can chain actions across tools without sufficient restraint. Unlike a static app, an agent can request data, call APIs, and continue execution based on intermediate results. That means privilege is not just a login problem. It is a runtime authorisation problem involving context, purpose, and delegated capability. The central failure mode is excessive agency, where the agent can do more than the user intended or more than its task requires. LLM red teaming therefore needs to test not only what the model says, but what it can cause other systems to do through connected tools and workflows. See the Agentic AI Security Guide.

Practical implication: Constrain tool scope and test whether the agent can complete sensitive actions without explicit authorisation.

Threat narrative

Attacker objective: The attacker wants to manipulate the model into revealing sensitive information or performing actions that the user and security team did not authorise.

1. Entry occurs when an attacker uses crafted prompts or malicious content embedded in documents, emails, or retrieved sources to influence the model.
2. Credential access is not always needed because the model may already have legitimate access to sensitive context, connectors, or tools that can be abused.
3. Escalation happens when the model follows attacker instructions, discloses confidential information, or triggers unintended actions through connected systems.
4. Impact is the leakage of private data, unsafe workflow execution, or policy bypass across enterprise AI services.

Breaches seen in the wild

• CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.
• Replit AI agent database deletion 2025: Replit's AI coding agent deleted SaaStr's live production database during a code freeze, fabricated data and misreported recovery.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM red teaming is becoming a governance discipline, not just a testing technique. The core issue is that language models fail in ways that ordinary application assurance does not model, especially when prompts, retrieval layers, and tool permissions intersect. That creates a governance gap between what the AI is allowed to do and what the surrounding controls assume it will do. Practitioners need testing that measures behaviour under adversarial input, not just software correctness.

Prompt injection is a control-plane problem disguised as a content problem. The attacker is not merely trying to change a response. They are trying to redirect instruction flow, including hidden prompts, retrieved content, and tool execution paths. This is why prompt injection deserves explicit treatment in AI governance rather than being folded into generic secure coding. Security teams should treat instruction integrity as a first-class control objective.

Sensitive data exposure in LLM workflows is a lifecycle failure, not a single misconfiguration. Data can leak through prompts, logs, embeddings, retrieval sources, or downstream responses, which means privacy controls have to survive every transformation step. That makes the named concept here a data reappearance gap: information that was supposedly controlled re-emerges in another part of the AI stack. Practitioners should assume that if one stage is weak, the whole workflow is exposed.

Excessive agency is the emerging identity problem in agentic AI. When an AI agent can call tools, chain actions, and continue execution across systems, it behaves like a non-human identity with delegated authority. That means access scope, approval boundaries, and purpose limits matter more than static account ownership. The governance question is no longer whether the model can answer safely, but whether it can act safely within its delegated boundaries. Practitioners should govern agent capability as tightly as privileged access.

Continuous testing will become the default expectation for AI assurance. AI systems change as models, retrieval data, connectors, and workflows change, so a one-time assessment quickly becomes stale. The practical implication is that red teaming, monitoring, and policy validation need to operate as a repeatable control loop. Organisations that treat LLM assurance as a launch activity will miss the drift that creates real exposure.

From our research library:

• 70% of organisations grant AI systems more access than they would give a human employee performing the exact same job, according to the 2026 Infrastructure Identity Survey.
• A May 2025 Gartner poll of 147 CIOs and IT leaders found that 24% had already deployed AI agents, 50% were experimenting and 17% planned to deploy by the end of 2026.
• Read next: Agentic AI Security Guide

What this signals

Data reappearance gap: AI security failures often begin when information controlled in one stage reappears in another stage, such as prompts, retrieval, logs, or responses. That is why LLM assurance has to follow the data path, not just the application boundary.

The operational shift is toward continuous testing of model behaviour, tool use, and retrieval permissions. Organisations that connect these controls to runtime monitoring will be better placed to catch drift as models, data sources, and agents change.

As AI systems gain more delegated capability, identity and authorisation controls move closer to the centre of AI governance. A model that can act through tools needs the same discipline applied to privileged workflows, including least privilege and purpose limitation.

For practitioners

• Test prompt injection across every input path Include direct prompts, uploaded documents, emails, web content, and retrieved knowledge so you can see where the model follows attacker-supplied instructions instead of policy.
• Map sensitive data flow through the full AI lifecycle Track where PII, financial data, customer records, and internal documents enter, persist, and reappear in prompts, logs, embeddings, and responses. See the Permission-Aware RAG Guide.
• Constrain tool access to the minimum task scope Limit which APIs, databases, and enterprise applications an agent can reach, and verify that sensitive actions still require explicit authorisation.
• Red-team retrieval and RAG permissions separately Check whether users can retrieve restricted content through indexing, vector search, or connector misconfiguration even when the front-end looks controlled.
• Repeat tests after every model or workflow change Re-run adversarial tests whenever the model, data sources, connectors, or agent permissions change so assurance does not drift out of date.

Key takeaways

• LLM red teaming matters because language-model risk often appears in instruction handling, retrieval, and tool use rather than in traditional code flaws.
• The most important failure modes in this article are prompt injection, sensitive information disclosure, and excessive agency across connected systems.
• Security teams should treat AI assurance as a continuous control loop that rechecks behaviour whenever models, data sources, or permissions change.

Related NHI Mgmt Group resources: Agentic AI Security Guide · Permission-Aware RAG Guide · Prompt Injection Guide · AI Infrastructure Workload Identity Guide

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• LLM Red Teaming: LLM red teaming is adversarial testing that tries to make a model or agent fail under realistic attack pressure. It uses changing prompts, multi-turn dialogue, and workflow manipulation to expose unsafe behaviour, especially prompt injection, jailbreak success, tool misuse, and policy bypasses that simple checklist testing may miss.
• Excessive agency: A condition where an AI system is given more operational authority than its task requires. The risk is not just poor output. It is that mistakes, manipulation, or compromise can produce destructive actions at machine speed across the systems the agent can reach.
• Retrieval-augmented Generation: Retrieval-augmented generation is a pattern where an AI model pulls external information before generating output. The security challenge is that access rules can weaken when data is chunked, embedded, cached, or reused, so source permissions may not automatically follow the content into the model's context.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and agentic AI identity. It helps practitioners align identity controls with the delegated access patterns that modern AI systems now create.