LLM red teaming exposes a new AI attack surface

At a glance

What this is: Lakera’s research frames LLM red teaming as a response to an attack surface where retrieved or embedded data can act like instructions and steer model output.

Why it matters: IAM and security teams need to treat LLM inputs, connected data sources, and assistant behaviour as governed attack paths, not just content sources or integration points.

Context

LLM red teaming is the practice of deliberately testing model-driven applications with adversarial inputs to see how they fail. In this article, the central governance problem is that LLMs do not reliably separate trusted instructions from untrusted external data, so the input boundary itself becomes part of the attack surface.

That matters for AI security programmes because the risk is not limited to a direct compromise of the model or its hosting stack. If external content can influence output, then search results, emails, documents, and other third-party data sources need identity-aware and behaviour-aware controls, not only conventional infrastructure safeguards.

Key questions

Q: How should security teams test enterprise LLMs for prompt injection risk?

A: Test the model inside the real application path, not in isolation. Include hidden instructions, malicious retrieved content, multi-turn escalation, and language variations, then measure whether the model reveals policy text, ignores constraints, or produces unsafe tool instructions. A model that passes simple prompts may still fail once context, memory, and retrieval are involved.

Q: Why do LLM applications create a different security review problem than ordinary web applications?

A: LLM applications change the review model because the model can launder untrusted input into output that appears safe but still drives a privileged action. In traditional apps, scanners can often rely on sanitisation patterns. With LLMs, reviewers must understand prompt sources, tool access, and downstream impact, because the model itself becomes part of the injection path.

Q: What breaks when an LLM application treats untrusted content as instruction?

A: Prompt injection works because the application collapses the line between data and control. Once retrieved text, memory, or user content can influence privileged prompts, the model may follow attacker-supplied instructions, expose sensitive context, or trigger tool calls that were never intended by the operator.

Q: Should organisations treat search results and email as part of AI security governance?

A: Yes. If an LLM can summarise, rank, or act on content from search or email, then those sources are part of the control environment. Governance has to cover content provenance, retrieval boundaries, and what the assistant is allowed to do with untrusted data.

Technical breakdown

Why external data becomes executable in LLM workflows

LLMs ingest heterogeneous content, then weight it during inference to produce a response. In the attack patterns described here, the model processes third-party text as candidate context and can follow instructions hidden inside that data. That breaks the normal assumption that only developer-authored prompts can shape execution. The issue is not a classic code execution bug. It is an instruction-separation failure, where retrieval, summarisation, and generation collapse into one trust boundary. Once that happens, malicious content can steer ranking, summarisation, or message composition without any direct system compromise.

Practical implication: Treat every retrieved or embedded input as potentially adversarial, even when it arrives through a trusted integration.

Prompt injection through search results and shared content

The article’s search example shows adversarial SEO, where a third-party site embeds hidden instructions that the model later retrieves. The assistant example shows the same pattern in workspace content, where invisible instructions in an email affect the assistant’s action. In both cases, the model is not merely reading content. It is interpreting data as if it were trusted instruction. That creates a control problem around source separation, content provenance, and whether the model is allowed to act on data that was never meant to be executable.

Practical implication: Separate read-only context from action-authorising context and test whether the model can be manipulated through untrusted content.

Why traditional patching logic does not map cleanly to LLM risk

Traditional security assumes a vulnerable component can often be fixed by patching logic, changing a rule, or isolating a bad endpoint. The article argues that LLM systems are different because the risky behaviour emerges from the model’s core capability to interpret flexible inputs. That means some attacks are not fixed by hardening the perimeter alone. Red teaming becomes a design control, because it reveals how the system behaves when adversarial content is folded into the reasoning loop. The security question shifts from 'can the system be reached?' to 'what can untrusted data make it do?'

Practical implication: Use adversarial testing to measure behavioural failure modes before deployment, not after a user or attacker discovers them.

Threat narrative

Attacker objective: The attacker wants the assistant or LLM application to produce misleading or malicious output that benefits the attacker while appearing legitimate to the user.

1. Entry occurs when the attacker places malicious instructions inside third-party content such as a web page or email that the LLM will later ingest.
2. Credential or access compromise is not required because the model itself processes the untrusted data and treats it as if it were actionable context.
3. Escalation occurs when the model follows the embedded instruction and produces a manipulated ranking, summary, or message containing the attacker’s desired effect.
4. Impact is the delivery of false or malicious output to the user, even though the underlying system was never directly breached.

Breaches seen in the wild

• Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM red teaming is a control for instruction-separation failure, not just prompt quality. The article shows that external content can steer model behaviour when the system does not keep developer instructions distinct from untrusted inputs. That is a governance problem because it changes what counts as a trusted control boundary. Practitioners should treat content provenance as part of AI security architecture, not a side issue.

Attack surface in LLM applications is defined by data pathways as much as by model weights. Search indexes, email, documents, and third-party websites can all become delivery paths for malicious instructions. This is why conventional perimeter thinking is too narrow for AI systems that reason over retrieved context. The practical conclusion is that AI threat modelling has to include untrusted content flows end to end.

Red teaming exposes behavioural failure modes that ordinary testing will miss. A model can appear functional in clean conditions and still fail when adversarial text is introduced into the input stream. That makes adversarial evaluation a programme requirement for any organisation that lets LLMs summarise, rank, or act on external content. Teams should measure how often the assistant obeys hostile instructions before they measure user experience.

The security-utility trade-off is now a governance constraint. The article notes that patching or retraining can degrade model performance, which means defenders cannot assume there is a simple hardening step that leaves capability untouched. The implication for identity and access governance is broader than LLMs alone: controls must be designed around behaviour under attack, not idealised behaviour in a clean lab.

Prompt injection is a named concept because it captures the core failure mode of this class of system. It is not merely bad prompting. It is the exploitation of a system that treats foreign content as instruction-bearing context. For practitioners, that means the question is whether the application can resist hostile instruction ingestion, not whether the model can generate fluent text.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: Red Teaming AI Agents for Identity Abuse

What this signals

Prompt injection turns content provenance into a security control. Once an LLM can be steered by retrieved or embedded text, the problem is no longer only model accuracy. Security teams need to govern which sources can influence decision-making, because untrusted content can change output without a traditional compromise.

The broader programme implication is that AI security must combine red teaming, source controls, and behavioural testing. That is especially true where assistants can summarise, recommend, or act on external data, because the attack surface lives in the relationship between context and action.

AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026. That trend reinforces the need to secure the entire AI stack, not just the model endpoint.

For practitioners

• Model untrusted content as executable input Inventory every place your LLM consumes third-party text, search results, emails, documents, or web pages, then classify which paths are allowed to influence output and which must remain read-only.
• Separate instruction channels from content channels Design prompts and tool flows so system instructions, user requests, and retrieved context are isolated and explicitly prioritised instead of blended into one conversation state.
• Red-team for prompt injection and content poisoning Test whether hidden instructions in retrieved data, emails, or documents can alter ranking, summarisation, or action selection under realistic attacker conditions.
• Set behavioural acceptance criteria before release Define what failure looks like for harmful instruction following, misleading output, and unsafe assistant actions, then gate deployment on those test results.

Key takeaways

• LLM red teaming is about exposing how untrusted content can steer model behaviour, not only whether the model is technically reachable.
• The article shows that search results, emails, and web content can all become instruction-bearing inputs when the system does not separate context from command.
• Practitioners should test adversarial behaviour before release, because clean-path accuracy does not reveal how the assistant will respond to hostile data.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Adversarial Red-Teaming: Adversarial red-teaming is the practice of actively trying to make a security model fail before it reaches production. The test uses crafted inputs, boundary probes, and mutation strategies to reveal weaknesses that ordinary accuracy testing will not show.
• Instruction Separation: Instruction separation is the design principle that keeps system rules, user requests, and external data in distinct trust zones. In LLM security, it is essential because once retrieved content can influence execution like a command, the model can be manipulated through the data it processes.
• Content Provenance: Content provenance is the practice of tracking where input came from and how trusted it should be before an AI system uses it. For agents, it helps separate instructions from retrieved or external data so malicious content is less likely to be treated as operational guidance.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.