LLM red teaming exposes a new AI attack surface

At a glance

What this is: This is a research post on why LLM red teaming must treat external data as an attack vector and why AI applications fail differently from traditional software.

Why it matters: It matters because IAM, NHI, and AI governance teams need controls that account for instruction smuggling, third-party data exposure, and runtime behaviour rather than only static permissions.

By the numbers:

• 96% of technology professionals identify AI agents as a growing security threat, and 66% believe this risk is immediate.

👉 Read Lakera's research on building a superhuman AI red teaming agent

Context

LLM red teaming is the practice of trying to make AI systems fail on purpose so their hidden attack paths become visible before real attackers exploit them. In identity terms, the hard problem is not only access control, but how models interpret third-party data, tool outputs, and user input as instructions that can alter behaviour.

That matters for AI governance because the model is not just processing data, it is acting on it. Once external content can influence decisions, the security question shifts from who can reach the system to what the system can be induced to do, which is why AI attack surface management belongs beside IAM and NHI governance.

Key questions

Q: How should security teams test LLM applications for prompt injection?

A: They should test the full input chain, not just the chat interface. That means placing malicious instructions in retrieved webpages, emails, documents, and tool responses, then checking whether the model follows them or safely ignores them. The goal is to prove the system can separate untrusted content from authorised instruction before any business action occurs.

Q: Why do LLM applications create a larger attack surface than traditional software?

A: LLM applications can treat external data as actionable context, which means attackers may influence behaviour without exploiting a code flaw or gaining direct access. Traditional software usually separates input from execution more cleanly. In LLM systems, the boundary is weaker, so content provenance and instruction handling become security controls, not just usability details.

Q: What do security teams get wrong about red teaming AI systems?

A: They often stop at a small set of static prompts and miss multi-step attacks that adapt to model responses. Real adversaries will vary techniques, combine poisoned content with retrieval abuse, and pursue the most effective path to influence. Good red teaming measures breadth, persistence, and the ability to find new failure modes.

Q: How can organisations keep LLMs from triggering unsafe actions?

A: They should insert a hard control gate between model interpretation and any privileged action, especially when tools, workflows, or secrets are involved. The model can suggest or draft, but a policy layer must decide whether the action is allowed. Without that separation, adversarial content can turn a model into an execution path.

Technical breakdown

Why LLMs turn data into an attack surface

Traditional applications keep instructions and data separate. LLM systems blur that boundary because the model may read retrieved webpages, emails, documents, or tool output and then follow embedded instructions inside them. That is why prompt injection works: the attacker does not need to break the model, only to place adversarial text where the model will process it as context. The security problem is structural, not just a bad prompt. Practical implication: teams need testing that targets how models treat untrusted context, not just how they answer benign queries.

Practical implication: Build adversarial test cases around retrieved content, email text, and tool output before trusting LLM workflows.

Adversarial SEO and poisoned retrieval in LLM applications

When an LLM uses search results or retrieved pages to answer a question, an attacker can seed those sources with hidden instructions or false claims. The model may then rank a fake product, surface a phishing link, or suppress the right answer because the retrieved content has become part of the decision path. This is a retrieval-layer failure mode, not a model-only flaw. Practical implication: search, retrieval, and summarisation pipelines need validation rules that treat untrusted external content as hostile until proven otherwise.

Practical implication: Inspect retrieval pipelines for hidden instructions, deceptive rankings, and content provenance gaps.

Why automated red teaming has to be agentic, not scripted

Lakera’s framing points to a deeper challenge: effective red teaming requires generating many variations of attacks, adapting to model responses, and pursuing multi-step failure modes. That means the tester must reason over outcomes, not just replay a fixed checklist. In practice, this pushes AI red teaming toward agentic behaviour, but only if the system can choose actions, select tools, and time those actions independently. Practical implication: evaluate red teaming systems on adaptive coverage and containment, not on static prompt libraries.

Practical implication: Measure whether red team workflows can adapt attack paths and coverage dynamically across sessions.

NHI Mgmt Group analysis

LLM red teaming is a governance discipline, not just a testing technique. The article shows that AI applications fail when untrusted data can steer model behaviour, which means the real issue is whether the organisation can distinguish input from instruction at runtime. That is an identity and access problem as much as a model-safety problem. Practitioners should treat adversarial evaluation as part of the control plane, not a one-off security exercise.

Instruction smuggling is the named failure mode this article exposes. The system breaks when externally supplied text is allowed to influence execution paths without a reliable boundary between context and command. That assumption was designed for structured software inputs, but it fails in LLMs because the model can reinterpret the content it receives. The implication is that teams must rethink how trust is assigned to content, not just how access is granted.

AI attack surface management now belongs alongside NHI governance. The moment an AI system can consume emails, search results, or documents and act on them, it behaves like a non-human actor operating across multiple trust zones. That makes provenance, context handling, and action authorization part of the same governance conversation that already covers service accounts and workload identities. Practitioners should align AI testing with identity security rather than leaving it siloed in application security.

Superhuman red teaming will be judged by adversarial breadth, not model elegance. The article is explicit that the goal is to surpass human ingenuity, but the security value comes from whether the system can explore more failure modes than a human team can sustain. That shifts evaluation toward coverage, reproducibility, and adversarial persistence. Security leaders should expect red teaming capability to be measured by what it exposes, not by how polished it looks.

OWASP Agentic Applications Top 10 and NIST AI governance frameworks are the right reference points for this problem space. The article’s examples map directly to prompt injection, tool misuse, and external data poisoning, which are now standard concerns in agentic AI security. That means AI assurance should be framed through established agentic threat models, not improvised red team scripts. Practitioners should use recognised frameworks to define test scope and residual risk.

From our research:

• 96% of technology professionals identify AI agents as a growing security threat, and 66% believe this risk is immediate, according to AI Agents: The New Attack Surface report.
• Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation, according to SailPoint.
• For a practical next step, review OWASP Agentic Applications Top 10 to map prompt injection, tool misuse, and external-data abuse to test cases.

What this signals

Instruction smuggling is now a programme-level risk: when external content can influence model behaviour, the security boundary moves from the prompt box to the entire retrieval and tool chain. That is why AI governance needs provenance tracking, policy gates, and adversarial testing embedded in delivery, not bolted on after launch.

With 80% of organisations reporting AI agents acting beyond intended scope in SailPoint's research, the next phase of governance will be judged by whether teams can observe, restrict, and explain agent behaviour under attack. The control gap is no longer theoretical.

For teams building agentic workflows, the practical question is whether the red team can expose failure paths faster than production can generate them. If the answer is no, the operating model is already behind the threat.

For practitioners

• Test untrusted context handling Build red team cases that embed malicious instructions in webpages, emails, documents, and retrieved search results, then verify the model ignores them when they conflict with system intent.
• Validate retrieval provenance Track where retrieved content came from, whether it was user supplied, third party, or internally curated, and block high-risk actions when provenance is unknown or weak.
• Separate content from command Enforce parsing and policy layers so LLM outputs cannot directly trigger privileged actions without an explicit control gate between interpretation and execution.
• Use adversarial benchmarks for red teaming Measure whether your testing workflow can vary prompts, chain actions, and adapt after failures instead of relying on a fixed prompt set that only finds known weaknesses.

Key takeaways

• LLM applications are vulnerable because external content can be treated like executable instruction, which breaks the old input-output security model.
• AI red teaming has to explore adversarial behaviour across retrieval, email, search, and tools, because static prompts will miss the real failure modes.
• The right control response is not only model hardening, but hard separation between interpretation and any privileged execution path.

Key terms

• Prompt Injection: Prompt injection is an attack where malicious instructions are embedded in content that an AI system reads and may follow. The weakness is not in the user interface alone, but in the model's ability to blur the line between trusted instructions and untrusted external text.
• Retrieval Attack Surface: Retrieval attack surface is the set of external sources an AI system consults before generating a response or taking action. When those sources are untrusted, poisoned, or manipulated, the model can inherit the attacker's content as part of its decision context.
• AI Red Teaming: AI red teaming is adversarial testing that tries to make an AI system fail under realistic attack conditions. It goes beyond benign evaluation by using deceptive inputs, chained prompts, and hostile content to reveal how the system behaves under pressure.

What's in the full article

Lakera's full research covers the operational detail this post intentionally leaves for the source:

• Illustrated examples of adversarial SEO and workspace assistant attacks showing how the model is manipulated step by step.
• The series roadmap for building a superhuman red teaming agent, including the benchmark design the post says will be released publicly.
• A deeper discussion of why LLMs create an attack surface that differs from classic code-execution vulnerabilities.
• The article's own framing of why AI is required to discover and generate effective attacks at scale.

👉 Lakera's full post expands on the attack examples, benchmark goals, and why LLM vulnerabilities differ from traditional security flaws.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an identity security programme, it is worth exploring.