LLM-as-a-judge fails as prompt injection defense for AI systems

At a glance

What this is: This analysis argues that LLM-as-a-judge is the wrong security boundary for prompt injection because the defender inherits the same weaknesses as the system it is meant to protect.

Why it matters: It matters because IAM, NHI, and agentic AI programmes need deterministic policy enforcement around model access and action, not probabilistic judgement layered inside the attack surface.

👉 Read Lakera's analysis of why LLM-as-a-judge fails at prompt injection defense

Context

Prompt injection is the failure mode where untrusted text changes model behaviour. In AI security programmes, that matters because the control boundary is not the model’s answer quality but whether the system can keep instructions, data, and actions separated under adversarial pressure. LLM-as-a-judge blurs that boundary by asking one model to police another.

The governance problem is broader than LLM moderation. Any security design that relies on the same probabilistic system to interpret and enforce policy creates a trust loop that can be steered by the attacker. For identity teams, this is the same structural question that appears in NHI and agentic AI governance: who, or what, is allowed to decide that a request is safe before it reaches production logic?

Key questions

Q: How should security teams defend against prompt injection in AI systems?

A: Use deterministic controls outside the model to decide what can reach the application, tools, or memory. A generative model can help classify risk, but it should not be the final security boundary. The best designs combine input screening, scoped tool access, and auditable policy enforcement before any action is taken.

Q: Why do LLM-based prompt defenses fail in real-world attacks?

A: They fail because the judge is exposed to the same adversarial techniques as the protected model. Attackers can hide intent in retrieved content, long prompts, or multi-step workflows, and the system may forward malicious input with no obvious alert. Static tests often miss this because they do not reproduce adaptive behaviour.

Q: What breaks when a model is allowed to grade its own prompts?

A: The trust boundary breaks. A model that interprets instructions cannot reliably police whether those instructions are malicious, because the attacker can steer that interpretation. Once the judge is part of the attack surface, failures become recursive and quietly propagate into production.

Q: How should teams decide where to place prompt injection controls?

A: Place hard controls at the point where untrusted text could influence tools, data, or execution. That means non-LLM enforcement before the model, plus separate governance for tool permissions and retrieval. If a control can be steered by the prompt it is meant to inspect, it is not a boundary.

Technical breakdown

Why LLM judges inherit the same prompt injection weakness

An LLM judge is still a language model, so it interprets text probabilistically rather than enforcing hard security rules. That means an attacker can target the judge with the same techniques used against the production model, including indirect injection through retrieved content, long prompts, or multi-step agent workflows. The result is recursive risk: the control and the protected system share the same failure mode. In security terms, the boundary is soft, so the attacker only has to shape interpretation, not break a deterministic rule.

Practical implication: do not rely on a generative model as the final policy gate for prompts that can influence tools, data, or downstream actions.

Why demo success does not prove runtime security

LLM-based judges often look effective in static tests because they are evaluated against a fixed prompt set, not against adaptive adversaries. In production, the attacker can rephrase, embed, or delay malicious intent until the judge has already given a safe verdict. That is why these systems fail quietly. They do not necessarily crash or throw obvious alerts. They simply forward the wrong content with confidence, which is worse for security operations because it delays detection and weakens trust in the control plane.

Practical implication: test prompt defenses with adaptive red-team inputs and indirect injection patterns, not just known jailbreak strings.

Why deterministic enforcement has to sit outside the model

A reliable prompt defense has to enforce policy without following instructions. That is why purpose-built classifiers are structurally different from generative judges. They do not reason about the prompt in the same way, so they are less exposed to instruction hijacking and more consistent under replay. This separation matters for AI security architecture: the system that interprets policy should not be the system that is being policed. The stronger design is layered enforcement, where the model can assist with context but cannot authorise its own boundary crossings.

Practical implication: place non-LLM enforcement in front of models that can call tools, retrieve data, or trigger autonomous actions.

Threat narrative

Attacker objective: The attacker aims to manipulate model behaviour indirectly so the system violates its own policy boundaries without triggering a security block.

1. Entry occurs when an attacker submits a prompt, retrieved document, or third-party text that hides malicious instructions inside otherwise plausible content.
2. Credential or control access occurs when the judge model misclassifies the input as safe and forwards it into the production LLM or agent workflow.
3. Impact follows when the protected model leaks sensitive data, overrides instructions, or triggers an unauthorized action through tools or downstream systems.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM-as-a-judge is a control-plane mistake, not just a model-quality mistake. The core failure is architectural: the same probabilistic system is being asked to interpret policy and enforce policy. That collapses the trust boundary and makes the defense attackable through phrasing, context, and retrieval. The implication is that prompt security cannot be built by layering more language interpretation on top of language interpretation.

Prompt injection is the clearest proof that security controls must be deterministic at the boundary. A model can help classify risk, but it cannot be the authority that decides whether untrusted text may reach tools, memory, or downstream execution. This is the same lesson identity teams already apply in NHI governance: the component that can act should not be the component that blesses its own action path. Practitioners should treat model enforcement as policy execution, not model opinion.

Purpose-built classifiers create a different failure surface than generative judges. They are not magic, but they remove instruction-following from the control point and therefore reduce recursive vulnerability. That makes them closer to a real security boundary than an LLM that can be steered by the attacker it is supposed to stop. The practical conclusion is simple: keep the model inside the system, not on top of the system.

Recursive boundary collapse: This pattern shows what happens when the security control shares the same interpretive weakness as the workload it guards. In AI security terms, that is not a tuning issue; it is a governance premise failure. Security teams should rethink where authority lives in the stack before they expand agentic workflows further.

The more autonomous the workflow, the less acceptable it is to let the model arbitrate its own safety. Once a model can select tools or trigger actions, any false negative becomes an execution event rather than a mere classification miss. That elevates prompt defense from content filtering to identity and access control for machine behaviour. Practitioners should treat model-side judgement as advisory only.

From our research:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%), according to AI Agents: The New Attack Surface report.
• Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation.
• That blind spot matters because OWASP Agentic AI Top 10 shows why policy enforcement must sit outside the model, not inside it.

What this signals

Recursive boundary collapse: once a model is asked to police its own prompts, the programme inherits a trust problem rather than a content-filtering problem. That is why teams should shift from “does the model recognise bad text” to “where is the hard boundary that stops untrusted text from becoming action.”

The governance signal is clear: AI security is moving from prompt moderation toward identity control for machine behaviour. With 80% of organisations already seeing AI agents act beyond intended scope, according to AI Agents: The New Attack Surface report, the operating assumption that model output can be safely interpreted after the fact no longer holds.

Teams that are serious about agentic AI should align prompt controls with NIST AI 600-1 Generative AI Profile and the broader NIST Cybersecurity Framework 2.0. That means enforcing policy before tool access, not relying on the model to self-assess compliance.

For practitioners

• Move enforcement outside the model Use deterministic policy checks before prompts can reach a model that may call tools, retrieve data, or trigger actions. Keep generative judgement for advisory classification, not final allow or deny decisions.
• Test against indirect and adaptive injection Red-team long prompts, retrieved documents, and multi-step agent paths. Validate that the control holds when malicious intent is hidden, reframed, or delayed until later in the interaction.
• Separate policy interpretation from policy enforcement Design the control stack so the system that explains behaviour is not the same system that authorises behaviour. This reduces recursive failure and makes audit decisions more defensible.
• Treat tool access as an identity boundary Apply explicit approval and scoped entitlements before any model can reach secrets, APIs, or external systems. The boundary should be enforced where action becomes possible, not where text is merely classified.

Key takeaways

• LLM-as-a-judge is a recursive security pattern because the defender shares the same prompt-injection weakness as the system it protects.
• Real-world prompt defense must be deterministic at the boundary, or adaptive attackers will eventually steer the control into a false safe decision.
• As AI systems gain tool access and autonomy, prompt enforcement becomes an identity and access problem, not a language-only problem.

Key terms

• Prompt Injection: A prompt injection is malicious text that changes how a model behaves by exploiting the fact that it cannot reliably separate trusted instructions from untrusted content. In practice, it can alter outputs, reveal data, or trigger unwanted tool use when the model sits too close to the security boundary.
• LLM-as-a-Judge: LLM-as-a-judge is a pattern where one language model evaluates another model’s output or input. It is useful for offline scoring and classification, but risky as a runtime control because the judge itself remains steerable, probabilistic, and exposed to the same adversarial techniques as the protected system.
• Deterministic Enforcement: Deterministic enforcement means a security control makes the same allow or deny decision for the same input, without depending on language interpretation or model reasoning. In AI systems, this is the property that turns a policy into an actual boundary rather than a best-effort opinion.
• Recursive Risk: Recursive risk occurs when a control inherits the same weakness as the system it is meant to guard. In AI security, that happens when a generative model is asked to police another generative model, creating a loop where both can be manipulated by the same prompt-based attack.

What's in the full article

Lakera's full article covers the operational detail this post intentionally leaves for the source:

• A fuller walkthrough of the recursive failure pattern behind LLM-as-a-judge and why it looks stable in demos.
• Examples of prompt injection defence patterns that use classifiers instead of generative models at the enforcement layer.
• A detailed comparison of runtime guardrails, offline evaluation uses, and where LLM judgement is still appropriate.
• Operational tradeoffs around latency, cost, and consistency when screening prompts at production speed.

👉 Lakera's full article covers the recursive failure mode, classifier-based enforcement, and deployment tradeoffs.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or governance in your organisation, it is worth exploring.