TL;DR: Prompt injection is a security problem, not a content problem, and using one LLM to judge another creates a recursive defense that fails quietly under adaptive attack, according to Lakera. Security boundaries need deterministic enforcement, because the model being protected and the model making the decision share the same adversarial weakness.
At a glance
What this is: This is an analysis of why LLM-as-a-judge is a weak prompt injection defense and why the defense breaks when the security boundary is another LLM.
Why it matters: IAM and security teams governing AI systems need deterministic enforcement boundaries, because recursive model-based screening creates false confidence, weak auditability, and a larger attack surface for agent and model access decisions.
Context
Prompt injection is a security problem, not a content-quality problem. It matters because an LLM cannot reliably distinguish trusted from untrusted instructions when both arrive as text, which breaks the assumption that a model can safely judge another model’s inputs.
The governance gap is not simply bad prompting. It is the lack of a boundary that is independent of the model under attack, which becomes more serious when LLMs sit in front of production systems, RAG pipelines, or agent workflows.
For AI security programmes, this is a control-design issue: if the same failure mode can affect both the protected model and the judge, then the defence is recursive rather than independent.
Key questions
Q: What breaks when an LLM is used to judge prompt injection risk?
A: The control breaks because the judge is exposed to the same language manipulation as the model it is meant to protect. That creates a recursive defense with no hard trust boundary, so a well-phrased attack can influence the security decision itself. In practice, the system may keep running while unsafe prompts slip through unnoticed.
Q: Why do LLM-based prompt defenses fail in real-world attacks?
A: They fail because the judge is exposed to the same adversarial techniques as the protected model. Attackers can hide intent in retrieved content, long prompts, or multi-step workflows, and the system may forward malicious input with no obvious alert. Static tests often miss this because they do not reproduce adaptive behaviour.
Q: How should security teams set boundaries for AI-assisted decisions?
A: Security teams should separate tasks AI can accelerate from decisions that carry accountability, approval, or risk acceptance. Use written decision classes for access, exceptions, and customer-impacting actions. If the final call changes people, privileges, or policy, keep a human in the loop with recorded evidence of review.
Q: What signs suggest a prompt defense is giving false confidence?
A: Common signs are clean-looking logs, good results on known jailbreak sets, and inconsistent decisions when prompts are slightly reworded or wrapped in longer context. If the defence works in demos but not across indirect prompts, retrieved documents, or multi-step agent flows, it is behaving like a classifier without a stable security boundary.
Technical breakdown
Why an LLM judge shares the same attack surface
An LLM judge is still a generative model, so it processes instructions probabilistically and can be steered by phrasing, context, and adversarial framing. In prompt injection scenarios, that means the supposed control and the protected model both interpret text through the same vulnerable mechanism. This is why a binary safe or unsafe decision can be manipulated by hidden instructions in prompts, retrieved documents, or third-party content. The weakness is architectural, not just operational. If the enforcement layer can be influenced by language, it is part of the attack surface rather than a barrier to it.
Practical implication: separate the security decision from any model that interprets user text.
Why demo success does not equal runtime security
LLM-based prompt filters often look effective in static tests because they are evaluated against known jailbreak examples, not adaptive adversaries. In production, attackers can vary wording, hide malicious intent in long context, or defer the harmful instruction until later in the interaction. That makes the judge’s classification inconsistent and difficult to audit. The dangerous part is quiet failure: the system keeps operating, logs look normal, and no obvious signal tells teams the control has been bypassed. For security programmes, this is the difference between a test harness and an enforced boundary.
Practical implication: evaluate prompt defenses against adaptive attacks, not fixed red-team samples.
Deterministic classification as the control boundary
Purpose-built classifiers do one job: determine whether an input violates policy without trying to comply with it or reason like the application model. That difference matters because deterministic behaviour is auditable, repeatable, and cheaper to operate inline than a second generative call on every request. In NHI terms, this is a boundary control problem, not a moderation problem. The control has to be independent of the system it protects, especially when prompts can influence downstream tool use, data exposure, or agent action. The result is a cleaner trust model for AI access control.
Practical implication: place non-LLM enforcement in front of model and agent access decisions.
Threat narrative
Attacker objective: The attacker aims to bypass prompt filtering so the production model or agent will comply with malicious instructions or leak sensitive information.
- Entry occurs when malicious instructions are embedded in user prompts, retrieved documents, or third-party content that reaches the model stack.
- Credential or policy-like context is then harvested indirectly when the judge model interprets the text and treats the attack as ordinary language.
- Escalation happens when the same weakness affects both the protected model and the judging model, allowing the attacker to steer the security decision itself.
- Impact follows when the system forwards unsafe prompts or agent actions, leading to unauthorized actions or sensitive data exposure.
Breaches seen in the wild
- EchoLeak (Microsoft 365 Copilot) 2025: A crafted email could make Microsoft 365 Copilot leak data from its context with no click, a zero-click prompt injection fixed as CVE-2025-32711.
- ForcedLeak (Salesforce Agentforce) 2025: A critical Agentforce flaw let attackers steer an AI agent through a Web-to-Lead form to leak CRM data via an expired allowlisted domain.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Recursive model enforcement is a boundary failure, not a tuning problem: when the judge and the judged system share the same language-driven vulnerability, security control becomes self-referential. That breaks the basic assumption that enforcement should sit outside the system being protected. The implication is that AI security teams must treat policy enforcement as an independent trust boundary, not as another prompt.
Prompt injection is closer to identity and authorisation failure than to content moderation: the attacker is not just changing output style, but altering what the system is allowed to do. That makes this a control-plane issue for AI applications, especially when prompts can reach tools, data, or delegated actions. Practitioners should think in terms of governed access, not simply model behaviour.
LLM-based judges create false confidence because they optimise for apparent coverage: they can seem broad, cheap, and model-agnostic while still failing under adaptive attacks. That pattern is especially dangerous in production AI because the failure mode is quiet and hard to detect. Security programmes need controls that fail closed, not controls that merely look intelligent.
Deterministic security boundary: the article’s core concept is that enforcement must be separate from interpretation. A prompt defense that can be influenced by the same text it is evaluating is not a boundary, only another opinion. Teams should design for deterministic policy decisions before language interpretation enters the stack.
AI security now depends on who gets to decide, not only what the model says: when AI systems can trigger tools or actions, the governance question becomes whether access decisions are made by something the attacker can steer. That is why prompt defense belongs in identity-aware control design, not in free-form model judgment. The practitioner task is to remove decision authority from the vulnerable layer.
From our research library:
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
- Read next: Red Teaming AI Agents for Identity Abuse
What this signals
Deterministic security boundary: model-based screening is useful only when the control cannot be influenced by the same text it is evaluating. Once a judge can be steered by prompt content, AI security shifts from enforcement to interpretation, which is a weaker and harder-to-audit trust model.
Prompt injection also changes how teams should think about AI governance. The practical question is no longer whether the model understands the policy, but whether any untrusted input can reach tool execution, data retrieval, or autonomous action without a non-LLM gate in front of it.
For practitioners
- Separate enforcement from interpretation Place a non-LLM policy layer in front of any model or agent that can receive untrusted text, and keep the judge model out of the security decision path.
- Test against adaptive prompt injection Red-team with indirect, embedded, and deferred instructions inside long prompts, retrieved content, and multi-step workflows instead of only using static jailbreak examples.
- Audit downstream tool exposure Map which prompts can reach tool calls, data retrieval, or action execution, then restrict those paths so an unsafe interpretation cannot become an unauthorized action.
- Instrument for quiet failure Monitor for cases where the model continues to answer normally after suspicious context appears, because silent bypass is the main operational hazard in judge-based defenses.
Key takeaways
- Prompt injection is a boundary problem, not a moderation problem, because the judge model can be manipulated in the same way as the protected model.
- The defence can appear to work in demos while failing under adaptive attacks, which makes quiet bypass the main operational risk.
- A deterministic policy layer in front of model access is the control that changes the trust model, because it does not share the model’s own language weakness.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATT&CK address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | The article is about AI systems being steered into unsafe actions through compromised judgment. |
| ASI02 — Tool Misuse | The defense fails when malicious input reaches tools or delegated actions through the model path. | |
| Recommendation — Treat prompt injection as identity and privilege abuse and remove decision authority from the vulnerable model layer. Constrain tool exposure so prompt-driven decisions cannot trigger unauthorized tool use. | ||
| NIST AI RMF | GOVERN — AI Governance and Accountability | The article centers on governance boundaries and who enforces policy in AI systems. |
| Recommendation — Define accountable AI control boundaries and keep enforcement independent from the model under supervision. | ||
| NIST CSF 2.0 | PR.AA-05 — Access Permissions, Entitlements and Authorizations | The core issue is whether untrusted input can reach privileged model, tool, or data paths. |
| Recommendation — Apply access and authorization controls to AI inputs and downstream actions so policy cannot be bypassed by language. | ||
| MITRE ATT&CK | TA0006;TA0008 — Credential Access; Lateral Movement | The article’s attack pattern centers on misuse of trusted context to reach broader system capability. |
| Recommendation — Map prompt injection paths to credential access and lateral movement so detection covers downstream abuse. | ||
Key terms
- Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
- Deterministic Enforcement: A policy model that produces a clear allow or block decision based on defined rules rather than probabilistic model judgment. For AI agents, deterministic enforcement is valuable because it can stop risky execution even when language is obfuscated, multi-turn, or designed to evade classifier-based guardrails.
- Immutable Security Boundary: A control limit that the agent or user cannot alter on its own. The concept matters when a system can modify network settings, tokens, or approvals, because those actions should be enforced by infrastructure the agent cannot touch or override.
- Recursive Defense: A recursive defense is a control structure where the protected system and the enforcement layer share the same weakness. In AI security, this happens when one LLM is asked to judge or constrain another LLM, creating a loop that can fail quietly under the same prompt-based attack.
Deepen your knowledge
NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.
Published by the NHIMG editorial team on July 5, 2026.
Updated on October 11, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org