Prompt injection detection still fails at the wrong metric

At a glance

What this is: ARMO argues that prompt injection detection should be judged by production-operating metrics, not model-card scores, because the same detector can look worse on AUC while performing far better at the false-positive threshold that matters.

Why it matters: IAM, NHI, and agentic AI teams need deployment metrics that reflect real agent loops, because weak measurement leaves credentials, tool calls, and workflow continuity exposed to the wrong control.

Context

Prompt injection detection is a control problem, not a benchmark problem. A detector can score text well and still fail where it matters if the operating point, the turn being inspected, or the false-positive budget does not match production reality.

For agents, that gap is an identity issue as much as a security issue. These controls sit in front of systems that already hold credentials and can take action, so teams have to govern the residual risk left after the detector makes its decision.

ARMO’s article argues that the right question is not whether a detector has a strong headline score, but whether it can be measured at the point where the agent loop actually consumes or executes untrusted text.

Key questions

Q: What breaks when prompt injection detection is judged by AUC alone?

A: AUC can improve or decline without telling you whether the detector works at the threshold your production system can tolerate. Teams end up optimising a ranking score instead of the false-positive budget, throughput, and residual exposure that actually determine whether the control survives deployment.

Q: Why do prompt injection controls need to be measured on different agent turns?

A: Because each turn has a different risk profile and message volume. A detector on user input may miss attacks arriving through retrieved context or tool output, while a detector on every turn can become too expensive or too noisy to keep enabled.

Q: How should teams evaluate prompt injection detectors before deployment?

A: Evaluate them at the false positive rate your workflow can absorb, using examples from your own traffic rather than a public benchmark. The key metric is recall at the chosen threshold, not headline AUC. If the detector looks good in theory but creates too many broken tasks in production, it is the wrong control for your environment.

Q: What should security teams do first when indirect prompt injection is a concern?

A: Start by mapping the full agent trace, not just the prompt. Identify every place untrusted text can enter the reasoning path, then restrict the tools and data sources reachable from that path. If you only scan user input, you will miss the channels that matter most in production agent workflows.

Technical breakdown

Why AUC misleads prompt injection deployments

AUC is a ranking metric, so it measures how well a classifier orders benign and malicious inputs across every possible threshold. Deployment is different: the team picks one threshold, then has to live with the resulting false positives and false negatives. A detector can therefore improve the operating point that matters while its overall AUC stays flat or even drops. In practice, prompt injection controls are judged on recall at the false-positive rate the business can absorb, not on a generic model-card score. That is why the same detector can look like a regression in a paper table and like a better control in production.

Practical implication: evaluate prompt injection detectors at the exact false-positive budget and threshold you expect to run in production.

Why placement in the agent loop changes the measurement

A prompt injection detector can sit on the user message, retrieved content, tool output, or tool call. Those turns are not equivalent. Input scoring is low volume but often misses payloads that arrive through retrieval or tools, while scoring every retrieved chunk or tool response multiplies the message count and changes latency, false positives, and cost. That means the same model can be acceptable in one rail and unusable in another. The detector is not the whole control; the turn it inspects is part of the control boundary.

Practical implication: measure detection performance separately for each turn type before deciding where to enforce the guardrail.

What residual risk means when a detector cannot block everything

Prompt injection detectors are probabilistic controls, so they never eliminate the attack surface. The residual is the set of instructions that still pass through, and every downstream safeguard has to assume those inputs may reach a model with tools, credentials, or delegated authority. That changes the governance question from perfect prevention to containment. If the agent can still act on the payload after the classifier fires, the real control objective becomes limiting blast radius, not pretending the classifier has closed the loop.

Practical implication: define containment controls for the residual path, not just detection quality targets.

Breaches seen in the wild

• EchoLeak (Microsoft 365 Copilot) 2025: A crafted email could make Microsoft 365 Copilot leak data from its context with no click, a zero-click prompt injection fixed as CVE-2025-32711.
• ForcedLeak (Salesforce Agentforce) 2025: A critical Agentforce flaw let attackers steer an AI agent through a Web-to-Lead form to leak CRM data via an expired allowlisted domain.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Prompt injection detection is a governance problem before it is a model problem: the useful question is whether the control measures what production exposure actually looks like. AUC can describe rank ordering, but production teams have to govern false positives, throughput, and residual access in the agent loop. The implication is that security programmes should treat benchmark headlines as context, not as evidence of deployability.

Placement is part of the control surface: the same detector behaves differently on user input, retrieved text, tool output, and tool arguments. That means the effective security boundary is not the model alone but the turn where it is applied. Practitioners should stop thinking of prompt injection detection as a single switch and start treating it as a set of measurable enforcement points.

Turn-level measurement gap is the real blind spot: model cards rarely tell teams how a detector behaves on the exact messages their agents process in production. A detector that performs well on public corpora can still create an unworkable false-positive rate when pointed at real tool traffic or internal retrieval. The implication is that operating metrics must be built from local traces, not borrowed from benchmark culture.

Residual exposure defines the security outcome: no prompt injection detector removes every malicious instruction, so downstream controls inherit whatever slips through. That is especially important when the agent already has credentials or can take actions on behalf of the organisation. Practitioners should govern the leftover path as a live risk channel, not as an edge case.

OWASP prompt injection guidance remains the right conceptual anchor, but deployment metrics are the deciding factor: the category is stable, yet the controls are only meaningful if they are measured against the specific turn, threshold, and traffic mix they will protect. The implication is simple: use standards to frame the risk, then validate with your own production data.

What this signals

Turn-level measurement is the control gap: a prompt injection detector is only meaningful when its threshold is tested on the same rail that will see the payload in production. If the team measures the wrong turn, the control can appear effective while leaving the real attack path untouched.

Residual exposure has to be governed like a first-class risk: any detector leaves some malicious instructions in circulation, so the downstream programme must assume partial failure and limit what the agent can still reach. That is a workload-identity and agent-governance problem, not just a model-evaluation problem.

Prompt injection controls should be evaluated as operating points, not model personalities: the practical question is how many broken tasks, blocked workflows, or escaped instructions the business can absorb before the guardrail becomes net disruptive. That pushes teams toward local evaluation, rail-specific tuning, and explicit blast-radius containment.

For practitioners

• Measure recall at your real false-positive budget Score candidate detectors at the threshold your traffic can absorb, using a corpus built from your own user messages, retrieved chunks, and tool outputs.
• Separate evaluation by agent loop turn Test user input, retrieved context, tool responses, and tool-call arguments as distinct rails because each turn has different volume and different payload paths.
• Calibrate latency against the highest-frequency rail Use the smallest model that can meet your inline latency target on the busiest turn, then reserve a heavier judge for escalation only.
• Plan containment for residual injections Assume some malicious text will pass the detector and define what the agent can still reach if the guardrail misses or times out.

Key takeaways

• Prompt injection detection can look worse on a headline metric while becoming far more useful at the operating threshold that matters in production.
• The placement of the guardrail inside the agent loop changes volume, latency, and the type of payload it can catch.
• Teams need local traffic, rail-specific testing, and residual containment because no classifier eliminates prompt injection risk outright.

Key terms

• Prompt Injection Detection: A control that scores text for malicious instructions intended to override an agent’s expected behavior. In practice, it is a thresholded classifier or judge that tries to intercept bad prompts before they influence tool use or model reasoning.
• False-positive budget: False-positive budget is the amount of legitimate traffic a security control can wrongly block before it becomes operationally unacceptable. For agentic systems, the budget must be defined against real turn volume and workflow sensitivity, because even a small error rate can create large numbers of broken tasks.
• Residual exposure: The risk that data, permissions, or session state remain available after the intended work period is over. It is a common failure mode in shared, shift-based, or temporary access environments, where the control design protects the session but not the artefacts created during it.
• Agent loop turn: An agent loop turn is one discrete message or event that passes through an AI agent workflow, such as a user prompt, retrieved chunk, tool response, or tool call. Detection quality changes materially by turn type, because each one has different volume, context, and attack paths.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.