Prompt injection detection still fails at the wrong metric

At a glance

What this is: This is an analysis of why prompt injection detection models must be measured by deployable thresholds, not headline benchmark scores.

Why it matters: IAM and security teams building agent controls need metrics that map to production impact, because false positives, turn volume, and residual risk determine whether a detector can sit in front of real workflows.

By the numbers:

• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes, and as quickly as 9 minutes in some cases.

👉 Read ARMO's analysis of prompt injection detection metrics and deployment trade-offs

Context

Prompt injection detection is only useful if it can be measured at the same point where it will actually be enforced. In agentic systems, that means judging a control by recall at an acceptable false positive rate, not by a model-card score that ignores placement, traffic volume, and the residual instructions that still reach the agent loop.

For AI agent governance, this is an identity problem as much as a content problem. The control sits in front of non-human identities that already hold tool access, so the real question is whether a detector can reduce risk without breaking production workflows that depend on delegated execution.

ARMO’s article frames the issue through production deployment rather than research optics. That starting point is typical for teams trying to operationalise agent safety in environments where access, latency, and workflow continuity all matter at once.

Key questions

Q: How should teams evaluate prompt injection detectors before deployment?

A: Evaluate them at the false positive rate your workflow can absorb, using examples from your own traffic rather than a public benchmark. The key metric is recall at the chosen threshold, not headline AUC. If the detector looks good in theory but creates too many broken tasks in production, it is the wrong control for your environment.

Q: Why do prompt injection controls fail in agentic environments?

A: They often fail because the control is measured or placed against the wrong turn. Attacks can arrive through retrieved content, tool output, or another agent’s message, while a detector on the input rail sees only one part of the workflow. The result is a gap between where risk enters and where the control operates.

Q: How do security teams decide where to place prompt injection detection?

A: Place the lightest viable control on the highest-frequency turn and reserve heavier reasoning models for escalation. User input, retrieval chunks, and tool output each create different cost and risk profiles, so placement should follow the agent loop rather than a generic guardrail design.

Q: What should organisations do with the residual risk after prompt filtering?

A: Assume some malicious instructions will still get through and design downstream containment accordingly. That means scoping tools, limiting delegated access, and monitoring execution so a successful prompt injection does not become unrestricted agent action. The detector reduces arrival rate, but it does not eliminate the need for runtime controls.

Technical breakdown

Why AUC is the wrong deployment metric for prompt injection detection

AUC measures ranking quality across all possible thresholds, but deployment requires a single operating point. A prompt injection detector can rank malicious and benign text correctly and still fail at the exact threshold your false positive budget allows. That is why the same model can look marginally worse on AUC while performing far better in production. In agentic systems, the threshold is the control. If the metric does not reflect the thresholded decision, it is describing model behavior, not security behavior.

Practical implication: evaluate detectors at the threshold and traffic conditions where they will actually run.

Why turn placement changes detector performance and cost

Agent loops create multiple scoring points: user input, retrieval chunks, tool responses, and tool calls. Each turn has a different volume profile, latency cost, and exposure pattern. A detector on the input rail sees one message per request, while a detector on retrieval or tool output sees far more traffic and catches the payload where it often actually arrives. That means architecture choice is inseparable from identity flow. The same model can be affordable on one turn and operationally impossible on another.

Practical implication: place the lightest viable control on the highest-frequency turn and reserve heavier checks for escalation.

Why residual risk defines the control stack

A prompt injection detector never eliminates risk because some payloads will still pass through, especially when the instruction is embedded in retrieved content, tool output, or long context windows. The residual is the part every downstream safeguard must absorb. That makes prompt detection a containment layer, not a complete policy. In practice, the point is not to achieve perfect blocking. It is to reduce attack arrival rate enough that downstream runtime controls can still operate on the remaining cases.

Practical implication: treat detection as one layer in a broader agent control stack, not as the final boundary.

Threat narrative

Attacker objective: The attacker wants the agent to accept malicious instructions as legitimate context and then carry them out through its own delegated access.

1. Entry occurs when malicious instructions are delivered through prompts, retrieved content, or tool outputs that the agent is likely to process as trusted context.
2. Escalation occurs when the detector misses the payload or fires on the wrong turn, allowing the agent to continue with delegated tools and credentials.
3. Impact occurs when the agent executes unintended actions, exposing data, misusing tools, or amplifying the original instruction across downstream systems.

Breaches seen in the wild

• Gemini AI Breach — Google Calendar Prompt Injection — Gemini AI assistant prompt injection attack leaks sensitive Google Calendar data.
• Amazon Q AI Coding Agent Compromised — Amazon Q AI coding agent compromised via prompt injection enabling hackers to inject data-wiping commands.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Prompt injection detection is a thresholding problem, not a model-card problem. Published scores are useful only when they map to the false positive budget a production team can absorb. A detector that ranks well but fails at the deployed threshold is not a governable control. For practitioners, the operational question is whether the metric reflects the actual decision point inside the agent loop.

Turn placement is the hidden identity-control decision in agent security. The same detector behaves differently on user input, retrieval chunks, tool output, and tool calls because those turns carry different volumes and different trust assumptions. This is where NHI governance meets runtime AI control: the actor already has delegated access, so the control must intercept the right message at the right point. Practitioners need to evaluate where the control sits before they evaluate what model sits behind it.

Residual prompt risk creates a containment requirement across the rest of the stack. No text classifier removes the need for downstream runtime controls because some instructions will always pass through, especially in long-context and tool-mediated workflows. That means the detector is only one part of the identity blast radius calculation, not the end of it. Security teams should size the rest of the control stack for what the classifier leaves behind.

Prompt injection governance is converging with non-human identity governance. The real issue is not just whether text is malicious, but whether an agent with valid access can be persuaded to misuse that access. That makes prompt filtering part of a broader NHI control plane that must account for tool delegation, execution scope, and post-decision containment. Teams that treat the detector as a standalone content filter will under-govern the identity risk.

Prompt injection detection models need environment-specific measurement to be trustworthy. The article’s central lesson is that published corpora and public benchmarks do not define your traffic distribution. A named concept emerges here: prompt detection residual, the portion of malicious instruction that survives thresholding and becomes downstream risk. Practitioners should measure that residual on their own traces before deciding where the control belongs.

From our research:

• The average estimated time to remediate a leaked secret is 27 days, despite 75% of organisations expressing strong confidence in their secrets management capabilities, according to The State of Secrets in AppSec.
• Companies are dedicating an average of 32.4% of their security budgets to secrets management and code security, with US organisations leading at 40.8%.
• Prompt defenders should read the Ultimate Guide to NHIs , Key Challenges and Risks for the broader access-sprawl context behind agent control failures.

What this signals

Prompt detection residual: the surviving fraction of malicious instruction is the metric that should shape runtime containment, because agents already hold delegated access and a missed prompt can become an executed action. The more workflows depend on retrieval and tool output, the more the control must be designed around residual risk rather than perfect blocking.

The category is maturing toward environment-specific measurement, which is the right direction for NHI governance. Public benchmarks will still matter, but practitioners should expect the decisive metric to be how much bad traffic survives inside their own agent loop, not how well a model generalises on someone else’s corpus. That is why control placement and instrumentation matter as much as model choice.

With 6 distinct secrets manager instances on average, per The State of Secrets in AppSec, fragmentation already weakens centralised control in many environments. Agent security inherits that same problem: if the identity and execution surface is fragmented, the detector can only be one layer in a broader governance model.

For practitioners

• Measure recall at your real false positive rate Build an evaluation set from your own user messages, retrieved chunks, and tool outputs, then score candidate detectors at the threshold your workflow can tolerate. Use the result as the only deployment-grade comparison.
• Map detector placement to agent turns Decide whether the model will score user input, retrieval, tool output, or tool calls, and model the latency and volume impact for each rail before choosing a control path.
• Quantify broken tasks per day Multiply measured false positive rate by the number of messages you plan to score each day, then treat the result as your expected production failure rate.
• Use a two-tier control shape Put a small, low-latency detector on the hot path and reserve a larger judge model for ambiguous cases that need deeper reasoning or escalation.
• Carry the residual into downstream controls Assume some malicious instructions will still reach the agent, so pair prompt filtering with runtime containment, tool scoping, and execution monitoring.

Key takeaways

• Prompt injection detection must be judged at the production threshold, not by benchmark optics.
• The control’s position in the agent loop changes both its accuracy and its operational cost.
• Residual risk remains after filtering, so downstream runtime containment is still required.

Key terms

• Prompt Injection Detection: A control that scores text for malicious instructions intended to override an agent’s expected behavior. In practice, it is a thresholded classifier or judge that tries to intercept bad prompts before they influence tool use or model reasoning.
• False positive closure rate: The share of alerts that are automatically identified as benign and closed with supporting evidence before reaching analyst queues. It is a useful SOC metric because it shows whether automation is reducing noise without hiding real threats.
• Residual Risk: Residual risk is the risk that remains after controls are applied. In identity-heavy environments, it often reflects over-permissioning, stale accounts, and exceptions that were accepted but never truly removed, which means the real exposure can be higher than the documented policy baseline.
• Turn Placement: The point in an agent loop where a control inspects data, such as user input, retrieved context, or tool output. Placement matters because different turns carry different volumes, trust assumptions, and attack paths, and those differences change both cost and efficacy.

What's in the full article

ARMO's full blog covers the operational detail this post intentionally leaves for the source:

• The model-by-model comparison table with latency and recall data across the detector generations.
• The turn-by-turn placement analysis for user input, retrieval, tool output, and tool-call scoring.
• The corpus construction approach for measuring prompt injection on your own traffic.
• The deployment trade-offs between small encoders and larger judge models.

👉 ARMO's full post covers the threshold logic, turn placement, and residual risk in more implementation detail.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an identity security programme, it is worth exploring.