Indirect prompt injection defenses fail when they inspect text upstream

At a glance

What this is: This analysis explains why indirect prompt injection defenses fail in production and shows that the deciding factor is where the control sits relative to the model’s action boundary.

Why it matters: It matters to IAM and PAM teams because agent governance depends on controlling capability, not just filtering text, especially when AI systems hold credentials or reach sensitive tools.

By the numbers:

• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes and as quickly as 9 minutes in some cases.

👉 Read ARMO's analysis of indirect prompt injection defenses and runtime control

Context

Indirect prompt injection is a control-placement problem, not just a prompt-hygiene problem. When the defense inspects text before the model acts, it is competing with an adaptive adversary on a surface that can be probed, varied, and retried until the seam is found. That makes the issue relevant to AI governance, identity governance, and any programme that lets an AI system act with credentials or tool access.

For IAM and NHI teams, the practical question is whether the agent is being governed as a text consumer or as a privileged runtime. Once an AI system can reach a ticketing system, code repository, cloud API, or customer data store, the control point shifts from classification to capability restriction and post-decision enforcement. That is why this topic sits at the intersection of agentic AI security and identity governance.

Key questions

Q: How should security teams govern AI agents that read untrusted text and can act on it?

A: Treat the agent as a privileged runtime with untrusted input, not as a chat interface. Limit tool access, shorten session scope, and remove at least one high-risk property such as external write access or access to sensitive systems. Then add monitoring or enforcement at the action boundary so the control evaluates what the agent actually does, not just what it reads.

Q: Why do prompt injection defenses fail when they only inspect the text?

A: Because they try to classify adversarial content before the model acts, which is a problem attackers can probe repeatedly. Different detectors can fail together for the same reason, even when they use different mechanisms. A text filter may reduce opportunistic attacks, but it does not provide a dependable governance boundary for an agent with real authority.

Q: What breaks when privileged AI agents can read untrusted input directly?

A: Prompt injection risk rises because the agent may treat user content as instruction instead of context. Once that happens, a malicious request can steer approval, reassignment, or data handling decisions without ever changing the agent's authorised role. The safe pattern is to sanitise untrusted input before any privileged step.

Q: Who is accountable when an AI agent acts outside its intended scope?

A: The organisation is accountable, but operational responsibility should sit with a named owner and a governance process that can explain the agent’s purpose, access, and recorded actions. Without that, autonomous behaviour becomes unassignable risk rather than managed automation.

Technical breakdown

Why upstream prompt injection defenses collapse against adaptive attackers

Upstream defenses classify text before the model acts. That includes detector models, perplexity filters, delimiters, paraphrasing, instructional prevention, and adversarial fine-tuning. All of them ask the same question: is this string data or instruction. Against a static benchmark, that can look effective. Against an adaptive attacker who can probe the system, the classifier becomes a target with a discoverable seam. The problem is structural, not implementation-specific, which is why very different mechanisms can fail together.

Practical implication: treat upstream filtering as volume reduction, not as a primary control for high-authority agents.

Capability controls and the Rule of Two

Capability controls do not try to understand the text. They limit what the agent is allowed to do, regardless of what the input says. The Rule of Two captures the design pressure well: within one session, an agent should not simultaneously process untrusted input, access sensitive systems, and change state or communicate externally. This is a Zero Trust style approach for AI execution, because the policy is enforced on permitted actions rather than inferred from content. It reduces blast radius even when the model is confused.

Practical implication: remove at least one high-risk property from any agent that reads untrusted content and can act externally.

Why action-boundary enforcement survives where text filters do not

Action-boundary controls observe execution rather than content. They compare the agent’s tool calls, API destinations, process activity, and system calls against a baseline for that specific workload. That makes them resistant to prompt variation because the attacker has to make the agent do something measurable, not just say something suspicious. The trade-off is that behavioral enforcement can produce false positives, so teams usually need an audit phase before enforcement. In identity terms, this is closer to runtime privilege governance than to content moderation.

Practical implication: build per-agent baselines from observed execution and enforce at the tool boundary, not only in the prompt layer.

Threat narrative

Attacker objective: The attacker wants a trusted AI agent to use its own legitimate authority to carry out actions on the attacker’s behalf.

1. Entry occurs when an attacker places indirect prompt injection content inside retrieved documents, tickets, web pages, or other untrusted text that an agent will process.
2. Credential and capability abuse follows when the model is induced to use its own authorized tools, credentials, or external communication paths in service of the attacker’s goal.
3. Impact occurs when the agent performs unintended actions inside approved flows, such as exfiltrating data, rewriting records, or taking administrative steps that look legitimate to upstream controls.

NHI Mgmt Group analysis

Upstream prompt filtering is a rate reducer, not a governance model. The published defenses in this article all sit before the model’s decision and therefore compete on text classification. That makes them useful for opportunistic noise but unreliable against a focused adversary who can adapt to the detector. For identity teams, the lesson is that classification does not equal authorization, and a trusted agent needs controls that survive beyond the prompt layer. The practitioner conclusion is to stop treating text inspection as the governance endpoint.

Agentic AI creates a new privilege problem, not just a new content problem. Once an AI system can call tools, touch data, or communicate externally, its risk resembles a privileged workload with untrusted inputs. That is where IAM, PAM, and NHI governance intersect: the question becomes what the agent can do, for how long, and under which session boundaries. The practitioner conclusion is to govern agent authority as a runtime identity issue, not as a chat safety issue.

Action-boundary enforcement is the most defensible control class for high-authority agents. Baselines derived from observed execution are harder to spoof than text classifiers because the attacker has to alter behavior, not just wording. This aligns with a broader Zero Trust approach in which the control point is the allowed action, not the interpreted intent. The practitioner conclusion is to use behavioral enforcement where the cost of a wrong action is higher than the cost of an occasional review.

Blast radius is the right unit of analysis for AI agent governance. Read-only agents, write-scoped agents, and infrastructure agents should not share the same control profile. An agent that can summarize documents is not equivalent to one that can modify tickets, push code, or reach cloud APIs with standing credentials. The practitioner conclusion is to size controls by the damage a single coerced session can cause, not by whether the model appears compliant in normal use.

What this signals

Action-boundary governance will become the deciding control plane for agentic systems. As more AI workloads receive tool access, teams will need to separate content safety from execution safety. The practical shift is toward per-agent policy, session-scoped authority, and behavioural monitoring that ties actions to a specific runtime identity rather than to the application container alone.

Indirect prompt injection is a preview of a broader delegated-authority problem. Any system that can read untrusted input and then act with authority inherits the same governance tension. That means identity teams should watch for agent sprawl, unclear ownership, and credential reuse across workflows, because those are the conditions that turn a prompt issue into a privilege issue.

The security programme implication is simple: controls that cannot distinguish between text and authority will be outpaced by agents that can. Teams should align AI governance with existing identity, PAM, and workload identity practices, then add runtime enforcement where the agent’s execution path intersects with sensitive systems.

For practitioners

• Segment agents by blast radius Classify each agent as read-only, write-scoped, or infrastructure-facing before assigning controls. Use stricter boundaries for any agent that reads untrusted input and can also change state or communicate externally.
• Remove at least one risky property Apply the Rule of Two to high-value agents so they do not simultaneously process untrusted text, access sensitive systems, and change state or communicate externally. Narrow tool access before expanding prompt defenses.
• Use behavioral baselines for enforcement Build per-agent baselines from observed tool use, API destinations, and process activity, then enforce at the action boundary. Audit first so you can measure deviations before you block them.
• Keep text filtering, but demote it Retain delimiters, detector models, and paraphrasing as a first-pass noise filter, but do not rely on them for agents with real authority. They reduce exposure, they do not guarantee safe action.

Key takeaways

• Indirect prompt injection breaks upstream defenses because text classification cannot reliably withstand an adaptive attacker.
• The real governance question is whether an AI agent can change state or reach sensitive systems with delegated authority.
• Per-agent behavioural enforcement and blast-radius control are the controls that change the risk equation for high-authority agents.

Key terms

• Indirect Prompt Injection: Indirect prompt injection is an attack where malicious instructions are hidden inside content that an AI system reads later. The model may treat that content as context rather than as hostile input, which can influence tool use, data access, or workflow actions if controls are weak.
• Action Boundary: The action boundary is the point where a user or system turns a request into a business-impacting decision, such as a payment approval or access grant. It is the most important place to add controls when attackers are using legitimate-looking messages to redirect trusted workflows.
• Behavior Baseline: A record of normal activity for a non-human identity, including typical consumers, resources, and actions over time. Baselines help security teams detect when an identity is being used in an unusual way and provide the context needed to enforce least privilege safely in dynamic environments.
• Confused Deputy: A confused deputy is a privileged system that is tricked into performing an action on behalf of an untrusted requester. In agentic AI, the agent may misread malicious input as legitimate intent and then use its own authority to act, which turns a logic problem into a security incident.

What's in the full article

ARMO's full blog covers the operational detail this post intentionally leaves for the source:

• Kernel-level telemetry examples showing how per-agent baselines are assembled from real execution
• Audit-to-enforce workflow detail for behavioural controls in AI workloads
• Practical distinctions between text filtering, capability restriction, and action-boundary enforcement
• Implementation notes for substituting agent credentials and constraining tool use

👉 The full ARMO post covers attack-path comparisons, control positioning, and enforcement detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security practitioners translate identity controls into runtime policy for AI-enabled workloads.