AI prompt analysis fails when attacks arrive through RAG

At a glance

What this is: This is an analysis of why prompt analysis misses indirect prompt injection when malicious instructions arrive through retrieved content, tool responses, or agent delegation.

Why it matters: It matters because AI and security teams need controls that inspect provenance, runtime behaviour, and scope enforcement, not just the text typed into a prompt.

Context

Prompt analysis is a request-plane control, which means it only evaluates content that arrives as a direct user prompt. That leaves a governance gap when AI systems consume retrieved documents, tool outputs, or delegated context that can carry instructions the request-plane never sees.

In AI agent and RAG environments, the real control problem is not only whether text looks suspicious, but whether the system can prove where it came from and whether it stays inside declared scope. Once retrieved content is treated as trusted context, prompt filtering alone cannot provide reliable prevention or detection.

Key questions

Q: What breaks when prompt analysis is used as the main defence against indirect injection?

A: Prompt analysis breaks because it inspects the request plane, while indirect injection arrives through retrieved content, tool output, or delegated context. The malicious instruction can look benign by the time it reaches the model, so detection depends on provenance, runtime telemetry, and scope enforcement rather than text filtering alone.

Q: Why do RAG and AI agents increase prompt injection risk?

A: They expand the number of places where untrusted instructions can enter the model’s working context. RAG pulls documents and records into the prompt, while agents convert model output into actions. That means a poisoned document or manipulated API response can shape both what the model says and what it does, which raises the impact from bad answers to unauthorized execution.

Q: How can teams tell whether prompt security is actually working?

A: Look for evidence that prompts are logged, constrained, and reviewed in the same way other security events are managed. If you cannot trace what was asked, what context was used, and what the model did, prompt security is not operationally effective.

Q: How can organisations tell whether an AI agent is acting outside its intended scope?

A: Organisations should look for behaviour that crosses expected tool boundaries, generates unusual credentials, or chains actions across systems that are not part of the original task. The signal is not simply high activity. It is a change in action pattern, delegation, or downstream access context.

Technical breakdown

Why lexical and classifier-based prompt filters miss indirect injection

Lexical analysis looks for known signatures such as regex patterns, encoded payloads, or unusual structure. ML classifiers score semantic intent and jailbreak likelihood, but they still inspect only the prompt content in front of them. Both mechanisms fail when the malicious instruction is written as ordinary prose, translated into another language, or hidden in retrieved context that appears benign by itself. In practice, they are useful for known-pattern triage, not as a complete control plane for AI attack detection.

Practical implication: Treat prompt filters as one telemetry source, not as a stand-alone prevention layer.

Why multi-turn drift is invisible to per-request controls

Multi-turn analysis tracks how a conversation changes across a session, including role-impersonation drift and instruction-precedence inversion. The key limitation is that perimeter tools usually see one request at a time, so they cannot compare the current turn with prior state unless that state is preserved inside the application layer. That is why an individual prompt can look harmless while the session as a whole becomes adversarial. Behavioural-textual analysis only works when the agent runtime keeps durable session context.

Practical implication: Instrument session state in the agent runtime if you want to detect conversation-level abuse.

How provenance signals expose indirect prompt injection

Provenance analysis traces where each token in the agent context came from, such as direct user input, RAG retrieval, tool output, or another agent. This is the only category that can surface indirect prompt injection because the attack does not begin in the request plane at all. The text may look legitimate once it reaches the context window, but source attribution and propagation chains can still reveal that it entered through a compromised data path. Without provenance metadata, the malicious instruction is indistinguishable from normal retrieved content.

Practical implication: Capture source attribution for retrieved and delegated content so provenance can be correlated with runtime behaviour.

Threat narrative

Attacker objective: The attacker aims to smuggle instructions into trusted retrieval or delegation paths so the agent executes actions or accesses data outside its intended scope.

1. Entry occurs when an attacker plants hidden instructions in a data source later consumed by the AI system, such as a wiki page retrieved through RAG or content returned through a tool call.
2. Credential or scope abuse follows when the agent accepts the retrieved text as trusted context and uses its existing permissions to invoke tools or query systems it would not normally touch.
3. Impact occurs when cross-surface correlation is missing and the agent reads or acts on data outside its declared scope, allowing silent data exposure or unauthorised workflow execution.

Breaches seen in the wild

• Twilio 0ktapus breach 2022: SMS phishing of employees exposed 209 Twilio customers and 1,900 Signal users, part of the 0ktapus campaign against 130+ firms.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Prompt analysis is only a surface control, not a detection strategy. Lexical filters and semantic classifiers can reduce noise, but they cannot govern instructions that arrive through RAG, tool output, or delegated agent context. That creates a false sense of coverage if teams treat request-plane inspection as the main control. Practitioners should treat prompt analysis as one signal source inside a broader AI runtime control model.

Indirect prompt injection is fundamentally a provenance problem. The decisive question is not whether text looks malicious, but whether the system can explain where that text came from and whether it should have been trusted. That is why provenance telemetry, runtime lineage, and source attribution matter more than ever in AI governance. Teams that cannot trace context origin cannot reliably separate legitimate retrieved data from malicious instructions.

AI governance now has a correlation requirement. A single guardrail category rarely produces a reliable finding on its own when the attack arrives through RAG or delegation. The control gap is not the absence of a better prompt filter, but the absence of a policy layer that correlates provenance, tool use, and scope. The practical conclusion is clear: detection must move from content screening to runtime correlation.

Runtime-derived context becomes an attack surface the moment it is trusted by default. Once an AI system can consume external content and act on it, the boundary between data and instruction collapses unless governance is explicit. That applies equally to agentic AI and to any workflow that mixes retrieval with action. Security teams should govern retrieved context as a first-class input class, not as passive background text.

From our research library:

• 96% of security operations teams report critical blind spots, most commonly in cloud infrastructure (74%) and identity and access behaviour (67%).
• Read next: Agentic AI Security Guide

What this signals

Correlation is the control boundary, not prompt hygiene alone. Teams should expect direct prompt filtering to catch only a narrow slice of AI abuse, while the materially important cases emerge when provenance, tool use, and scope are evaluated together. The programme implication is that AI security has to be instrumented inside the runtime, not only at the perimeter.

Indirect prompt injection turns trusted context into an execution path. Once retrieved documents or delegated content can influence actions, the governance question becomes whether that context was sourced, tagged, and authorised correctly before it reached the model. That creates a direct overlap with agentic AI identity and runtime authorisation, especially where AI systems can invoke tools on behalf of users.

For practitioners

• Instrument provenance for every retrieved token Track whether context came from direct user input, RAG, tool output, or delegated agent state so suspicious lineage can be correlated later.
• Correlate prompt telemetry with runtime behaviour Join request-plane signals with tool-call history and scope-deviation events, because indirect injection often becomes visible only after the agent starts acting.
• Enforce declared scope at execution time Block or challenge tool calls that fall outside the agent’s approved access pattern, even when the triggering prompt appears benign.
• Preserve session state for behavioural analysis Keep durable conversation history and agent runtime state so multi-turn drift, role impersonation, and instruction precedence changes can be detected.

Key takeaways

• Prompt analysis alone cannot reliably detect indirect injection because the attack often enters through retrieved content or tool output rather than the user request.
• The important failure mode is not a weak prompt filter but a missing correlation layer across provenance, behaviour, and scope.
• AI teams need runtime instrumentation that treats retrieved context as governed input and verifies what the agent is allowed to do with it.

Key terms

• Indirect Prompt Injection: Indirect prompt injection is an attack where malicious instructions are hidden inside content that an AI system reads later. The model may treat that content as context rather than as hostile input, which can influence tool use, data access, or workflow actions if controls are weak.
• Provenance Signal: A control signal that records where a piece of context came from and how it entered the model’s working memory. In AI governance, provenance is what lets teams distinguish trusted retrieval, delegated content, and direct user input from poisoned or untrusted sources.
• Runtime correlation: Runtime correlation is the practice of joining identity state changes with security activity while an investigation is still active. It lets teams evaluate whether access use matches expected behaviour, which is more useful than reviewing entitlement records after the fact.
• Package Scope: Package scope is the GitHub permission boundary that determines whether a token can read or publish packages such as container images. In GHCR, read:packages allows pulling private or public packages, while write:packages allows pushing. Correct scope selection is essential because authentication can succeed even when authorisation later fails.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and secrets management. It helps security and IAM practitioners translate identity controls into runtime governance for modern AI systems.