Prompt injection and indirect prompt injection expose AI trust gaps

At a glance

What this is: This article explains how direct and indirect prompt injection exploit LLMs by blurring the line between instruction and data, with indirect attacks posing the larger governance challenge.

Why it matters: It matters because AI teams, IAM leads, and security architects now have to govern model behaviour, data ingestion, and delegated tool use as part of a single control problem.

By the numbers:

• Only 44% of organisations have implemented any policies to manage their AI agents, despite 92% agreeing that governing AI agents is critical to enterprise security.
• Systems with least-privileged AI access had a 17% incident rate versus 76% for over-privileged systems, making proper scoping 4.5 times more protective.
• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes and as quickly as 9 minutes in some cases.

👉 Read AppSOC's analysis of prompt injection and indirect prompt injection

Context

Prompt injection is an AI security problem in which malicious text influences a model to ignore intended instructions, but indirect prompt injection is more difficult because the payload is hidden inside content the system already trusts. That trust boundary matters for prompt injection governance, because LLMs do not inherently know whether text is a command, data, or attacker-controlled input.

For practitioners, the issue extends beyond model safety into IAM and NHI control design. When retrieval pipelines, plugins, or agent workflows can trigger actions, the model becomes part of an access path that needs clear boundaries, monitored delegation, and tighter control of the identities and credentials behind the workflow.

Key questions

Q: What breaks when hidden instructions are allowed into AI retrieval pipelines?

A: Hidden instructions can turn trusted documents, emails, or pages into execution triggers inside an LLM or agent. The model may summarise, disclose, or act on attacker-controlled text as if it were legitimate context, which breaks separation between data and instruction. That can expose sensitive information, trigger unsafe tool use, and create audit gaps that traditional controls do not see.

Q: Why do AI agents make prompt injection more dangerous than chat-only tools?

A: AI agents are more dangerous because they can act, not just generate text. When a model can invoke tools, access records, or send messages, a hidden instruction can become a real enterprise action. The risk rises sharply if the agent inherits broad NHI permissions instead of narrowly scoped access.

Q: What do security teams get wrong about prompt injection defence?

A: They often assume better blocklists will solve the problem, but obfuscation simply changes the shape of the payload. Real defence requires examining meaning across the full interaction, including retrieved content and model responses. If the control cannot interpret intent, it will keep missing the attack class it is meant to stop.

Q: How should organisations govern AI-generated content before it is published?

A: Organisations should treat AI-generated content like any other controlled business output. Require human review for factual accuracy, legal exposure, privacy impact, and brand sensitivity before publication. The safest model is a documented approval workflow with named owners, logging, and the ability to retract or correct content quickly when errors are found.

Technical breakdown

Why direct prompt injection works against LLM instruction handling

Direct prompt injection works because LLMs process text statistically rather than semantically distinguishing trusted instructions from untrusted user content. If the application gives user input the same conversational weight as system instructions, a malicious prompt can override guardrails, extract hidden context, or trigger unsafe tool use. The weakness is not a traditional code exploit. It is a control failure in how the application separates authority levels inside the prompt stack and how it constrains model output and actions.

Practical implication: enforce explicit instruction hierarchy and runtime policy checks before the model can act on user-supplied text.

How indirect prompt injection abuses RAG and agent workflows

Indirect prompt injection hides malicious instructions in external content such as documents, web pages, emails, or knowledge bases. When a retrieval augmented generation pipeline or AI agent ingests that content, the model may treat attacker-authored text as legitimate context and execute it as if it were part of the task. The risk scales with automation because the injected content can persist, reappear across sessions, and influence downstream actions without an obvious visible prompt.

Practical implication: treat retrieved content as untrusted input and add filtering, provenance checks, and output constraints before any action executes.

Why conventional AppSec controls miss semantic attacks on AI

Firewalls, static input validation, and standard DLP tools are designed to inspect packets, fields, and content rules, not the meaning of natural language once it reaches a model. That gap matters because prompt injection attacks manipulate interpretation rather than code execution. Without runtime visibility into prompt composition, retrieved context, and model responses, defenders cannot easily tell when the model has crossed from analysis into unintended instruction-following.

Practical implication: add AI-specific monitoring and auditability that observes prompt, retrieval, and response paths together.

NHI Mgmt Group analysis

Indirect prompt injection is a governance problem, not just a model-safety problem. The attack succeeds because organisations often assume retrieved content is context, not command. Once an AI system can read documents, emails, or web pages and then take action, the trust model collapses unless the surrounding controls distinguish data from instruction. Practitioners should treat every retrieval path as an access boundary that needs explicit governance.

AI agents turn prompt injection into a delegated-authority risk. A model that can only answer questions creates one class of exposure, but an agent that can call tools, write tickets, or move data creates another. The relevant issue is not whether the model is autonomous in the abstract, but whether hidden instructions can influence actions that inherit human or service-account authority. That makes identity, approval, and runtime policy part of the same control plane.

Hidden prompt injection creates AI governance debt: the longer organisations rely on unverified content sources, the more embedded the risk becomes in workflows, audit trails, and downstream decisions. This is especially true in RAG architectures, where the source of a model’s answer is often operationally more important than the answer itself. Practitioners should assume that unmanaged content sources will eventually become attack surfaces.

Least privilege must extend to AI workflows, not just human users. If an agent can read broadly but act narrowly, the blast radius stays bounded; if it can read broadly and act broadly, hidden instructions can convert one poisoned document into enterprise-wide misuse. That means identity, access scope, and delegation boundaries have to be designed around the workflow, not the model alone.

What this signals

Hidden prompt injection creates a control gap that spans AI governance and identity governance. Once a model can consume untrusted content and act on it, the right question is no longer only whether the output is correct. It is whether the workflow can be tricked into using legitimate identities and delegated access against the organisation's own data and systems.

AI systems that can call tools need identity-scoped controls, not just content filters. The more a model behaves like an operational actor, the more organisations need to apply least privilege, approval boundaries, and provenance controls to the identities behind the workflow. That is where AI security starts to overlap directly with IAM and PAM practice.

The next governance step is to align RAG and agent controls with external guidance such as the OWASP Agentic AI Top 10 and NIST AI risk methods, then map those requirements into runtime policy and auditability.

For practitioners

• Define trust boundaries for retrieved content Classify every external and internal data source feeding an LLM or agent as untrusted until provenance, validation, and intended-use rules are explicit. Apply separate handling for documents, web pages, CRM records, and knowledge base entries so hidden instructions cannot move straight into model context.
• Constrain agent actions with policy and approval gates Limit what AI agents can do after interpretation by binding tool use, write actions, and data export to policy checks and human or programmatic approval where risk is high. Pair this with least-privilege credentials and short-lived access for the identities the agent uses.
• Log prompt, retrieval, and response paths together Capture the retrieved documents, the assembled prompt, the model response, and any downstream action in one audit trail so investigators can reconstruct whether hidden instructions influenced behaviour. This is essential for incident response and governance review.
• Segment AI workloads from sensitive identity workflows Keep AI systems that consume untrusted content away from privileged identity operations such as account changes, secrets retrieval, and access approvals. Where those workflows must connect, add step-up verification and separate service identities.

Key takeaways

• Prompt injection works because LLMs cannot reliably tell instruction from untrusted text, which makes prompt construction itself a security boundary.
• Indirect prompt injection is harder to detect because it hides inside the documents, web pages, and other sources that AI systems already trust.
• Effective defence requires runtime policy, provenance checks, and least privilege for the identities and tools behind AI workflows.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Indirect Prompt Injection: Indirect prompt injection is an attack where malicious instructions are hidden inside content that an AI system reads later. The model may treat that content as context rather than as hostile input, which can influence tool use, data access, or workflow actions if controls are weak.
• Retrieval-augmented Generation: Retrieval-augmented generation is a pattern where an AI model pulls external information before generating output. The security challenge is that access rules can weaken when data is chunked, embedded, cached, or reused, so source permissions may not automatically follow the content into the model's context.
• AI Workflow Delegation: AI workflow delegation is the practice of allowing a model or agent to carry out actions on behalf of a user or system. Once delegation includes tool use, write access, or data movement, hidden prompt attacks become governance issues because the model can exercise real authority.

What's in the full article

AppSOC's full research covers the operational detail this post intentionally leaves for the source:

• A side-by-side breakdown of direct and indirect prompt injection examples in real AI workflows
• Specific runtime defence patterns for RAG and agent-based architectures
• The vendor's control approach for monitoring AI interactions and auditing downstream actions
• Practical examples of how malicious instructions can be hidden inside ordinary business content

👉 AppSOC's full post covers the attack mechanics, defensive controls, and architecture implications in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, secrets management, and workload identity. It helps security practitioners connect identity controls to emerging AI workflows without overextending the role of the model.