Prompt injection exposes the control gap in enterprise AI assistants

At a glance

What this is: This is an analysis of prompt injection, where adversarial text causes AI assistants to ignore instructions, leak data, or misuse connected tools.

Why it matters: It matters because AI assistants increasingly operate across enterprise data and action surfaces, so identity, access, and control boundaries now have to hold inside the model workflow as well as around it.

By the numbers:

• A paper on Benchmarking Web Agent Security reports partial attack success up to 86% and end-to-end success up to 16% of the time in realistic browsing flows.
• The authors of Data Extraction Attacks in Retrieval-Augmented Generation via Backdoors show that with only 3% poisoned data, their backdoor-based method is highly effective.
• Assessing Prompt Injection Risks in 200+ Custom GPTs found a 97.2% success rate in extracting system prompts and 100% success in file leakage.
• An attack achieved nearly 80% end-to-end success against Mistral’s LeChat and ChatGLM in the Imprompter study.

👉 Read Knostic's analysis of prompt injection risks and enterprise defenses

Context

Prompt injection is an AI control problem, not just a prompt-quality problem. The attacker places instructions in text, files, web pages, or metadata that the model reads as if they were legitimate directions, then uses that confusion to drive disclosure or actions.

For identity and access programmes, the relevant question is who or what is allowed to influence an AI system’s output and tool use. That makes prompt injection a governance issue for NHI, agentic AI, and human approval workflows alike, especially where assistants can read internal content or call business systems.

Key questions

Q: What breaks when prompt injection is not contained at the retrieval layer?

A: When retrieval is not contained, attacker-controlled documents can become trusted context and override intended behaviour. The result is often silent because the assistant still appears to answer normally while following hidden instructions. Teams need provenance checks, source allow-lists, and content normalization before any chunk becomes eligible for generation.

Q: Why do AI assistants with tool access increase prompt injection risk?

A: Tool access converts prompt injection from a wording issue into an execution issue. A malicious instruction can trigger API calls, file writes, or messages with attacker-chosen parameters. The risk rises sharply when permissions are broad, egress is open, and the assistant can act without task-scoped approval.

Q: How do security teams know whether prompt injection controls are actually working?

A: They should measure whether injected content is blocked before retrieval, whether unsafe tool calls are denied, and whether alerts correlate prompts with retrieved sources and actions. Red-team tests, audit logs, and replayable traces show whether controls fail closed or merely detect after exposure.

Q: Who is accountable when an AI assistant leaks data or takes an unsafe action?

A: Accountability sits with the organisation that deployed the assistant, but ownership should be split across model operations, data governance, and access control. If a business process allows untrusted content to drive sensitive actions, the control failure is shared and must be documented in governance and incident review.

Technical breakdown

Direct override and hidden instruction injection

Direct prompt injection uses explicit commands such as ignore previous instructions, while indirect prompt injection hides the instruction inside content the assistant consumes from email, documents, pages, or tickets. The model does not distinguish between trusted system policy and attacker-provided text unless the application enforces that separation before the prompt reaches the model. In practice, this means the highest-risk surfaces are not only chat input fields but also any retrieval path that feeds untrusted content into the context window. Practical implication: harden every inbound content path before it reaches the model, not just the chat box.

Practical implication: harden every inbound content path before it reaches the model, not just the chat box.

Retrieval poisoning in RAG pipelines

Retrieval-augmented generation adds a second attack surface because the assistant can be steered by poisoned or cloaked source material. If the retriever surfaces attacker-controlled chunks, the model may treat them as authoritative and repeat injected instructions, expose secrets, or alter downstream reasoning. The core failure is provenance blindness, where the system cannot reliably tell which retrieved fragment influenced the answer. That is why chunking, source tagging, signed documents, and strict allow-lists matter as much as model-side guardrails. Practical implication: treat retrieval as a trust boundary and validate provenance before any chunk is eligible for generation.

Practical implication: treat retrieval as a trust boundary and validate provenance before any chunk is eligible for generation.

Tool and function abuse in agentic systems

When an assistant can call APIs, write files, or send messages, prompt injection becomes an action problem rather than a text problem. The attacker is no longer trying only to change the answer, but to make the model execute a legitimate tool with attacker-chosen parameters. That turns the assistant into a control plane for lateral misuse, data exfiltration, or unsafe operational actions. Least privilege, scoped tool permissions, explicit approval gates, and egress restriction are the controls that shrink that blast radius. Practical implication: constrain every tool call to the minimum verbs, destinations, and data scopes required for the task.

Practical implication: constrain every tool call to the minimum verbs, destinations, and data scopes required for the task.

Threat narrative

Attacker objective: The attacker wants the AI assistant to disclose data, bypass policy, or perform unintended actions while appearing to behave normally.

1. Entry occurs when malicious instructions are placed in chat, email, documents, webpages, or metadata that an AI assistant will read during normal operation.
2. Escalation occurs when the model treats attacker text as higher priority than system policy and then exposes data or invokes tools on the attacker’s behalf.
3. Impact occurs when the assistant leaks secrets, sends messages, writes files, or returns corrupted answers that the business treats as trustworthy output.

NHI Mgmt Group analysis

Prompt injection is a governance failure, not just a model failure. The attack works because enterprise systems often let untrusted content enter the same decision context as trusted policy. That collapses the boundary between data and instruction, which is exactly where conventional prompt reviews and static filters break down. Practitioners should treat the model context window as a policy enforcement surface, not a neutral text buffer.

Retrieval trust is now an identity problem for AI systems. Once assistants ingest documents, pages, and metadata, the question becomes which artifacts are authorised to shape machine behaviour. That makes provenance, signing, and retrieval allow-lists part of identity governance for AI, especially where agents act across shared knowledge systems. Teams that ignore source trust will keep discovering that access control at the repository layer is not enough.

Tool scope is the new privilege boundary for agentic AI. Prompt injection becomes materially more dangerous when the model can write, send, query, or trigger workflows. Least-privilege design must therefore extend to tools, destinations, and data classes, not just user roles or service accounts. The operational conclusion is simple: every allowed action must be bounded as tightly as a privileged API token.

Knowledge boundaries are becoming the practical analogue of zero standing privilege for assistants. An AI system should not retain broad, always-on exposure to sensitive context if the task only needs a narrow slice of information. This is where identity and NHI governance intersect most clearly, because the assistant and its tools behave like a non-human principal. Security teams should design for task-scoped access and deny by default rather than assume the model will self-limit.

Prompt injection exposes the limits of human-in-the-loop as a sole safeguard. Manual review helps, but it does not scale to every retrieval event or tool call in production workflows. The better model is layered control, with human approval reserved for high-impact actions and technical controls handling the routine path. Organisations that rely only on review will still miss the low-friction injections that succeed inside normal workflow volume.

What this signals

Prompt injection will increasingly be treated as an identity and access governance issue for AI systems. As assistants move from passive summarizers to action-capable agents, the operational question becomes whether a non-human principal is allowed to read, decide, and act across sensitive data without persistent overexposure. That is why practitioners should align retrieval trust, tool scope, and human approval with the same rigor used for privileged accounts and service identities.

Knowledge boundaries need to become enforceable runtime policy, not just content policy. The control pattern that will matter most is the ability to prove that an assistant only saw data it was authorised to use, with lineage back to the source artifact. For practitioners, that means combining provenance logging with access governance and investigating high-risk assistants as if they were privileged workloads.

The next phase of AI security will be measured less by whether attacks exist and more by how quickly organisations can prove they were contained. Teams that can replay prompts, retrieved content, and tool calls will have a far better chance of demonstrating control effectiveness, especially where MITRE ATT&CK Enterprise Matrix tactics map cleanly to credential abuse, lateral misuse, and impact.

For practitioners

• Harden inbound content before model ingestion Sanitize email, HTML, document, and metadata inputs before they enter retrieval or prompting paths. Strip hidden instructions, normalize markup, and reject untrusted MIME types where the assistant does not need them.
• Restrict tool permissions to task-scoped verbs Limit each assistant to the smallest possible set of APIs, destinations, and write actions. Separate read-only retrieval from action-capable workflows so a single injected prompt cannot cross from answer generation into execution.
• Enforce provenance and source lineage checks Tag every retrieved chunk with source, time, and trust status, then block generation when provenance is missing or ambiguous. This makes it easier to trace which artifact influenced a response during incident review.
• Log prompts, retrievals, and tool calls together Correlate the user request, expanded context, retrieved artifacts, guardrail decision, and every downstream tool invocation. That evidence trail is essential when investigating whether the assistant followed a legitimate request or an injected instruction.

Key takeaways

• Prompt injection succeeds because enterprise AI systems often fail to separate trusted instructions from untrusted content.
• The strongest evidence points to material risk across retrieval, browsing, and tool use, with some attack paths showing very high success rates.
• The practical answer is layered control: sanitize inputs, constrain retrieval, restrict tools, and log every decision path for auditability.

Key terms

• Prompt Injection: Prompt injection is adversarial text that changes how an AI assistant behaves by smuggling instructions into the model’s context. It can be direct, such as override commands in chat, or indirect, such as hidden instructions in documents, pages, or metadata that the model reads during retrieval.
• Retrieval-Augmented Generation: Retrieval-augmented generation is an architecture where a model fetches external content at inference time and uses that material to produce an answer. It improves relevance, but it also expands the trust boundary because any retrieved artifact can influence the model if provenance and filtering are weak.
• Tool Abuse: Tool abuse occurs when an AI assistant is manipulated into making actions through connected APIs, plugins, or workflows that the user did not intend. The risk is not only incorrect output, but execution of legitimate systems with attacker-chosen inputs or destinations.
• Provenance: Provenance is the record of where a piece of content came from, how it was transformed, and whether it can be trusted. In AI systems, provenance helps security teams identify which source, chunk, or artifact influenced a response and whether that input should have been allowed.

What's in the full article

Knostic's full analysis covers the operational detail this post intentionally leaves for the source:

• Benchmarks and attack-path breakdowns for direct override, indirect injection, retrieval poisoning, and tool abuse.
• Examples of how prompt injection appears in emails, SharePoint content, web metadata, plugins, and RAG pipelines.
• Control guidance on input hardening, retrieval hygiene, policy guardrails, and least-privilege actions in AI workflows.
• Auditability and lineage approaches that help investigators reconstruct which prompt or source caused the unsafe output.

👉 Knostic's full article covers attack types, success rates, and practical prevention controls in more depth.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, IAM, and secrets management. It helps practitioners translate identity controls into practical safeguards for AI-enabled workflows and privileged non-human access.