Prompt injection exposes LLM trust gaps in enterprise AI systems

At a glance

What this is: This is an analysis of prompt injection in LLMs, showing that crafted prompts can override intended instructions and drive harmful outputs or actions.

Why it matters: It matters because IAM, data, and AI governance teams need to treat LLM prompts, outputs, and monitoring as part of the control plane, not as a harmless interface layer.

Context

Prompt injection is a control failure in which malicious or misleading input changes how a large language model behaves at runtime. The model is not being hacked in the traditional code-exploit sense, but its instruction hierarchy, context handling, and trust boundaries are being manipulated. In enterprise settings, that creates a direct security and governance problem wherever LLMs can read data, call tools, or influence decisions.

The identity angle is real when an LLM becomes an agentic control point that can access systems, disclose sensitive content, or perform actions on behalf of a user or workflow. In that setting, prompt handling is no longer just content moderation. It becomes part of authentication, authorization, auditability, and operational monitoring for AI-enabled systems.

Key questions

Q: How should security teams stop malicious instructions from entering LLM context?

A: Security teams should control the sources that can contribute to runtime context, validate the provenance of retrieved content, and strip instruction-bearing material before it reaches the model. The goal is to make context assembly observable and policy-driven, so untrusted content cannot silently become operational input.

Q: Why does prompt injection create risk in connected AI workflows?

A: Because connected workflows bring external data into the model’s decision surface. If retrieved documents, API responses, or knowledge base entries contain malicious instructions, the model may treat them as valid guidance. That makes the risk operational, not just conversational, especially when the system can access data or trigger actions.

Q: What are the signs that an LLM is vulnerable to prompt injection attacks?

A: Repeated instruction overrides, unexpected language shifts, disclosures from protected context, and actions that do not match the user’s intent are all warning signs. A model that behaves correctly only in clean test prompts but fails under paraphrase or hostile context also needs attention.

Q: When should organisations prioritise adversarial testing over more prompt tuning?

A: When the model will process untrusted content, retrieve internal data, or call external tools. Prompt tuning can improve behaviour, but adversarial testing tells you whether the control boundary holds when an attacker actively tries to break it.

Technical breakdown

How prompt injection overrides instruction hierarchy

LLMs do not separate trusted and untrusted text in the way a security control would. They process prompts as context, so an attacker can embed instructions inside content the model is asked to read and the model may treat those instructions as more important than the original task. That is why prompt injection can succeed even when the user prompt looks benign. The underlying issue is not only model robustness, but the absence of a reliable trust boundary between system instructions, user input, and retrieved content.

Practical implication: treat all model inputs as untrusted and segment system instructions from user-controlled context.

Why prompt injection creates output and action risk

Prompt injection matters because the harm is not limited to wrong answers. Once a model can be pushed off task, it can disclose sensitive information, generate misleading content, or trigger unintended actions in connected workflows. In an enterprise stack, that becomes a control problem across retrieval, tool use, and downstream automation. The model may appear compliant while actually following an attacker’s embedded instruction, which makes simple content review insufficient as a safeguard.

Practical implication: scope model permissions tightly and review any workflow where outputs can trigger external actions.

Why pre-production testing needs adversarial perturbations

Standard evaluation often misses prompt injection risk because normal test prompts do not stress instruction conflict, obfuscation, or malicious context. Adversarial perturbations and red-teaming expose how the model behaves when the prompt is paraphrased, appended to, or surrounded by hostile instructions. That matters for LLM governance because a model can look safe in a demo and still fail under adversarial prompting. The testing goal is not perfect immunity, which does not exist, but measurable visibility into failure patterns before deployment.

Practical implication: add adversarial prompt testing to model acceptance criteria before production rollout.

Threat narrative

Attacker objective: The attacker wants to steer the model away from its intended task so it reveals information, bypasses controls, or carries out unintended behaviour.

1. Entry occurs when an attacker supplies crafted text that is later consumed by the LLM as part of its working context.
2. Credential or instruction abuse follows when the model treats the injected content as a higher-priority command than the original request.
3. Impact occurs when the model emits harmful output, reveals sensitive information, or performs unintended actions in connected workflows.

Breaches seen in the wild

• EchoLeak (Microsoft 365 Copilot) 2025: A crafted email could make Microsoft 365 Copilot leak data from its context with no click, a zero-click prompt injection fixed as CVE-2025-32711.
• ForcedLeak (Salesforce Agentforce) 2025: A critical Agentforce flaw let attackers steer an AI agent through a Web-to-Lead form to leak CRM data via an expired allowlisted domain.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Prompt injection is a trust-boundary failure, not just a model-quality problem. The real risk is that enterprise systems often treat LLM input and output as if they were normal text fields, when they can actually reshape decisions and trigger actions. Once the model sits inside a workflow, the attack surface includes retrieval, orchestration, and downstream automation. Practitioners should govern prompt trust the same way they govern any other privileged input path.

Instruction hierarchy drift is the governance gap this article exposes. Enterprises often assume system prompts, policy text, and filtering layers will remain dominant, but injected content can still distort behaviour at runtime. That means the control failure is not simply “the model was fooled.” It is that the system lacks enforceable separation between trusted policy and untrusted context. Practitioners need to design for bounded failure, not assumed obedience.

LLM monitoring must shift from usage analytics to security telemetry. Logging prompts and outputs is useful only when it supports detection of suspicious patterns, repeated override attempts, and unsafe action chains. In AI governance terms, the model becomes part of the control environment, not just an application feature. Practitioners should require observability that can support incident triage, audit, and containment when prompt manipulation appears.

Adversarial evaluation belongs in the AI risk lifecycle before deployment. The article’s testing approach shows why ordinary validation is too shallow for hostile input conditions. This is especially important for AI systems that can access internal content or call tools, because the consequence of a successful injection is no longer a bad answer but a security event. Practitioners should treat prompt injection testing as a release gate, not an optional hardening exercise.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: Red Teaming AI Agents for Identity Abuse

What this signals

LLM programmes now need security controls that assume prompt trust is conditional, not fixed. Once models can read internal data or trigger actions, the question is no longer whether they answer well, but whether they can be manipulated into crossing a boundary the business intended to keep closed.

Instruction hierarchy drift: this is the practical pattern behind many prompt injection failures, where untrusted text competes with policy and wins often enough to matter. For practitioners, the response is to move control enforcement closer to issuance, retrieval, and action approval rather than relying on post-hoc review.

AI governance teams should expect prompt injection to show up first in systems that combine retrieval, browser use, or workflow automation. Those are the environments where a single successful override can become a security incident rather than a model-quality defect.

For practitioners

• Implement strict prompt boundary controls Separate system instructions, user input, and retrieved content so that untrusted text cannot silently override policy or task constraints.
• Add adversarial prompt testing to release gates Use paraphrases, injected directives, token tricks, and misleading context to measure how often the model ignores intended instructions before production.
• Limit tool and data access for LLM workflows Constrain each model to the minimum data sources and actions needed so a prompt injection cannot easily convert into disclosure or unsafe execution.
• Instrument logs for prompt abuse signals Track repeated override attempts, suspicious prompt patterns, and abnormal output behaviour so security teams can triage model abuse as an incident.

Key takeaways

• Prompt injection is a governance problem because untrusted text can alter model behaviour inside enterprise workflows, especially when the model can retrieve data or trigger actions.
• The article shows that adversarial prompting can bypass intended instructions and produce harmful outputs, which means clean demos are not enough to establish trust.
• Adversarial testing, strict input boundaries, and restricted tool access are the controls that most directly reduce the blast radius of this failure mode.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Instruction Hierarchy: Instruction hierarchy is the order of authority a model applies when interpreting context, system prompts, role messages, and user input. When attackers can influence that hierarchy through templates or wrappers, they can steer behaviour without needing to change the model itself.
• Adversarial Prompt: A prompt designed to alter model behaviour in a harmful or unintended way. In enterprise settings, it often hides the real request inside normal-looking text so filtering and review controls are less likely to catch it before the model acts.
• LLM Telemetry: LLM telemetry is the logging and monitoring data used to observe how a model is being used and how it responds. For security teams, it should support detection of suspicious prompts, repeated override attempts, and abnormal output or action patterns.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and related control patterns. It helps identity and security practitioners govern AI-enabled systems without assuming that model behaviour is inherently trustworthy.