Prompt injection exposes LLM trust gaps in enterprise AI systems

At a glance

What this is: This is Fiddler’s deep dive on prompt injection in large language models, showing how crafted prompts can override model instructions and trigger harmful or unauthorized behaviour.

Why it matters: It matters because AI teams need to govern prompt handling, output trust, and monitoring as part of identity and access decisions around AI systems, agents, and the data they can reach.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.
• 96% of technology professionals identify AI agents as a growing security threat, and 66% believe this risk is immediate.

👉 Read Fiddler's deep dive on evaluating LLMs against prompt injection attacks

Context

Prompt injection is a control failure in which untrusted input is treated as if it were instruction. In practice, that means the model, orchestration layer, or downstream application can be induced to ignore the intended task boundary and produce outputs that were never authorised by the operator. For AI programmes, this is not just a model quality issue; it is a governance problem around trust, privilege, and data handling.

For identity and access teams, the intersection is real even when the model itself has no password. LLMs connected to tools, data sources, and internal workflows can act like non-human identities in the runtime stack, which means prompt abuse can become a path to data exposure, tool misuse, or delegated action outside policy. That makes this topic relevant to AI governance, NHI controls, and security monitoring together.

Key questions

Q: What breaks when prompt injection defenses are not in place for LLMs?

A: When prompt injection defenses are weak, attacker-controlled text can override the model's intended instructions, especially when the text arrives through retrieved content or external documents. The result is not always obvious compromise. It can be silent instruction hijacking, unsafe output, leaked context, or model behaviour that downstream systems trust too much.

Q: Why do LLMs make access and data governance harder for security teams?

A: LLMs complicate governance because they can combine user input, retrieved data, and tool output into one decision path. If those inputs are not separated by policy, the model may act on untrusted content as though it were authorised context. That creates a new control problem for data access, privilege, and auditability.

Q: How do teams know whether prompt injection controls are actually working?

A: Look for end-to-end visibility across prompts, retrieved content, memory, tool calls, and outputs, plus evidence that blocked actions stay blocked under realistic test cases. If the system can only be evaluated with static prompts, the controls are probably too narrow. Behaviour drift under multi-turn workflows is the signal to watch.

Q: Who is accountable when an AI system makes a harmful decision?

A: Accountability should follow the identity chain that authorized, configured, or triggered the action, including the human owner, the platform team, and any delegated agent or tool account. If the organisation cannot name that chain, the governance model is too weak for regulated AI use.

Technical breakdown

How prompt injection overrides model instruction hierarchy

Prompt injection works by placing adversarial text into a channel the system later treats as legitimate context. Because many LLM applications blend system instructions, user prompts, retrieved content, and tool output into one inference flow, the model can be tricked into giving higher weight to malicious instructions than the intended policy. The attack does not need model compromise; it exploits the application design around the model. This is why prompt boundaries, content separation, and instruction precedence matter as much as the model choice itself.

Practical implication: isolate trust zones in the prompt pipeline so untrusted text cannot override system-level policy.

Why input validation alone does not stop prompt injection

Input validation helps, but it is not a complete defence because prompt injection is often semantic rather than syntactic. Attackers may use paraphrase, encoding tricks, hidden instructions, or misleading context that still looks normal to a filter. The issue becomes sharper when retrieval-augmented generation pulls in external content, because the model may treat retrieved text as authoritative even when it is adversarial. Effective controls need layered filtering, context tagging, and output checks, not just a blocklist.

Practical implication: combine sanitisation with context-aware controls and post-generation validation.

How red-teaming exposes prompt robustness gaps before production

Red-teaming for prompt injection tests whether a model or AI application can resist adversarial instructions under realistic variation. Fiddler’s approach, using custom perturbations, reflects a broader engineering need: measure how often injected instructions succeed, where failures happen, and whether unsafe outputs propagate into downstream systems. This kind of testing is especially relevant when the model can trigger tools, retrieve records, or automate actions. In those cases, the impact is not just incorrect text, but unauthorized operations.

Practical implication: add adversarial prompt testing to pre-production AI assurance and release gates.

Threat narrative

Attacker objective: The attacker wants to hijack model behaviour so the AI system produces unsafe output, exposes data, or performs actions outside the operator’s intent.

1. Entry occurs when an attacker places crafted instructions into a user prompt, retrieved document, or other context that the application will pass into the LLM.
2. Escalation happens when the model accepts the injected instruction as higher priority than the original system intent and follows it instead of the approved task.
3. Impact follows when the LLM reveals sensitive information, produces harmful content, or triggers unintended actions through connected tools and workflows.

NHI Mgmt Group analysis

Prompt injection is an identity and authorisation problem disguised as an LLM problem. The attack succeeds when the application accepts untrusted text as executable instruction, which means the control failure sits in the trust boundary rather than the model weights. For NHI and agentic AI programmes, that boundary is where delegated actions, tool use, and data access must be governed. Practitioners should treat prompt channels as policy surfaces, not just user input fields.

AI systems need a named concept for this class of failure: instruction boundary collapse. That is what happens when system directives, user content, retrieved documents, and tool output are blended without clear precedence. Once boundary collapse occurs, the model can no longer reliably distinguish policy from payload. For practitioners, this means AI governance must define which instructions are authoritative before any deployment touches sensitive data or tools.

Prompt injection becomes materially more dangerous when the LLM is connected to real workflows. A standalone hallucination is a quality issue; a compromised agentic workflow is a security issue. If the model can query systems, retrieve records, or call functions, then prompt manipulation can become delegated misuse at machine speed. The governance takeaway is simple: the more authority an AI system has, the less tolerance there is for ambiguous instruction handling.

Pre-production testing needs to move from confidence scoring to adversarial resilience measurement. Fiddler’s perturbation approach is useful because it evaluates whether a prompt set can survive realistic manipulation, not just whether the model answers politely. That aligns with the broader direction of AI security assurance, where robustness, monitoring, and runtime containment matter together. Practitioners should require measurable resistance to injected instructions before granting broader access.

Prompt injection will keep resurfacing until organisations classify AI prompts as a control plane. The enterprise habit of treating prompts as disposable text misses their operational role in task execution, data retrieval, and downstream action. Once the prompt is a control plane, logging, review, privilege separation, and rollback become governance requirements. Teams should expect prompt security to sit beside IAM, NHI, and AI risk management rather than under model evaluation alone.

What this signals

Prompt injection is a reminder that AI governance cannot stop at model evaluation. Once an LLM is connected to data or tools, the prompt becomes part of the control surface, which means logging, separation of duties, and privilege containment need to be designed into the workflow from the start.

Instruction boundary collapse: when system prompts, retrieved content, and user text are not cleanly separated, the model’s behaviour becomes harder to predict and easier to manipulate. That is why AI security teams should align prompt controls with NIST AI 600-1 Generative AI Profile and adversarial testing practices.

For identity teams, the practical signal is that AI systems are increasingly behaving like governed non-human identities. If an assistant can read records, call functions, or trigger workflows, then the question is no longer whether the model is accurate, but whether its authority is bounded and auditable.

For practitioners

• Define trust boundaries for every prompt path Separate system instructions, user input, retrieved context, and tool output so untrusted text cannot override policy. Review where prompt concatenation is still used and replace it with structured messaging and explicit instruction precedence.
• Test AI workflows with adversarial perturbations Red-team prompts before production using paraphrase, encoding, and context poisoning scenarios that reflect real attack paths. Measure success rates, unsafe completions, and tool-triggering behaviour under each test condition.
• Limit tool authority for LLM-connected systems Grant the minimum function scope needed for the workflow and require human approval for high-impact actions such as record changes, credential retrieval, or external communication. Tool access should expire with the task and be logged per session.
• Add runtime monitoring for injected instruction patterns Log prompt chains, retrieved content, and model outputs so security teams can detect repeated override phrases, hidden directives, or abnormal action sequences. Monitoring should support incident triage and post-event audit.
• Gate deployment on prompt safety evidence Require release criteria that include adversarial robustness results, not only task accuracy. If injected instructions succeed in a meaningful share of tests, the application is not ready for sensitive workflows.

Key takeaways

• Prompt injection exploits the way AI applications trust context, which makes it a governance failure as much as a model vulnerability.
• The risk rises sharply when LLMs can reach data or tools, because manipulated instructions can become delegated action at machine speed.
• Security teams should test, log, and constrain prompt paths before production, treating AI prompts as a control plane rather than disposable text.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Instruction Boundary: The line between content the model should interpret and directives it should obey. In practice, this boundary is often blurred when workflows mix user data, embedded instructions, and tool permissions in one session. Strong governance depends on keeping that boundary explicit and enforceable.
• Adversarial Perturbation: An adversarial perturbation is a deliberate variation applied to input so a system can be tested for fragility under manipulation. In AI security, perturbations help teams measure whether a model remains stable when attackers paraphrase, encode, or reframe prompts.
• Tool-Connected LLM: An LLM that can do more than generate text because it is wired to external systems such as email, calendars, ticketing, or CRMs. The security risk changes materially once the model can trigger side effects, because prompt manipulation can become access abuse.

What's in the full article

Fiddler's full blog covers the operational detail this post intentionally leaves for the source:

• The exact custom transformation pattern used to generate adversarial prompt variants and test model robustness.
• The code structure for implementing the prompt injection transform in the Auditor framework.
• A concrete walkthrough of how the model responded across five injected prompt variations.
• The red-teaming workflow used to evaluate OpenAI text-davinci-003 against manipulated instructions.

👉 Fiddler's full post shows the custom perturbation approach, example code, and test outcomes in detail.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, secrets management, and agentic AI identity. It helps practitioners connect AI control gaps to the broader identity programme they already own.