Prompt injection works by shifting the model’s frame, not breaking it

At a glance

What this is: This is an analysis of prompt injection and the frame problem in LLMs, showing that attackers can manipulate model behaviour by reshaping context rather than bypassing a traditional code boundary.

Why it matters: It matters because teams building GenAI features need to govern LLM behaviour, tool use, and output constraints as security controls, not assume prompt-layer safety filters will reliably separate benign from malicious intent.

By the numbers:

• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.

👉 Read Sprocket Security's analysis of how prompt injection shifts LLM behaviour

Context

Prompt injection is a governance problem as much as a technical one. In LLM systems, instructions, retrieved content, and user input all compete inside the same context window, so the model can be steered by framing rather than by a clean code and data boundary. For teams building AI features, the primary question is not whether a prompt looks malicious, but whether the system can prevent untrusted text from becoming authoritative.

This article matters to identity and access teams because the same trust assumptions that govern human requests also shape how LLMs interpret roles, authority, and delegation. When an application allows the model to act on behalf of a user, the security boundary is no longer just authentication. It also includes instruction provenance, tool scoping, and output enforcement, which makes AI governance intersect directly with IAM and NHI controls.

Sprocket Security’s examples are representative of a wider class of LLM weaknesses rather than an isolated edge case. The compounded framing described here is exactly why prompt filters alone rarely hold up under real attacker creativity.

Key questions

Q: How should security teams handle prompt injection in production LLM applications?

A: Security teams should treat prompt injection as a runtime control issue, not a content-moderation problem. The practical response is to inspect prompts, retrieved content, and tool outputs for hostile instructions, then block or downgrade unsafe sessions before the model can act on them. The model should never be the only enforcement layer.

Q: Why do LLMs treat malicious prompts as legitimate in some cases?

A: Because they resolve language through context, not through a clean code and data boundary. If an attacker creates a coherent role, genre, or workflow frame, the model may produce the response that best fits that frame rather than the one the defender intended. The risk is probabilistic, not absolute.

Q: What do security teams get wrong about prompt injection defence?

A: They often assume better blocklists will solve the problem, but obfuscation simply changes the shape of the payload. Real defence requires examining meaning across the full interaction, including retrieved content and model responses. If the control cannot interpret intent, it will keep missing the attack class it is meant to stop.

Q: Who is accountable when an LLM follows an attacker’s frame and causes harm?

A: Accountability sits with the organisation that deployed the system, because it chose the model, the permissions, the retrieval paths, and the control boundaries. Governance frameworks should assign ownership for prompt sources, tool scopes, output validation, and incident review before the system is put into production.

Technical breakdown

Why prompt injection works in a shared context window

Traditional injection attacks exploit a parser that can distinguish code from data. LLMs do not have that boundary in the same way. User instructions, system prompts, retrieved documents, and embedded text all live in the same context window, and the model resolves them through probabilistic pattern matching. That means a malicious prompt can compete with legitimate instructions by creating a stronger frame, not by breaking syntax. Once the model treats the attacker’s framing as coherent, refusal can become less likely than compliance.

Practical implication: move security controls outside the prompt layer and into architecture, scoping, and output constraints.

How authority transfer and genre shifting change model behaviour

Authority transfer works by persuading the model that it is speaking to a different principal, such as a researcher, auditor, or educator. Genre shifting then changes which language game is active, for example from assistant-user support to fiction, training, or tabletop analysis. These moves matter because the model has learned different continuations for each frame during training. It is not discovering truth about the user. It is selecting the most statistically coherent response for the newly implied role and setting.

Practical implication: test LLMs with compounded frames, not only direct harmful prompts, because the attacker often uses legitimacy as camouflage.

Why safety classifiers miss frame-shifted attacks

LLM safety has constitutive layers, such as training and RLHF, and repressive layers, such as classifiers and policy checks. Frame shifting attacks target the constitutive layer by making the harmful response look natural inside the new context. That is why a classifier trained on direct dangerous requests can miss the same harmful content when it appears inside a pedagogical, fictional, or role-based frame. The failure is not only detection latency. It is that the system is judging the wrong signal.

Practical implication: evaluate safety controls against indirect and role-based prompts, then enforce hard output rules for high-risk actions.

Threat narrative

Attacker objective: The objective is to make the model accept the attacker’s frame and emit unsafe or over-privileged output while appearing to function normally.

1. Entry begins when the attacker wraps unsafe instructions inside a legitimate-looking role, such as researcher, instructor, or support user, to create a coherent frame for the model.
2. Escalation occurs as the attacker layers context, genre, and authority cues until the model treats the malicious request as a normal continuation rather than a policy violation.
3. Impact follows when the model produces harmful or overbroad output, revealing sensitive content, enabling unsafe actions, or redirecting an AI agent into the attacker’s intended workflow.

NHI Mgmt Group analysis

Prompt injection is an instruction provenance failure, not a simple content-filter failure. The article shows that LLMs do not reliably distinguish trusted policy from attacker-supplied context once both occupy the same channel. That means the real governance gap is not just unsafe text, but the absence of provenance controls for instructions, retrieved material, and delegated actions. For practitioners, the lesson is to treat instruction trust as an access-control problem, not a moderation problem.

Frame shifting creates a named governance gap: context authority drift. Once a model accepts a different role or genre, the weight of system instructions can erode without any obvious exploit signal. That drift is especially relevant where AI systems can act like agents, because tool access and downstream actions may follow the shifted frame. For identity teams, this means LLMs need scoped authority and explicit action boundaries, much like privileged workflows do in PAM and NHI programmes.

Safety built only at the prompt layer is structurally fragile. The article correctly separates constitutive controls from repressive controls, and that distinction matters for governance. If the model is free to reinterpret context, then post-generation filtering and prompt nudges will always be after-the-fact controls. Practitioners should read this as a design warning: prompt engineering can reduce noise, but it cannot substitute for architectural enforcement. The field should move toward enforceable instruction hierarchies and output restrictions.

LLM risk increasingly overlaps with NHI and agentic AI governance. When a model can call tools, retrieve data, or act on behalf of a user, the model becomes a non-human actor that can inherit authority from context. That creates a policy problem similar to over-permissioned service accounts, except the trigger is language rather than credentials. The broader identity implication is that AI systems need lifecycle governance, scope review, and auditable delegation before they are trusted with operational actions.

The model can be redirected without being broken, which changes the defender’s mindset. This article’s most useful point is that many attacks look like normal system behaviour from inside the application. That shifts emphasis away from spotting obviously malicious prompts and toward constraining what the system can do even when it is confidently wrong. For security programmes, the citable conclusion is simple: trust should be bounded by design, not inferred from model tone.

What this signals

Prompt injection research should push teams toward stronger runtime boundaries, not better wording. The practical shift is to treat LLM applications as systems that need instruction provenance, scoped delegation, and constrained outputs before they can be trusted in business workflows.

Context authority drift: once a model accepts a different role or genre, its safety posture can change without any visible security event. That means defenders need to monitor for frame changes in multi-turn sessions and design controls that survive the model being redirected.

For identity programmes, the closest analogue is privileged access without lifecycle governance. If an AI system can call tools or disclose data on behalf of a user, its permissions should be as tightly reviewed as any other non-human identity, with clear ownership and revocation paths.

For practitioners

• Enforce instruction provenance controls Separate system instructions, retrieved content, and user input in application logic so untrusted text cannot silently compete with trusted policy. Treat every retrieved document as untrusted until it is explicitly classified and scoped for model use.
• Constrain tool access by role and task Limit what an LLM can call, read, or change based on the minimum task scope required. Use short-lived, auditable permissions for model-assisted workflows, especially where the application can trigger actions beyond text generation.
• Test against compounded prompt frames Red-team with authority transfer, genre shifts, and embedded context attacks, not only direct malicious prompts. Include scenarios where the model is asked to act as a professional peer, instructor, or analyst so you can see where safety degrades.
• Hard-code output boundaries for high-risk actions Use schema enforcement, allow-listed response types, and server-side validation for anything that could expose data or trigger operations. If the model can only return structured values, it has less room to turn a hostile frame into an operational action.

Key takeaways

• Prompt injection works because LLMs share instructions, user input, and context in one channel, which lets attackers redirect behaviour by changing the frame.
• The strongest governance gap is not just unsafe text detection, but the lack of provenance, scope, and output controls for model-driven actions.
• Security teams should test for compounded framing, constrain tool use, and enforce hard output boundaries before they trust LLMs with operational decisions.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Frame Shifting: The practice of changing the model’s perceived role, genre, or audience so a different response pattern becomes more likely. In security terms, it is a way to make unsafe output look coherent within a new context, which can reduce the effectiveness of naive refusal logic.
• Identity Provenance: Identity provenance is the record of how an agent was created, what authority it received, and what actions it performed over time. It turns agent activity into an auditable chain of trust that supports compliance, incident response, and post-event accountability.

What's in the full article

Sprocket Security's full analysis covers the operational detail this post intentionally leaves for the source:

• The article’s worked examples showing how compounded framing changes model responses in practice
• The breakdown of safety layers, including constitutive versus repressive controls, and where each fails
• The detailed discussion of red-team testing patterns such as authority transfer, genre shifting, and embedded context
• The later-stage defensive recommendations for output monitoring and architectural enforcement

👉 Sprocket Security's full article expands the examples, safety-layer analysis, and defence patterns in more detail.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management for practitioners building governed access models. It helps security teams connect identity controls to broader risk, even when AI systems and automation are part of the workflow.