What breaks in practice when untrusted text and trusted instructions are joined before the model sees them?

The boundary between data and instruction disappears at the token stream level. Once retrieved content, user input, and system instructions are concatenated, the model has no built-in way to know which spans are authoritative. That makes plain-language injections effective, especially when the application later treats model output as a tool call or command with real privileges.

Why This Matters for Security Teams

When untrusted text is concatenated with trusted instructions before inference, the application creates a single prompt stream and then asks the model to infer boundaries that no longer exist. That is why injection works: the model can be persuaded to reinterpret retrieved content, user input, or even policy text as higher-priority instruction. The practical risk is not just bad answers, but unsafe tool use, leaked secrets, and privilege-bearing actions triggered by malformed or adversarial content.

For teams governing non-human identities, the failure becomes more serious when the model’s output is wired into APIs, ticketing systems, or workflow automation. A prompt that looks harmless to a human can still alter the model’s intent at runtime, especially when the agent is allowed to plan, call tools, or chain steps. Current guidance from NIST SP 800-53 Rev 5 Security and Privacy Controls supports separation of duties and control enforcement, but it does not solve the token-stream collapse by itself. NHI Management Group has highlighted how often identity and secret hygiene are already weak, with the Ultimate Guide to NHIs noting that 79% of organisations have experienced secrets leaks and 97% of NHIs carry excessive privileges.

In practice, many security teams discover prompt injection only after a model has already exposed data or invoked a downstream action that should never have been reachable from plain text.

How It Works in Practice

The core problem is that most applications still build prompts by concatenating system instructions, retrieved documents, and user content into one sequence. Once that happens, the model no longer has a native mechanism to distinguish authoritative policy from hostile prose. A malicious snippet can therefore issue commands, override task framing, or redirect the model toward tool calls that were never intended by the application owner.

Safer designs treat the model as an untrusted reasoning component and move trust decisions outside the prompt. That usually means separating data from instructions, passing content through structured fields, and validating any action request before execution. For agentic workflows, the better pattern is runtime policy evaluation rather than a static “allowed tools” list. Intent-aware controls, short-lived credentials, and workload identity help ensure the agent can only act within the current task.

• Keep untrusted text in a data channel, not inside instruction text.
• Require the application to classify and gate tool calls after inference.
• Issue just-in-time credentials with tight scope and short time to live.
• Use workload identity and policy-as-code so access is checked at request time.
• Assume retrieved content can be adversarial unless it has been filtered and bounded.

This is why guidance around agentic systems increasingly emphasizes runtime authorization and cryptographic identity rather than trust in prompt phrasing alone, and it aligns with NIST SP 800-53 Rev 5 Security and Privacy Controls for enforcement and monitoring. The Ultimate Guide to NHIs is useful here because prompt injection often becomes an identity problem once the model is allowed to act as a service account or delegated operator.

These controls tend to break down when the application silently promotes model output into production actions without a separate approval, validation, or policy gate.

Common Variations and Edge Cases

Tighter prompt handling often increases engineering overhead, requiring organisations to balance usability against stronger separation and validation. That tradeoff matters because some teams want the model to summarize, retrieve, and act in one flow, but collapsing those steps is exactly what makes injection more damaging.

There is no universal standard for this yet, but current guidance suggests treating different content classes differently: system policy, retrieved evidence, and user-supplied text should not share the same authority level. In retrieval-augmented systems, a document may be legitimate but still unsafe to treat as instruction. In multi-agent pipelines, one compromised agent can poison the next agent’s context if outputs are passed forward without normalization.

Edge cases also appear when the model is asked to quote text, translate it, or transform it. Those tasks are often safe only if the application preserves provenance and blocks the transformed text from being reinterpreted as policy. The strongest practical pattern is to keep untrusted content inert, explicitly label it, and require downstream systems to make their own trust decisions rather than inheriting the model’s.

If a workflow cannot preserve that separation, then the risk is not just prompt injection but accidental command execution through a model-mediated interface.

Related resources from NHI Mgmt Group

• What breaks when a model can be persuaded to treat untrusted text as system-level instruction?
• What breaks when AI assistant skills can run code before the model sees the prompt?
• What breaks when an AI assistant accepts instructions before a human reviews them?
• What breaks when AI agent access to ServiceNow is not inspected before the model sees the response?