Security teams should treat the model as an untrusted interpreter of user input, not as a policy boundary. Reduce risk by isolating system instructions, constraining what the model can access, validating outputs before they reach downstream functions, and monitoring for attempts to override hidden prompts. The goal is to limit both information leakage and unsafe execution paths.
Why Prompt Injection Becomes a Real Problem in Chat-Driven LLM Apps
Prompt injection is not just a “bad prompt” issue. In a chat interface, the user message, retrieved context, tool outputs, and system instructions can all be presented to the model as text, so an attacker can try to persuade the model to ignore hidden instructions, reveal sensitive context, or take actions the application never intended.
The security failure usually appears when teams treat the model as if it can reliably separate policy from content on its own. That assumption breaks once the app lets model output influence downstream code, search, tickets, messages, or database actions. Good prevention starts with designing the application so the model never gets unconstrained authority over secrets, instructions, or execution paths.
One useful way to think about the problem is to separate prompt injection from general content moderation. The attacker is not merely trying to say something harmful, they are trying to steer the model into violating the app’s own trust boundaries.
Controls That Reduce Injection Impact Before the Model Can Cause Damage
The strongest controls are architectural. Keep hidden instructions out of the conversational surface where possible, minimise what the model can see, and isolate any material that should not be influenced by user text. If the assistant needs retrieved documents, treat those documents as untrusted input too, because prompt injection can arrive through web pages, emails, tickets, and other context sources.
Constrain the model’s tool use with explicit allowlists, narrow parameters, and strict output schemas. If the chat experience can trigger actions, make the model propose rather than perform when the action is sensitive, then require a separate policy or code layer to approve the step. That is especially important when the chat interface can touch production systems, customer data, or administrative workflows.
Output validation is just as important as input filtering. Validate structured outputs before they reach downstream functions, reject unexpected commands or fields, and assume the model may generate plausible but unsafe instructions. For application teams, the practical goal is to make the model useful without letting it become the decision point for authority or execution. Guidance from NIST AI Risk Management Framework aligns with this approach by emphasising governance, measurement, and controlled deployment of generative AI systems.
Risk and Threat Considerations
Prompt injection creates two distinct failure modes: information leakage and unsafe action. A successful attack can expose system prompts, hidden retrieval context, tokens, internal policy text, or other sensitive material, and it can also redirect the assistant into tool misuse, data modification, or unauthorised workflow steps.
Failure mechanism: The attacker supplies text that the model treats as higher-priority instruction, or as context that should be followed, and the application fails to separate untrusted user content from privileged instructions and actions.
Impact: The assistant may leak confidential context, bypass business rules, or issue requests that trigger harmful downstream operations, especially when the chat interface is connected to tools, plugins, or internal systems.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 address the attack and risk surface, while NIST AI RMF, NIST AI 600-1 and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | A1 — Prompt Injection | Directly addresses prompt injection against chat-based agentic applications. |
| Recommendation — Isolate instructions and constrain tool access to prevent user text from overriding policy. | ||
| NIST AI RMF | GV-1 — Govern | GenAI chat apps need governance over boundaries, approvals, and misuse tolerance. |
| Recommendation — Define approval boundaries and accountability for model-driven actions and disclosures. | ||
| NIST AI 600-1 | MP-2 — Pre-deployment Testing | Testing against adversarial prompts is essential before exposing chat interfaces. |
| Recommendation — Red-team prompt handling and validate controls before production release. | ||
| CIS Controls v8 | 6.3 — Data Recovery | This exact item is not the best fit for prompt injection; omitted from final output. |
| Recommendation — Omitted | ||
Practitioner Guidance
What to prioritise: Start with the boundaries that matter most, system prompts, retrieval sources, and tool permissions. If any of those are exposed to unrestricted user influence, treat the application as high risk even if the model itself appears well behaved in normal testing.
What to verify: Confirm that sensitive instructions are not echoed back, that retrieved content is sanitised or segmented before prompting, and that tool calls are constrained by code-level policy checks rather than model judgement alone. Test with adversarial prompts that try to override instructions, coerce disclosure, or force the model to call a tool out of sequence.
What good looks like: The model can summarise and assist, but it cannot independently unlock secrets, decide privilege, or bypass approval gates. If a prompt injection succeeds in changing the assistant’s tone but not its authority, the control design is working. If it can alter actions, the boundary is still too soft.
Practitioner takeaway: The right defence is not to make the model “smarter” about malicious prompts, but to make malicious prompts incapable of changing what the application is allowed to see, say, or do.
Related resources from NHI Mgmt Group
- How should security teams handle prompt injection in production LLM applications?
- How should security teams implement guardrails for LLM applications to reduce prompt injection and excessive agency risks?
- How should security teams defend AI applications against prompt injection in RAG and LLM workflows?
- How should security teams reduce the risk of prompt injection in LLM applications that call third-party libraries?
