The key controls are to separate trusted instructions from untrusted content, minimise what the model can access, and validate any tool use or output before execution. Organisations should apply input filtering, least-privilege connectors, human approval for sensitive actions, and logging for prompt and tool activity, because injected instructions can redirect the model or leak data.
How prompt injection defeats LLMs
Prompt injection works when untrusted text is treated as if it were instruction. In practice, the attack usually arrives through email, web pages, tickets, documents, chat history, or retrieved context, then tries to override the developer or system prompt, change the model’s task, or steer it toward disclosure and unsafe tool use. The core failure is a broken trust boundary between instructions and content.
The risk is highest when the model can act on behalf of a user or system. A harmless-looking instruction can become a command if the application lets the model blend retrieved data, user input, and operational tools without separation. That is why prompt injection is not only a content problem, but also an access-control and orchestration problem.
For a concrete example of zero-click leakage, see EchoLeak (Microsoft 365 Copilot) 2025, which shows how a crafted input can make an assistant expose context without user interaction. The broader attack surface is explained in Agentic AI Security Guide, which treats prompt handling, tools, and identity as part of one control plane.
Controls that matter before the model acts
Prevention starts with architectural separation. Trusted instructions, user prompts, retrieved documents, and tool outputs should be handled as different trust classes, not merged into one blob of text. The model should never be allowed to infer that a user-supplied sentence outranks policy, workflow rules, or system constraints.
Access scope matters just as much as prompt hygiene. If the model can read sensitive context, send messages, trigger workflows, or query systems, every one of those actions becomes an injection target. Minimise what the model can see, narrow what connectors can reach, and make tool permissions specific to the task rather than broad by default.
That is especially important for assistants that can browse, call APIs, or update records. The control objective is not just “better prompting”, but reducing the blast radius of any injected instruction. For practitioner patterns around agent capabilities, tool misuse, and privilege abuse, OWASP Agentic AI Top 10 is the best external baseline, and MITRE ATLAS adversarial AI threat matrix helps map prompt injection to adversary techniques.
Output validation is the final gate, not a substitute for safe prompting. If the model is proposing an action, producing a query, or generating content that could be executed, the application should validate structure, destination, and policy constraints before anything is carried out. Human approval is still warranted for sensitive actions such as payments, privilege changes, data export, or external posting.
Logging, testing, and recovery after an injection attempt
Logging is essential because prompt injection often succeeds by changing what the system decides to do, not by crashing it. Teams need records of prompt inputs, retrieved context, tool calls, refusals, and approval steps so they can reconstruct whether the model followed policy or an injected instruction. Without those records, you cannot distinguish benign model drift from abuse.
Testing should include malicious inputs that arrive through normal business channels, not just obvious jailbreak strings. Reviewers should probe indirect injection through documents, web pages, shared notes, ticket fields, and other content the model is likely to trust. The question is whether the system still behaves correctly when hostile text is embedded in a legitimate workflow.
The most useful external governance view is NIST AI Risk Management Framework, because it encourages structured risk treatment, measurement, and monitoring around AI behaviour. For agentic systems, CSA MAESTRO agentic AI threat modeling framework is useful for thinking about orchestration failure, autonomy, and tool-driven impact.
Risk and Threat Considerations
Prompt injection is dangerous because it can turn ordinary content channels into control channels. Once injected instructions influence retrieval, tool invocation, or data display, the result can be data exposure, unauthorized actions, or silent policy bypass that looks like normal model behaviour.
Failure mechanism: The application lets untrusted text compete with trusted instructions, then gives the model enough authority to act on the wrong instruction path, including reading sensitive data or calling tools.
Impact: Attackers can redirect the assistant, exfiltrate context, trigger unsafe actions, or abuse connected systems, often without obvious signs to the end user.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 addresses the attack and risk surface, while NIST AI RMF, NIST SP 800-53 Rev 5 and OWASP ASVS set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI02 — Tool Misuse | Prompt injection often drives unsafe tool invocation and action chaining. |
| ASI03 — Identity & Privilege Abuse | Injected prompts can redirect an agent into misusing delegated authority. | |
| ASI09 — Human-Agent Trust Exploitation | Prompt injection exploits misplaced trust in content presented to the agent. | |
| Recommendation — Constrain tool permissions and validate every model-initiated action before execution. Scope agent privileges narrowly and require step-up approval for sensitive actions. Separate untrusted content from trusted instructions and verify provenance before acting. | ||
| NIST AI RMF | GV — Govern, Map, Measure, and Manage | Prompt injection prevention needs governance, testing, monitoring, and risk treatment. |
| Recommendation — Define AI risk ownership, test hostile inputs, and monitor model actions continuously. | ||
| NIST SP 800-53 Rev 5 | AC-6 — Least Privilege | Minimising model and connector permissions directly limits injection blast radius. |
| AU-2 — Audit Events | Prompt and tool telemetry is needed to investigate injection attempts and model actions. | |
| SI-10 — Information Input Validation | Filtering and normalising untrusted text helps reduce malicious instruction content. | |
| Recommendation — Restrict model-connected accounts and tools to the minimum needed for each task. Log prompts, retrievals, tool calls, and approvals with enough detail for review. Validate and sanitise untrusted inputs before they can influence model behaviour. | ||
| OWASP ASVS | V4 — API and Web Service | Tool calls often behave like API actions and need explicit request and access control. |
| V8 — Authorization | Sensitive model actions must be permission-checked before execution. | |
| Recommendation — Authenticate and authorise model-driven API actions as if they were untrusted client requests. Enforce per-action authorization for any model output that can change data or state. | ||
Practitioner Guidance
What to prioritise: Put trust-boundary design ahead of prompt tuning. If the model can touch sensitive data or live systems, constrain those paths first, because prompt quality alone will not contain a compromised instruction stream.
What to verify: Confirm that tool calls require explicit policy checks and that high-impact actions cannot be executed directly from model output. If approval exists only in process but not in the code path, the control is not real.
Common mistake: Teams often harden the system prompt while leaving retrieval, connectors, and output execution too open. That leaves the highest-risk part unchanged: what the model can do after it has been influenced.
Practitioner takeaway: The safest LLMs are not the ones that “understand” prompt injection best, but the ones whose permissions, tools, and execution paths make successful injection much less useful to an attacker.