They place controls on the wrong layer. Jailbreaks target model safety tuning, but prompt injection targets the application boundary where trusted instructions and untrusted content are mixed. If a team uses only jailbreak-style detection, indirect injection can slip through because the payload may look legitimate inside normal content.
Why the Distinction Matters at the Control Layer
Jailbreaks and prompt injection fail in different places, so the defence model has to fail in different places too. A jailbreak is an attempt to weaken model behaviour through the prompt or conversation, while prompt injection is an application-layer problem where untrusted content is allowed to compete with trusted instructions. If teams collapse them into one bucket, they often tune the model while leaving the instruction boundary exposed.
The practical error is treating the model as the only enforcement point. That works poorly when the risky content arrives through retrieval, documents, emails, web pages, tickets, chat messages, or tool outputs that the application later presents to the model. In those cases, the model may appear to be “following the user” when it is actually obeying injected instructions embedded in content that the system trusted too early.
That distinction also changes what “good” looks like. A jailbreak control is mostly about model response behaviour, refusals, and safety tuning. Prompt injection controls are about content handling, instruction hierarchy, trust boundaries, and whether the application separates data from directives before the model sees it.
Where Teams Usually Misdiagnose the Problem
Teams usually misdiagnose prompt injection when they look only for obviously malicious prompts. Indirect attacks are often disguised as normal business content, so the payload is not always the thing that looks suspicious. The injection can sit in a document, a webpage, a support ticket, a calendar item, or a poisoned retrieval result and still steer the model if the system hands that content to the model as if it were safe instruction material.
That misdiagnosis leads to brittle controls. If the only detection is jailbreak-style pattern matching, the application may miss attacks that never resemble an adversarial user prompt. If the only guardrail is model tuning, the system may still execute unsafe tool actions, reveal context, or follow hidden instructions that arrived through a trusted path.
The right mental model is to ask which layer is being abused. If the attacker is trying to make the model disobey policy, that is closer to jailbreak behaviour. If the attacker is trying to smuggle instructions through content, context, or retrieval so the application relays them to the model, that is prompt injection and the control problem is boundary management.
What the Boundary Problem Means for Defenders
Once the boundary is clear, the defensive priorities change. The application must decide what content is data, what content is instruction, and what content is allowed to influence downstream actions. That means sanitising or separating untrusted input, limiting which retrieved items are elevated into context, and making tool use contingent on explicit authorization rather than model confidence alone.
This is why layered testing matters. A team that only red-teams the model can miss indirect prompt injection paths that are only visible when the full workflow is exercised, including retrieval, rendering, memory, and tool calls. The application boundary is where the compromise becomes operational, so the test plan has to reach that boundary.
For practitioners, the simplest rule is that model safety tuning should not be treated as a substitute for application trust design. If an attacker can place hostile text inside content the system considers legitimate, the model can be “safe” and the product can still be unsafe.
Risk and Threat Considerations
Collapsing jailbreaks and prompt injection into one AI risk creates a false sense of coverage. The result is usually control drift, where teams spend effort hardening the model while leaving content ingestion, retrieval, and tool orchestration exposed to indirect instruction attacks.
Failure mechanism: Untrusted text is promoted into a trusted context, then interpreted as instruction material by the model or an attached tool workflow. Because the payload can be embedded inside ordinary-looking content, it may bypass jailbreak-focused filters and slip past reviewers who are only checking for obvious hostile prompts.
Impact: The system may leak context, trigger unsafe tool actions, or follow attacker-chosen instructions while appearing to behave normally. In agentic or retrieval-augmented systems, that can turn a content boundary failure into data exposure or unauthorized action.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 addresses the attack and risk surface, while NIST AI RMF and OWASP ASVS set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI02 — Tool Misuse | Prompt injection can steer tools through untrusted content. |
| ASI03 — Identity & Privilege Abuse | The wrong boundary lets attacker content drive privileged agent actions. | |
| ASI06 — Memory & Context Poisoning | Indirect injection poisons context that the model later trusts. | |
| Recommendation — Bind tool execution to explicit authorization and validate inputs before any tool call. Separate instruction sources from trusted identities and limit agent privileges to the minimum. Filter and isolate untrusted context before it enters memory or retrieval. | ||
| NIST AI RMF | Govern | AI risk governance must distinguish model safety from application-boundary abuse. |
| Recommendation — Define oversight for input trust boundaries, testing, and human escalation paths. | ||
| OWASP ASVS | V15 — Secure Coding and Architecture | Separating data from directives is an architecture issue. |
| V16 — Security Logging and Error Handling | Prompt injection needs observability when content causes unexpected behaviour. | |
| Recommendation — Design application flows so untrusted content cannot be treated as executable instruction. Log content provenance and model-triggered actions so abnormal instruction paths are reviewable. | ||
Practitioner Guidance
What to prioritise: Treat prompt injection as an application trust-boundary problem first and a model-behaviour problem second. The first question is not “did the model refuse?”, but “did the system ever allow untrusted content to compete with trusted instructions?”
What to verify: Test the full path from input to retrieval to tool use, because indirect injection usually succeeds where content is elevated without explicit trust separation. A useful control is one that still works when the payload is hidden inside apparently legitimate content.
Common mistake: Using jailbreak detection, refusal tuning, or prompt filters as the primary defence for systems that consume external content. That approach leaves the application boundary under-protected and misses the attack class most likely to survive normal-looking inputs.
Practitioner takeaway: If the control only protects the prompt, it does not protect the application. The durable fix is to manage which content can become instruction, which can trigger actions, and which must remain data-only.
Related resources from NHI Mgmt Group
- How should security teams reduce indirect prompt injection risk in AI systems?
- How should security teams reduce prompt injection risk in AI agents?
- How should security teams test AI-enabled mobile apps for prompt injection risk?
- How should security teams secure Microsoft 365 Copilot extensions and AI agents against prompt injection and remote execution risk?