A jailbreak no longer ends with a chatbot saying something it should not: in agentic systems it is the step that turns a model's refusal into a tool call, a leaked secret or an attack run on the attacker's behalf. MITRE ATLAS tracks this as AML.T0054 LLM Jailbreak and places it in two tactics, Defense Evasion (AML.TA0007) and Privilege Escalation (AML.TA0012). Jailbreak technique families, content screening layers and the phased defence programme are covered in the LLM Jailbreaks Guide; this page uses the ATLAS view to answer two narrower questions: how to detect jailbreak attempts in production telemetry, and how to design agents so that a successful jailbreak cannot do much. It is for AI platform owners, agent developers and SOC analysts.
Source note: technique, case study and mitigation details follow the MITRE ATLAS data release 2026.10 (modified 28 October 2026). Confirm against the official text before relying on it.
Key takeaways
- Assume some jailbreaks succeed. ATLAS notes that providers keep updating guardrails as new prompts appear and that jailbreaks can be generated automatically at scale; plan for bypass, not perfection.
- Privilege escalation is the agent risk. ATLAS lists jailbreaking an agent to invoke tools for the attacker's purposes; the blast radius is set by the agent's permissions, not by the prompt.
- Jailbreaks ride inside injections. Several ATLAS case studies pair indirect prompt injection with jailbreak framing so that planted instructions are obeyed.
- Detection signals already exist. Guardrail services and audit logs can flag suspected jailbreaks per message; route them to the SOC with the identity and session attached.
- Some jailbreaks need no prompt at all. Modified or "uncensored" model weights remove safety behaviour entirely, which makes model provenance part of this technique.
What the technique is, in plain terms
ATLAS defines AML.T0054 as inducing an LLM to ignore, circumvent or override its safety and alignment behaviour or guardrails to obtain outputs the model is meant to withhold. It notes two routes: adversarial prompting, and modifying model weights or safety mechanisms. The technique was created on 25 October 2023, last modified on 15 September 2026, is rated "Realized" in ATLAS 2026.10, and applies to the Generative AI and Agentic AI platforms.
The two tactics explain why it matters to security teams rather than only to trust and safety teams:
| Tactic | What the jailbreak is for | Follow-on techniques ATLAS names |
|---|---|---|
| AML.TA0007 Defense Evasion | Get past the model's own guidelines and guardrails to reveal information or produce harmful content | AML.T0085 Data from AI Services; AML.T0069 Discover LLM System Information; AML.T0102 Generate Malicious Commands; AML.T0052.000 Spearphishing via Social Engineering LLM |
| AML.TA0012 Privilege Escalation | Make an agent use its tools and permissions for the attacker, or use the agent as a command and control channel | AML.T0053 AI Agent Tool Invocation; AML.T0108 AI Agent (command and control) |
ATLAS describes prompt strategies at a high level, including instruction override, role-play, fictional framing, multi-turn escalation, constrained output formats, structured data injection and obfuscation, as well as algorithmic generation of jailbreak prompts and weight modification. Obfuscation has its own technique and guide: see MITRE ATLAS AML.T0068 LLM Prompt Obfuscation. Weight modification is covered in Uncensored and Abliterated AI Models.
How it shows up in real cases
The ATLAS case studies linked to AML.T0054 show the shift from content harm to agent misuse.
| Case | How the jailbreak was used | What limited or would have limited the damage |
|---|---|---|
| AML.CS0069 GTG-1002 Claude Code Espionage Campaign (2025) | Operators told the model it worked for a legitimate security firm doing defensive testing and split the operation into small, innocent-looking tasks, so the agent never saw the full intent | Provider-side detection of the activity pattern, account bans and classifier updates. See Anthropic GTG-1002 Campaign 2025. |
| AML.CS0057 Storm-2139 Azure OpenAI Guardrail Bypass | A criminal network used credentials scraped from public sources to reach customers' generative AI accounts, then used custom tooling to defeat safety controls and sold access onwards | The entry point was leaked customer credentials; key hygiene and abuse monitoring on the account side |
| AML.CS0067 Claude Code GitHub Action Secret Exposure | Compliance-review framing and an instruction to strip a credential prefix got around the model's refusal to output a recognisable API key | Keeping CI secrets out of the agent's reach on untrusted content; output scanning for secret formats is weakened by such transformations |
| AML.CS0046 Data Destruction via Indirect Prompt Injection Targeting Claude Computer-Use | Instructions in a PDF claimed the session was a security test environment where dangerous commands were acceptable | Sandboxed execution and approval for destructive commands |
| AML.CS0052 LLMSmith | Lightweight jailbreaks used where the target application's agent refused, as a step towards remote code execution | Sandboxing code interpreters in LLM-integrated applications |
The common thread: in each agent case the jailbreak only mattered because the model could reach a tool, a secret or a shell. Where it could not, the jailbreak produced words, not actions.
Limiting the blast radius
The ATLAS mitigations for AML.T0054 work on the model: AML.M0020 Generative AI Guardrails, AML.M0021 Generative AI Guidelines, AML.M0022 Generative AI Model Alignment and AML.M0035 AI Red Team. They lower the success rate. The controls below, NHI Mgmt Group's mapping from ATLAS agent mitigations, limit what a successful jailbreak can achieve.
| Control | ATLAS reference | Effect on a jailbroken agent |
|---|---|---|
| Authority in code, not in the prompt | AML.M0028 Enforce Access Controls for AI Agent Tools | The tool refuses actions outside policy whatever the model asks |
| Delegated, least-privilege identity per agent | AML.M0026, AML.M0027 | Tool calls carry only the user's or the agent's narrow rights |
| Approval for consequential actions | AML.M0029 Human In-the-Loop for AI Agent Actions | Sending, deleting, paying and executing need a human decision |
| No authority expansion at run time | AML.M0037 AI Agent Authority Expansion Controls | A jailbroken agent cannot request new scopes or credentials for itself |
| Scope drift detection | AML.M0038 AI Agent Scope Drift Detection | Planned actions that stray from the authorised objective are flagged before they run |
| Sandboxed execution | AML.M0032 Segmentation of AI Agent Components | Shell and code tools cannot reach secrets, production networks or other tenants |
| Model provenance | AML.M0014 Verify AI Artifacts | Only approved, verified weights are deployed, which blocks safety-stripped variants |
See the OWASP LLM Top 10 LLM06 Excessive Agency Guide for designing tool sets, and never place secrets or authorisation rules in a system prompt (the OWASP LLM Top 10 LLM07 System Prompt Leakage Guide explains why).
Detection: what to log and where
Jailbreak detection works best as a security signal joined to identity, not as a content filter result that nobody reads. AML.M0024 AI Telemetry Logging and Monitoring is the foundation.
| Source | Field or signal | Use |
|---|---|---|
| Azure AI Content Safety Prompt Shields | User prompt attack detection (categories include attempts to change system rules, conversation mockups, role-play and encoding attacks); document attack detection for retrieved content | Per-request flag. In Microsoft Foundry, user prompt attack detection is on by default; document attack detection is off by default and must be enabled |
Microsoft Purview audit, CopilotInteraction records | Messages.JailbreakDetected; AccessedResources.XPIADetected for cross-prompt injection | Join to user and agent identity for SOC triage |
| Self-hosted guardrails and AI gateways | Input and output classifier verdicts with scores | Trend attempts per user, key and tenant |
| Agent runtime telemetry | Tool calls proposed and denied by policy after a flagged message | A denied high-risk tool call shortly after a flagged prompt is a strong indicator of a working jailbreak |
| Model provider usage logs | Repeated refusals, retries with small variations, sudden spend on unusual keys | Iteration is visible as volume; Storm-2139 iterated on blocked prompts |
Three detection patterns are worth building first: a user or key with many flagged prompts in a short window (iteration); a flagged prompt followed by a policy-denied tool call in the same session (escalation attempt); and a session in which refusals stop abruptly after a long run of them (possible success). Treat classifier flags as leads, not verdicts: they miss novel and multi-turn attempts and occasionally flag legitimate security questions.
Metrics
- Flag rate: suspected jailbreak attempts per thousand sessions, by application.
- Escalation blocks: tool calls denied by policy after a flagged prompt, per month.
- Red team bypass rate: share of the regression jailbreak set that succeeds after each model, prompt or guardrail change.
- Agents with consequential tools and no approval step: target zero.
- Unverified models in production: deployed weights without a recorded source and checksum, target zero.
How to test
- Keep a regression set by family, not by prompt. AML.M0035 asks for recurring, threat-informed red team exercises and, where appropriate, for confirmed failures to become regression tests and detection logic. NHI Mgmt Group recommends that the set covers multi-turn, multilingual, encoded, transformed and multimodal jailbreaks. The AI Red Teaming Guide and AI Red Team Test Plan Template cover scope and rules of engagement.
- Test the blast radius, not only the refusal. For each agent, assume the model has been jailbroken and test what its tools allow: can it read secrets, send externally, execute code or widen its own access?
- Test jailbreaks delivered through content. Include documents, tickets and web pages carrying jailbreak framing, since that is how several ATLAS cases reached agents.
- Re-test after every change to the model version, system prompt, guardrail policy or tool set, and roll back the change if the bypass rate rises.
Choosing where to invest
- Chat-only assistant with no tools: provider guardrails, output filtering and abuse monitoring are proportionate; the harm is content and brand.
- Agent with read tools: add per-user delegated access and retrieval logging, because a jailbreak becomes a data leakage route.
- Agent with write, send or execute tools: add approval gates, sandboxing and escalation detection; treat jailbreak as expected.
- Self-hosted open-weight models: add external guardrails and strict model provenance, since there may be no provider safety layer and modified weights are easy to obtain.
Small teams should rely on managed guardrails and keep agent tool sets small. Large platforms should run a central AI gateway that records classifier verdicts with identity and forwards them to the SOC.
Where identity fits
Storm-2139 started with leaked customer credentials, and GTG-1002 ran through accounts the operators controlled until the provider banned them. A jailbreak defence that ignores who is calling the model misses the cheapest controls: protect AI service keys, give every agent its own identity, and make sure flagged sessions can be traced to an account that can be suspended. See the LLMjacking Guide for AI key protection.
Common failure patterns
- Policy in the system prompt: the only barrier to a tool action is an instruction the model can be argued out of.
- Guardrail flags nobody reads: detections are logged in the AI platform but never reach the SOC or carry the user identity.
- Screening the chat box only: retrieved documents and tool outputs carry jailbreak framing too.
- One-off red teaming: a model upgrade or prompt change silently reopens bypasses that were fixed.
- Unvetted model downloads: a safety-stripped variant is deployed because it scored well on a benchmark.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| MITRE ATLAS | AML.T0054 LLM Jailbreak; tactics AML.TA0007 Defense Evasion and AML.TA0012 Privilege Escalation; mitigations AML.M0020, AML.M0021, AML.M0022, AML.M0035 | The technique and its listed mitigations |
| MITRE ATT&CK v20 | T1699 Direct Prompt Injection | Closest enterprise technique for user-supplied jailbreak prompts; see MITRE ATT&CK T1699 Direct Prompt Injection |
| OWASP Top 10 for LLM Applications 2025 | LLM01 Prompt Injection (which describes jailbreaking as a form of prompt injection); LLM06 Excessive Agency; LLM07 System Prompt Leakage | Classification and the agency that sets blast radius |
| NIST SP 800-53 Rev. 5 | AC-6 Least Privilege; SI-4 System Monitoring; SI-10 Information Input Validation | Agent privilege, monitoring and input screening |
| NIST CSF 2.0 | PR.AA-05; DE.CM-09 | Least-privilege permissions and runtime monitoring |
Practitioner checklist
- Move every authorisation rule and secret out of system prompts and into code and tool permissions.
- Give each agent a least-privilege identity and require approval for consequential actions.
- Enable jailbreak and document attack detection in your guardrail layer and forward flags to the SOC with identity.
- Alert on iteration, on flagged prompts followed by denied tool calls, and on abrupt ends to refusal runs.
- Sandbox code and shell tools away from secrets and production networks.
- Deploy only verified model weights from approved sources.
- Maintain a jailbreak regression set by family and re-run it after every model, prompt or guardrail change.
- Protect AI service keys and make every flagged session traceable to a suspendable account.
Standards and references
- MITRE ATLAS: LLM Jailbreak (AML.T0054)
- MITRE ATLAS: AI Red Team (AML.M0035)
- OWASP: Top 10 for LLM Applications 2025
- MITRE ATT&CK: Direct Prompt Injection (T1699)
- Microsoft Learn: Prompt Shields
- Microsoft Learn: Audit logs for Copilot and AI applications
- Anthropic: Disrupting the first reported AI-orchestrated cyber espionage campaign
- Microsoft On the Issues: Storm-2139 and the abuse of generative AI services
Related NHI Mgmt Group resources: MITRE ATLAS Guide · Prompt Injection Guide · AI Guardrails and AI Firewalls Guide · MITRE ATLAS Mitigations Guide