Before a prompt injection or jailbreak works against your system, someone usually had to make it work, and that tuning often happens on your own endpoint where you can see it. MITRE ATLAS tracks this as AML.T0065 LLM Prompt Crafting, in the AI Attack Adaptation tactic (AML.TA0001). ATLAS describes adversaries using knowledge of the target generative AI system to craft prompts that bypass its defences so malicious instructions execute, and iterating on the prompt until it works consistently. This page is about that adaptation phase: what crafting and probing look like in prompt and gateway logs, how to rate-limit and fingerprint it, and how to use it as early warning. It is for teams that operate LLM applications, AI gateways and agent platforms. It does not repeat the injection and jailbreak techniques themselves: see the Prompt Injection Guide and the LLM Jailbreaks Guide.
Key takeaways
- Crafting is a phase, not a payload. AML.T0065 sits between reconnaissance and execution: the attacker turns what they learned about your system into a prompt that works reliably.
- Iteration leaves a trail. Many near-identical prompts, rising rates of guardrail hits followed by a sudden clean success, and template reuse across accounts are all visible if you log at the right layer.
- Some crafting happens elsewhere. ATLAS case study AML.CS0026 records researchers tuning prompts by trial and error on a separate Copilot instance, so the first time you see the final prompt may be the attack.
- ATLAS maps no direct mitigation. As of the ATLAS 2026.10 data, no mitigation is mapped to AML.T0065; the useful controls come from query limits, guardrails, telemetry, red teaming and honeypots.
- Treat probe detections as intelligence. A blocked crafting session tells you which defence the attacker is working on and what they are trying to reach.
What the technique covers
ATLAS created AML.T0065 on 12 March 2025, last modified it on 28 October 2026 and rates its maturity as Realized, meaning it has been used in real incidents, not only research. The 2026.10 update extends the description to systematic prompt sets used to collect model outputs for distillation (AML.T0005 Replicate AI Model Behavior), noting that their adversarial purpose may only be visible in aggregate. It is one of the most frequently referenced techniques in ATLAS case studies, because almost every prompt-based attack needs a working prompt. The ATLAS data gives it no ATT&CK equivalent: it is an AI-specific step.
The case studies show three distinct forms, and each is visible in a different place.
| Form | ATLAS example | Where you can see it |
|---|---|---|
| Offline tuning on a copy of the system | AML.CS0026: prompts that bypassed Copilot's system instructions were developed by trial and error on a separate Copilot instance | Not on your endpoint; only the final attempt arrives. Rely on content guardrails and outcome monitoring. |
| Live probing of the target | AML.CS0037: researchers sent prompts designed to probe whether an AI agent was monitoring an inbox, before crafting the data-collection prompt | Your inputs, your agent's tool calls and outbound actions |
| Bulk prompt generation at scale | AML.CS0056: three AI labs generated large datasets of prompts designed to extract capabilities from Claude | Traffic analytics across accounts, payment and network signals |
| Classifier evasion | AML.CS0059 (EchoLeak): instructions crafted to evade Microsoft's indirect prompt injection classifier and look like ordinary business content | Retrieved content, rendered output, outbound requests |
Two live incident pages show the evasion form. In EchoLeak 2025 the email was worded to avoid Microsoft's cross-prompt injection classifiers. In Gemini AI Calendar Prompt Injection 2026, Miggo Security reported that instructions phrased like a normal user request passed Google's separate prompt detection model. In both cases the crafting goal was to look benign to a classifier, which is why detection cannot rely on the classifier alone.
What crafting and probing look like in logs
Most live crafting has a recognisable shape: an identity sends a burst of related prompts, guardrails fire on early attempts, the wording shifts step by step, and either the attempts stop or one gets through. Anthropic's February 2026 report on distillation campaigns gives a scale example: about 24,000 fraudulent accounts and more than 16 million exchanges, with highly repetitive, templated prompts sent tens of thousands of times across hundreds of accounts and concentrated on narrow capabilities. Anthropic describes detecting this with classifiers, behavioural fingerprinting of API traffic and tools for spotting coordinated activity across accounts.
| Signal | Log source | What to compute |
|---|---|---|
| Near-duplicate prompts | Prompt logs, AI gateway | Similarity clusters per identity per hour (embedding or n-gram distance); many small edits of one base prompt |
| Guardrail hit then success | Guardrail or classifier results | Sessions where several blocked or flagged inputs are followed by an allowed input on the same topic |
| Template reuse across accounts | Gateway, account and billing data | Same prompt skeleton from many identities; synchronised timing; shared payment method or network |
| Self-referential questions | Prompt logs | Questions about system instructions, tools, sources or filters, which often precede crafting |
| Encoding and format changes | Prompt logs, guardrail categories | Sudden switches to encodings, other languages, role-play or conversation mock-ups within a session |
| Agent probe outcomes | Agent tool-call and outbound logs | Replies or tool calls to destinations never seen for that agent, triggered by inbound content |
Classifier categories help label sessions. Microsoft Prompt Shields, for example, reports user prompt attacks and document attacks, and lists categories such as attempts to change system rules, embedded conversation mock-ups, role-play and encoding attacks. The service returns a detection result and leaves blocking to the application, so count detections per identity rather than only blocking single requests. Microsoft also notes the models are primarily trained on English, so test each language you serve. Obfuscation as a technique in its own right is covered in MITRE ATLAS AML.T0068 LLM Prompt Obfuscation.
Controls that raise the cost of crafting
- Limit query volume and rate per identity (AML.M0004). ATLAS recommends limits by user, API key, tenant or device, combining short-term rate, burst and concurrency limits with longer-term quotas. Iteration needs many attempts; quotas make each attempt cost more. See the Rate Limiting and API Abuse Guide.
- Add a guardrail-hit budget. Beyond a plain rate limit, slow down or suspend an identity after a set number of guardrail detections in a window. This targets crafting specifically without hurting normal heavy users.
- Put guardrails on inputs, retrieved content and outputs (AML.M0020). Crafted prompts are designed to pass one check; checking retrieved documents and outgoing actions as well means a prompt that passes input screening can still be caught by what it tries to do.
- Restrict tool use when untrusted data is in context (AML.M0030). Many crafted prompts in the ATLAS case studies aim at tool invocation: sending email, running commands, writing files. If tool use is limited while untrusted content is present, a successful prompt achieves less.
- Keep system details out of reach. ATLAS describes crafting as using acquired knowledge of the target. Limit what users can learn about your system prompt, tools and filters; see the OWASP LLM Top 10 LLM07 System Prompt Leakage Guide.
- Strengthen account verification where abuse concentrates. Anthropic reported that educational, security research and startup account types were commonly exploited and described stronger verification for them. For your own service, tie higher quotas to stronger identity proofing.
- Deploy decoys (AML.M0039 AI Honeypots). A decoy agent, inbox or endpoint that no legitimate user should touch turns any probing of it into a high-confidence alert.
Detection and measurement
All of the above depends on AML.M0024 AI Telemetry Logging and Monitoring: inputs, outputs, guardrail results, agent intermediate steps, tool calls and the identity behind each request. Log at the AI gateway so every application is covered the same way; see the AI Gateway and LLM Gateway Guide and LLM Observability Guide.
- Session rule: three or more guardrail detections from one identity within 15 minutes, followed by an allowed request on a similar topic, opens a review. Tune the numbers to your traffic.
- Cluster rule: more than a set number of identities sending prompts with the same skeleton within an hour.
- Agent rule: any agent tool call to a new external destination following inbound content from an outside sender.
| Metric | Why it matters |
|---|---|
| Share of LLM traffic with full prompt, guardrail and identity logging | Crafting detection is impossible without it; target 100% |
| Guardrail detections per 1,000 requests, by identity type | Baseline for spotting probing spikes |
| Sessions matching the hit-then-success pattern per week | Direct measure of successful adaptation attempts |
| Median time from crafting alert to review | Probing is early warning only if someone looks |
| Red team bypasses found and fixed per quarter | Shows whether your own crafting finds gaps before attackers |
How to test
Use AML.M0035 AI Red Team: run your own authorised crafting campaign and check that it is seen. The aim is to test the detection, not only the model.
- Build or reuse a prompt injection and jailbreak test set; see the AI Red Team Test Plan Template.
- Run it from a test identity in iterative mode: start with known-blocked prompts and vary them, as an attacker would.
- Confirm the session rule, the guardrail-hit budget and the rate limits fire, and record the time to alert.
- Repeat from several test identities with the same template to exercise the cluster rule.
- Feed any successful bypass back into guardrail tuning, then re-run. Keep tests in a non-production tenant where the model has no real tools or data.
For a structured approach, see the OWASP GenAI Red Teaming Manual Guide.
Common failure patterns
- Blocking without counting. Each blocked prompt is handled alone, so a hundred attempts from one identity never raise an alert.
- Trusting one classifier. EchoLeak and the Gemini calendar case both passed a dedicated injection classifier because the text was crafted to look ordinary.
- Logging only completions. Without guardrail results and tool calls in the same record, the hit-then-success pattern cannot be reconstructed.
- Quotas per key, not per identity. Attackers rotate keys or accounts; Anthropic described networks of fraudulent accounts replaced as they were banned.
- Red teaming the model but not the monitoring. A test that finds a bypass but never checks whether the SOC saw it misses half the value.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| MITRE ATLAS | AML.T0065, tactic AML.TA0001 AI Attack Adaptation | The technique |
| MITRE ATLAS | AML.M0004, AML.M0020, AML.M0024, AML.M0030, AML.M0035, AML.M0039 | Query limits, guardrails, telemetry, tool restriction, red team, honeypots |
| MITRE ATLAS | AML.T0051 LLM Prompt Injection, AML.T0054 LLM Jailbreak, AML.T0068 LLM Prompt Obfuscation | What the crafted prompt is usually used for |
| OWASP Top 10 for LLM Applications 2025 | LLM01 Prompt Injection, LLM07 System Prompt Leakage, LLM10 Unbounded Consumption | Injection target, recon input, volume abuse |
| NIST SP 800-53 Rev. 5 | SI-4 System Monitoring, SI-10 Information Input Validation, AU-6 Audit Record Review, Analysis, and Reporting, SC-5 Denial-of-service Protection | Monitoring, input checks, review, volume limits |
Practitioner checklist
- Route all LLM traffic through a gateway that logs prompt, guardrail result, tool calls and identity.
- Set per-identity rate, burst and quota limits, not only per-key limits.
- Add a guardrail-hit budget that slows or suspends repeated offenders.
- Alert on the hit-then-success session pattern and on template reuse across identities.
- Screen retrieved content and outbound agent actions, not only user input.
- Restrict tool use when untrusted content is in context.
- Deploy at least one decoy agent or endpoint and alert on any use.
- Run a quarterly authorised crafting campaign and measure time to alert.
Standards and references
- MITRE ATLAS: LLM Prompt Crafting (AML.T0065)
- MITRE ATLAS: Limit AI Service Query Volume and Rate (AML.M0004)
- MITRE ATLAS: Model Distillation Campaigns Targeting Anthropic Claude (AML.CS0056)
- MITRE ATLAS: EchoLeak (AML.CS0059)
- OWASP: Top 10 for LLM Applications 2025
- Anthropic: Detecting and preventing distillation attacks
- Microsoft Learn: Prompt Shields
- Zenity Labs: A Copilot Studio Story, discovery phase in AI agents
Related NHI Mgmt Group resources: LLM Jailbreaks Guide, OWASP LLM Top 10 LLM01 Prompt Injection Guide, Model Extraction and Distillation Defence Guide, AI Red Teaming Guide, MITRE ATLAS Guide.