MITRE ATLAS AML.T0061 LLM Prompt Self-Replication: Detection, Mitigation and Response

A prompt that makes a model copy the prompt into its own output turns one injection into something that spreads between assistants, inboxes and retrieval stores without further attacker effort. MITRE ATLAS tracks this as AML.T0061 LLM Prompt Self-Replication, in the Persistence tactic (AML.TA0006). ATLAS describes a crafted prompt injection that causes the LLM to replicate the prompt in its output, so it propagates to other LLMs and persists on the system, usually paired with other malicious instructions such as a jailbreak or data leakage. The general mechanics of prompt injection are in the Prompt Injection Guide and the framework overview is in the MITRE ATLAS Guide. This page covers what they do not: how self-replication propagates, where it can persist, how to detect worm-like behaviour in GenAI pipelines, and how to contain an outbreak. It is for teams building email, chat and document assistants that read content and write content that other assistants later read.

Key takeaways

• Replication needs a write path. A self-replicating prompt only spreads where model output is stored, sent or indexed somewhere another model will read it, so mapping those write paths is the first defensive task.
• Retrieval stores are the persistence layer. In the Morris II research recorded as ATLAS case study AML.CS0024, the malicious email was stored in a RAG database and re-retrieved for future replies, which is how it persisted and spread.
• Zero-click is the norm. Auto-reply, auto-summarise and auto-forward features let the prompt move without any human reading it.
• ATLAS names three mitigations. AML.M0020 Generative AI Guardrails, AML.M0021 Generative AI Guidelines and AML.M0022 Generative AI Model Alignment are linked to AML.T0061; our view is that architecture controls on what output may be stored or sent matter just as much.
• Response is containment first. Pause automatic sending, find every store that holds the prompt, purge it, and only then re-enable automation.

How the technique works

ATLAS created AML.T0061 in March 2025 and rates its maturity as demonstrated. It sits after initial prompt injection (AML.T0051, usually the indirect sub-technique AML.T0051.001) and before whatever the attacker wants the payload to do. The replicating part keeps the infection alive; the payload part does the harm.

The defining feature is that the model's output becomes the next model's input. A one-off injection ends when the session ends. A self-replicating one survives because the application writes it somewhere durable or sends it to someone new.

The Morris II research

The reference case is the Morris II work by Stav Cohen, Ron Bitton and Ben Nassi, published on arXiv as 2403.02817 and recorded in ATLAS as AML.CS0024. The researchers framed the adversarial self-replicating prompt as a confusion between data and code, comparable to SQL injection, and tested two kinds of email assistant.

• RAG-based assistant (passive, zero-click): an auto-responder stored new correspondence in a RAG database. A malicious email was stored, later retrieved as context for an unrelated reply, and the generated reply carried the prompt and sensitive data onward. The paper tested this against GPT-4 and Gemini Pro and reported reliable replication and payload execution at small context sizes, with success falling as more correspondence was retrieved.
• Flow-steering assistant (active): a prompt embedded in an image made a multimodal model classify the message as one to forward, so the client sent it to new recipients straight away.
• Payloads demonstrated: the paper describes spam, propaganda, phishing and exfiltration of personal data as payloads that ride along with replication.

The paper proposed countermeasures rather than measuring them: rephrase model output so it does not mirror input fragments, harden models against jailbreaks, monitor agent interactions and third-party services such as SMTP and messaging for worm-like patterns, and use a non-active RAG that does not store new replies or forwarded mail, at a cost to usefulness. The RAG Security Guide covers retrieval hardening more broadly.

Where it can propagate in your estate

Inventory every path where model output is written somewhere another model will read. The table lists the common ones, with the control that breaks each path.

A useful rule of ours: any loop where model output can reach another model's context without a human or a deterministic filter in between is a potential replication path, and should be on the risk register.

Detection

ATLAS mitigation AML.M0024 AI Telemetry Logging asks for logs of model inputs and outputs and, for agents, intermediate steps, data access and tool use. Self-replication is visible in that record if you look for repetition across boundaries rather than for a single bad prompt.

D3FEND D3-UBA User Behavior Analysis applies to the fan-out signal when assistants act under user identities, and D3-OTF Outbound Traffic Filtering applies to the mail and messaging egress where replication leaves the organisation.

Measuring it

• Write-path coverage: share of identified output-to-input loops that have a human or deterministic filter in between.
• Echo rate: share of completions where the output reproduces a long span of untrusted input; trend it after guardrail changes.
• Auto-send share: percentage of AI-generated outbound messages sent without human review.
• Time to quarantine: time from first detection of a replicating prompt to the last copy being removed from every store.

Prevention and hardening

The steps below are NHI Mgmt Group recommendations, drawn from the ATLAS mitigations and our own practice.

• AML.M0020 Generative AI Guardrails: ATLAS says guardrails can help prevent replication attacks in model inputs and outputs. In practice that means an output check that flags completions repeating untrusted input or containing instruction-like text addressed to an AI. See the AI Guardrails and AI Firewalls Guide.
• AML.M0021 Generative AI Guidelines: system prompt rules that the model must never reproduce instructions found in retrieved content. Treat this as a weak layer: it is itself a prompt.
• AML.M0022 Generative AI Model Alignment: ATLAS notes alignment can increase resistance; this is mainly the model provider's lever, so check provider evaluations rather than assuming it.
• Break the loop: do not auto-send, do not index assistant output into the same collection as human content, and keep agent-written records labelled so other agents can treat them as untrusted.
• Restrict tools on untrusted input (AML.M0030): when an email or document from outside has entered context, block or require confirmation for send, forward, post and write tools.
• Least privilege for assistant identities: an assistant that can only draft, not send, cannot spread anything by itself. The assistant's mail and API permissions are non-human identity decisions; review them like any service account.

Response runbook

1. Contain: switch affected assistants to draft-only or disable automatic actions; block the identified text pattern at the mail gateway and messaging platform.
2. Scope: search mailboxes, retrieval indexes, memory stores, shared threads and generated documents for the prompt and close variants; list every user and agent that processed it.
3. Purge: remove copies from every store, then rebuild or re-embed affected index segments so no stale vectors remain.
4. Assess payload impact: check what data the payload targeted and whether it left the organisation; follow the Prompt Injection Incident Response Playbook for evidence and notification decisions.
5. Re-enable in stages: restore automation for a pilot group with the new output check in place, watch echo and fan-out metrics, then widen.
6. Feed back: add the sample to regression tests so the guardrail and filters are tested on every model or prompt change; see AI security regression testing.

Common failure patterns

• Treating it as a model problem only: relying on the model to refuse while the application auto-sends and auto-indexes everything it writes.
• Indexing everything: adding assistant replies and summaries to the same retrieval collection as human records, which gives the prompt a permanent home.
• Purging one store: deleting the original email but leaving copies in other mailboxes, memories and the vector index.
• No output logging: logging prompts but not completions, so echo and fan-out cannot be measured.
• Agent-to-agent trust: letting one agent's free-text output drive another agent's tools with no validation.

Framework mapping

Practitioner checklist

• Inventory every path where model output is stored, sent or indexed where another model reads it.
• Put a human approval or deterministic filter on each of those paths.
• Keep assistant-written content out of human-content retrieval collections, or label it as lower trust.
• Log prompts, retrieved context and completions, and compute an input echo score.
• Alert on fan-out spikes in AI-sent mail, posts and forwards.
• Block send, forward and write tools when untrusted content is in context, or require confirmation.
• Rehearse a purge across mailboxes, indexes, memories and threads, including re-embedding.
• Add replication samples to regression tests for every model and prompt change.

Standards and references

• MITRE ATLAS: LLM Prompt Self-Replication (AML.T0061)
• MITRE ATLAS: Morris II Worm: RAG-Based Attack (AML.CS0024)
• arXiv: Cohen, Bitton and Nassi, Morris II adversarial self-replicating prompts (2403.02817)
• MITRE ATLAS: Generative AI Guardrails (AML.M0020)
• OWASP: Top 10 for LLM Applications 2025
• Microsoft MSRC: How Microsoft defends against indirect prompt injection attacks
• NIST: SP 800-53 Rev. 5

Related NHI Mgmt Group resources: MITRE ATLAS Guide, Prompt Injection Guide, Self Propagating Supply Chain Worms, RAG Security Guide, MITRE ATLAS AML.T0080 AI Agent Context Poisoning.