When an AI agent or assistant has acted on instructions hidden in content it read, the incident is not over when you stop the agent: the poisoned content is still sitting in a mailbox, a CRM field, a web page or a vector store, waiting for the next agent to read it. This playbook covers the specific case of a confirmed or suspected prompt injection, direct or indirect, that caused an agent, copilot or LLM application to leak data, call tools, change records or send messages it should not have. It is a fill-in template: replace the bracketed placeholders and attach it to your incident response plan as an annex. General agent incidents (runaway cost, stolen agent credentials, malicious MCP servers) are covered by the AI Agent Incident Response Playbook and the AI Agent Misuse Incident Playbook; this page adds the steps that are particular to injected instructions.
How to use this template
- Customise: replace every [Placeholder] with your teams, tools and contacts, and delete rows that do not apply to your estate.
- Decisions: steps marked Decision: need a named owner, a recorded outcome and a timestamp in the incident log.
- Assume residual risk: the UK NCSC states that prompt injection may never be totally mitigated, because current LLMs do not enforce a boundary between instructions and data. Plan for recurrence, not for a one-off fix.
- Rehearse: run a tabletop at least [twice a year] using a planted, harmless injection in a test mailbox or document library, and time each phase.
- Keep it short in the moment: the numbered sections are the run sheet; the Implementation guidance section is preparation work to do before you need it.
1. Scope and triggers
This playbook applies to [Organisation]'s LLM applications, copilots, RAG assistants and autonomous agents, whether built in house, embedded in SaaS (for example CRM or office suite assistants) or used through a provider API. Prompt injection is MITRE ATLAS AML.T0051 LLM Prompt Injection, with sub-techniques AML.T0051.000 Direct and AML.T0051.001 Indirect, and OWASP LLM01:2025 in the Top 10 for LLM Applications.
| Trigger | Typical signal | Initial severity |
|---|---|---|
| Agent output contains a URL or image link carrying data in its query string | Egress or proxy log shows requests to an unfamiliar domain right after an agent response renders | High (possible exfiltration, ATLAS AML.T0085 Data from AI Services) |
| Agent called a tool the task did not need | Tool-call trace shows email send, file share, record update or code execution outside the user's request (AML.T0053 AI Agent Tool Invocation) | High if the tool writes or sends; Medium if read only |
| Data sent out through a tool call | Outbound message, webhook or API call containing records the user did not ask for (AML.T0086 Exfiltration via AI Agent Tool Invocation) | Critical where personal or regulated data is involved |
| Instruction-like text found in content the agent reads | Content scanner, classifier (for example a prompt-shield service) or user report flags hidden text in email, tickets, form fields, PDFs or web pages | Medium until you confirm whether an agent processed it |
| Poisoned memory or retrieved context | Agent repeats an instruction across sessions, or a vector store chunk contains directives (AML.T0080 AI Agent Context Poisoning) | High: affects every user of that memory or index |
| Vendor or researcher disclosure | Advisory for an assistant you use, as with the EchoLeak Microsoft 365 Copilot vulnerability | Set by exposure: did the vulnerable feature process external content in your tenant? |
Severity guide: Critical = confirmed exfiltration of regulated data or an irreversible action (payment, deletion, external message). High = tool misuse or data exposure inside [Organisation]. Medium = injection present and processed, no harmful action found yet. Low = injection found in content no agent has read. Response targets: Critical [15 minutes to containment decision], High [1 hour], Medium [same business day].
2. Roles and contacts
| Role | Responsibility in a prompt injection incident | Contact |
|---|---|---|
| Incident manager | Runs the incident, owns the log and every Decision step | [Name, phone] |
| SOC analyst | Pulls prompt, response and tool-call traces; egress and proxy logs; builds the timeline | [SOC rota] |
| Agent or application owner | Knows the system prompt, tools, data sources and memory; executes containment on the agent | [Owner per agent, from the agent registry] |
| Content source owner | Owns the system the injection came from (mail, CRM, document library, website, ticketing) and quarantines the carrier | [Per source] |
| IAM or NHI team | Revokes and reissues the agent's tokens, OAuth grants and API keys; reviews what the agent's identity can reach | [IAM on call] |
| AI platform or ML engineer | Inspects retrieval, memory and vector stores; deploys guardrail and filter changes | [Platform team] |
| Data protection officer and legal | Decides on GDPR and contractual notification; advises on regulator and customer wording | [DPO, counsel] |
| SaaS or model provider | Provider-side logs, fixes and confirmation for embedded assistants | [Support route, contract reference] |
3. Detection and triage
- Preserve before you change anything. Export the full conversation or task trace: user prompt, system prompt version, retrieved chunks with their source IDs, tool calls with arguments and responses, model output as rendered, and timestamps. Snapshot the agent's memory store and configuration. Hash the exports and record where they are stored.
- Find the carrier. Identify the exact content that held the injected instruction: the email message ID, CRM record and field, document path and version, URL and fetch time, ticket number or tool response. Copy the payload text into the evidence store as inert text (for example in a .txt file) and never paste it into another LLM tool for analysis.
- Classify the path. Direct (a user typed it), indirect (the agent read it from external content), or stored (it came from memory or a vector index that earlier content poisoned). The path decides who else is exposed.
- Reconstruct actions. List every tool call and output made after the carrier entered context. For each: what it did, which identity and scopes it used, whether it can be reversed.
- Check for outbound channels. Search agent outputs for links, Markdown images (including reference-style syntax), HTML, and calls to webhooks or messaging tools. Correlate with proxy, DNS and CDN logs to see whether a client or server actually fetched them.
- Measure fan-out. Query for every other agent, user session and scheduled job that retrieved the same carrier or index chunk in the exposure window. Injected content in a shared mailbox, CRM object or knowledge base usually reaches more than one session.
- Decision: confirm severity from section 1 and whether this is a vulnerability in the product (vendor fix needed), a design gap in [Organisation]'s agent (excessive tools or access), or both.
4. Containment
| Action | How | When |
|---|---|---|
| Quarantine the carrier | Move or tag the email, lock or redact the CRM field, unpublish or exclude the document, block the URL from retrieval | Always, first |
| Remove poisoned content from indexes and memory | Delete affected chunks from the vector store by source ID; clear or roll back the agent's long-term memory to a pre-incident snapshot | Stored or indirect paths |
| Close the exfiltration channel | Block the destination domain at proxy and DNS; disable external image and link rendering in agent output; restrict the content security policy and URL allowlist to domains you control | Any outbound signal |
| Restrict the agent | Remove write and send tools, switch to read only, or require human approval for every action | High |
| Pause or kill the agent | Use the kill switch described in the AI Agent Kill Switch and Containment Guide | Critical, or when fan-out is unknown |
| Revoke the agent's credentials | Revoke OAuth grants, refresh tokens and API keys the agent used; reissue only after eradication | Any confirmed tool misuse |
Quarantining the carrier comes before stopping the agent, because stopping one agent does not stop a second agent, a scheduled job or a colleague's copilot from reading the same content. In the ForcedLeak Agentforce case the payload sat in a public Web-to-Lead Description field until an employee asked the agent to process the lead, and data left through an image URL on an allowlisted domain that had expired.
5. Eradication
- Hunt for siblings: search every source the agent reads for the payload, its distinctive phrases and variants (other languages, encodings, white-on-white text, document metadata, alt text). Payloads are often split across several items so that each looks harmless.
- Write a detection from the payload: add the indicators to content scanning and to the classifier tuning set, accepting that rephrased payloads will evade phrase lists.
- Fix the design gap: remove tools the task does not need, scope the agent identity to the minimum data, and require approval for irreversible or external actions. See the Human-in-the-Loop for AI Agents Guide.
- Separate untrusted input: mark external content as data in the prompt (delimiting, datamarking or encoding, the spotlighting approaches Microsoft describes) and keep agents that read untrusted content away from privileged tools.
- Rotate what was reachable: any secret, token or document link that appeared in the agent's context during the exposure window should be treated as disclosed.
- Apply vendor fixes: for embedded assistants, obtain written confirmation of the provider-side fix and any tenant settings you must change.
6. Recovery
- Reverse unauthorised actions: recall or correct messages sent, restore changed records from audit history or backup, cancel scheduled tasks the agent created.
- Rebuild affected indexes from clean sources rather than patching them in place, if you cannot prove which chunks were poisoned.
- Replay the original payload and a set of variants against the fixed agent in a staging environment. Re-enable only when none trigger a tool call or outbound request.
- Decision: approve return to service, with reduced scope where appropriate, by the agent owner and the incident manager.
- Run heightened monitoring for [14 days]: alert on any tool call outside the agent's baseline and on any output containing external URLs.
7. Communication and notification
- Internal: brief affected users on what the agent did in their name and what they should check, such as sent items or shared files.
- Personal data: if personal data left [Organisation], the DPO assesses GDPR Article 33 notification to the supervisory authority within 72 hours of becoming aware, and Article 34 notification to individuals where the risk is high. See the Data Breach Notification Checklist.
- Sector and AI rules: check sector incident reporting (for example DORA or NIS2) and whether the system falls under EU AI Act incident obligations. See the EU AI Act Provider vs Deployer Obligations Guide.
- Providers: report the injection path to the SaaS or model provider through its security disclosure route, with the payload as inert text.
- Accuracy: say "an assistant followed instructions hidden in [source]", not "the AI was hacked", and do not state that no data left until egress logs confirm it.
8. Post-incident review
- Hold a blameless review within [10 working days] with the agent owner, content source owner and SOC.
- Answer: which untrusted source could reach the agent, why the agent had the tool that was misused, why the output channel was open, and how long the carrier sat before detection.
- Update the agent's threat model with the path observed. See the AI Agent Threat Model Template.
- Add the payload and variants to regression tests that run on every prompt, tool or model change.
- Record time to detect, time to quarantine the carrier and number of sessions exposed, and report them to the AI governance forum.
Implementation guidance
Prepare before an incident
- Trace logging: retain prompts, retrieved chunk IDs, tool calls and rendered outputs for at least [90 days]. The NCSC recommends logging inputs, outputs, tool use and API calls, including failed tool calls, which can signal an attack in progress.
- Source lineage: every chunk in a vector store must carry its source ID and ingestion time, or you cannot delete poisoned content selectively. See the RAG Security Guide.
- Agent registry: owner, tools, data sources and identity for every agent, so triage does not stall.
- Output controls: disable rendering of external images and links in agent output by default and allow only domains you own and monitor.
- Containment switches tested: read-only mode, approval mode and full stop for each agent, rehearsed.
Framework mapping
| Framework | Reference | Playbook section |
|---|---|---|
| NIST SP 800-61 Rev. 3 (CSF 2.0 Community Profile) | Incident response integrated with CSF 2.0 | Whole playbook |
| NIST CSF 2.0 | DE.AE-02, RS.AN-03, RS.CO-02, RC.RP-01 | 3, 3 and 8, 7, 6 |
| NIST SP 800-53 Rev. 5 | IR-4 Incident Handling, IR-5 Incident Monitoring, IR-6 Incident Reporting, IR-8 Incident Response Plan, SI-4 System Monitoring, AU-6 Audit Record Review | 3 to 8 |
| MITRE ATLAS | AML.T0051, AML.T0053, AML.T0080, AML.T0085, AML.T0086 | 1, 3 |
| OWASP Top 10 for LLM Applications 2025 | LLM01:2025 Prompt Injection | 1, 5 |
Metrics
- Mean time from first detection to carrier quarantine.
- Percentage of agents with complete trace logging and chunk-level source lineage.
- Number of agents that read untrusted content and also hold write or send tools without approval.
- Regression pass rate for the injection test set after each release.
- Exercises completed per year against target.
For background on the attack class and prevention controls, see the Prompt Injection Guide and the OWASP LLM Top 10 LLM01 Prompt Injection Guide.
Standards and references
- NIST: SP 800-61 Rev. 3 Incident Response Recommendations and Considerations for Cybersecurity Risk Management
- OWASP: Top 10 for LLM Applications 2025
- MITRE ATLAS: LLM Prompt Injection
- NCSC: Prompt injection is not SQL injection (it may be worse)
- Microsoft MSRC: How Microsoft defends against indirect prompt injection attacks
- NIST: SP 800-53 Rev. 5
Related NHI Mgmt Group resources: AI Agent Incident Response Playbook · Prompt Injection Guide · AI Agent Goal Hijacking Guide · AI Agent Observability and Incident Response Guide