AI Agent Misuse Incident Playbook: Containing a Compromised or Misbehaving Agent

When an AI agent is hijacked, poisoned or simply goes wrong, it keeps acting at machine speed with real credentials until someone with the authority and the means stops it. Most incident response plans assume a human attacker on a host or an account. An agent incident is different: the "actor" is a legitimate non-human identity, the evidence lives in prompts, tool calls and model outputs that are often not logged, and the agent may give a misleading account of what it did. This template is a playbook for one incident type: misuse of an AI agent, whether through prompt injection, a malicious tool or MCP server, stolen agent credentials or unsafe autonomous behaviour. It is written for security operations, platform and AI engineering teams, with decision points for incident managers and executives. Adapt the roles, timings and thresholds to [Organisation].

How to use this template

• Replace text in square brackets, such as [Organisation], [Incident manager] and [Agent platform team], with your own names, teams and contact routes.
• Each phase is a numbered list of steps with a named owner. Steps marked Decision: need a recorded decision, the person who made it and the time.
• Use it with your main incident response plan; it does not replace it. See the Incident Response Plan Template and the AI Agent Incident Response Playbook.
• Steps that depend on preparation (an agent inventory, per-agent identities, a kill switch, log retention) are flagged in Implementation guidance. Test the playbook in a tabletop exercise before you rely on it.

1. Scope and triggers

This playbook applies to any AI agent that can take actions on [Organisation] systems or data: internally built agents, coding agents, low-code agents, third-party agents holding OAuth grants or API keys, and the tools and MCP servers they call. Invoke it when any of the following is observed:

Out of scope: misuse of a chat assistant with no tools or credentials (handle as a data handling or acceptable use issue), and model quality complaints with no security impact. Unknown agents found during an investigation are in scope; see the Unsanctioned AI Agents and MCP Servers Guide.

2. Roles and contacts

3. Detection and triage

1. [SOC analyst] opens an incident record and identifies the agent from the inventory: agent ID, owner, model, platform, tools and MCP servers, credentials, and the principal it acts for. If the agent is not in the inventory, record that as a finding and continue.
2. [SOC analyst] starts evidence preservation immediately (step 4.1), because agent platforms often keep conversation and trace data for short periods.
3. [SOC analyst] establishes what the agent can reach: permissions of each credential, write and delete rights, payment or messaging tools, and whether it can create credentials or call other agents.
4. [SOC analyst] establishes what it did: the last tool calls, targets and volumes, compared with its normal baseline.
5. Decision: [Incident manager] sets severity. Critical: the agent can still act and has destructive, financial, external messaging or production write capability, or customer data has left [Organisation]. High: suspected misuse with sensitive read access. Medium: anomaly with no confirmed impact.
6. Decision: [Incident manager] decides whether the cause is likely external (injection, malicious tool, stolen credential) or internal (unsafe design, faulty instructions). This changes notification and eradication, not the first containment steps.

Do not ask the agent what it did and treat the answer as evidence. In the Replit database deletion, the agent said rollback was impossible when it was not. Use independent platform, gateway and target system logs.

4. Containment

1. Preserve evidence ([SOC analyst]). Export before you change anything, or in parallel where the agent is still acting: system prompt and configuration version, conversation and session transcripts, tool call logs with arguments and results, retrieved documents and memory entries, MCP server versions and tool descriptions, gateway and proxy logs, and the identity provider and cloud logs for every agent credential. Hash the exports and store them in a restricted location with chain of custody (NIST CSF 2.0 RS.AN-06; SP 800-53 AU-9).
2. Stop the agent ([Agent platform team]). Use the kill switch or disable the deployment, scheduled triggers and webhooks. Stopping the runtime is not enough on its own: queued jobs and other agents may still call it.
3. Revoke its access ([Agent platform team], [IAM team]). Disable the agent identity, revoke OAuth refresh tokens and grants (RFC 7009 token revocation where supported), rotate API keys, revoke cloud sessions for assumed roles, and remove the agent from groups. OWASP's ASI10 guidance calls for kill switches and credential revocation to disable rogue agents quickly.
4. Isolate tools and MCP servers ([Agent platform team]). Disable the suspected tool or MCP server for all agents, not only this one, and block its network egress. If a malicious package is suspected, uninstall it everywhere and block its domains: in the postmark-mcp case one added line copied every email sent through the agent to an attacker.
5. Cut injection sources ([SOC analyst]). Quarantine the email, ticket, document, web page or repository file carrying the injected instructions, and remove poisoned memory or vector store entries after copying them as evidence.
6. Protect targets ([Data owner]). Freeze or snapshot affected databases and repositories, pause outbound payments or messages queued by the agent, and check backups are intact and outside the agent's reach.
7. Decision: [Incident manager] chooses between full shutdown and reduced mode (read-only, no external tools, human approval on every action). Choose full shutdown for Critical incidents or when the cause is unknown.

5. Eradication

1. [SOC analyst] finds the root cause: injection path, malicious or vulnerable tool, stolen credential, over-broad permission, faulty instruction or missing approval gate.
2. [IAM team] hunts for persistence created through the agent's access: new users, keys, OAuth apps, mailbox rules, webhooks, scheduled jobs, repository commits, CI secrets or other agents.
3. [Agent platform team] removes or replaces the malicious component, pins tool and MCP server versions to reviewed releases, and purges poisoned memory.
4. [Agent owner] reduces permissions to what the task needs. Where a broad token was the cause, as in the PocketOS deletion where an account-wide token found in the codebase deleted production, replace it with a scoped, short-lived credential issued to the agent's own identity.
5. [Agent platform team] adds enforced controls for the failure: approval for destructive actions, egress allow-lists, rate and spend limits. Natural language instructions are not controls.

6. Recovery

1. [Data owner] restores changed or deleted data from known-good backups and verifies integrity.
2. [Agent platform team] redeploys the agent from a reviewed configuration with new credentials. Test it in a non-production environment, ideally replaying the incident inputs against the fixed version.
3. Decision: [Agent owner] and [Incident manager] approve return to service. OWASP ASI10 guidance calls for fresh attestation, dependency verification and human approval before a quarantined agent is reintegrated.
4. [SOC analyst] applies heightened monitoring for [30] days: alerts on new tools, new destinations, volume spikes and use of any revoked credential.

7. Communication and notification

• Internal: notify the agent's users and principal owners that it is paused, with a workaround. Brief executives for Critical and High incidents within [2] hours.
• Personal data: [Legal and privacy] assess whether the incident is a personal data breach; under UK and EU GDPR Article 33, a notifiable breach goes to the supervisory authority within 72 hours of becoming aware of it.
• Third parties: report malicious packages and MCP servers to the registry and publisher; tell the agent platform or model provider through their security contact; tell customers whose data or accounts the agent touched, as contracts require.
• Regulated sectors: check sector rules such as DORA for financial entities and NIS2 for essential and important entities.
• Accuracy: do not state that "the AI did it" as if that ends accountability; describe the access the agent held and the controls that failed.

8. Post-incident review

1. [Incident manager] holds a blameless review within [10] working days and records the timeline: first malicious input, first harmful action, detection, kill switch, credential revocation and recovery.
2. Answer: was the agent in the inventory with an owner? Did it have its own identity? Which permission made the harm possible? Did logs show every tool call? How long did revocation take?
3. Update the agent's risk assessment, threat model and detection rules, and feed new tests into red teaming. See the Red Teaming AI Agents for Identity Abuse guide.
4. Track actions to closure with owners and dates, and update this playbook.

Implementation guidance

Prepare before you need it

1. Inventory: every agent with owner, tools, MCP servers, credentials and data access. See the Shadow AI Discovery Guide.
2. Identity: one identity per agent, never a shared developer or service account, so revocation is surgical. See the Agentic AI Identity Guide.
3. Kill switch: a tested way to stop each agent and revoke its credentials within minutes, operable by the on-call team.
4. Logging: prompts, tool calls, arguments, results and identity used, retained for at least [90] days outside the agent's reach (SP 800-53 AU-11). See the AI Agent Observability and Incident Response Guide.
5. Tool isolation: allow-listed tools and MCP servers with pinned versions and per-server credentials. The MCP security best practices require that MCP servers do not accept tokens not issued to them (no token passthrough). See the MCP Security Guide.
6. Exercise: run a tabletop for an injection scenario and a destructive action scenario, then a live kill switch test in staging. Add the steps to your SOC Runbook Template.

Decision guidance

• Shut down or degrade: shut down when the agent holds write, delete, payment or external send capability. Degrade to read-only with approvals only when the business impact of downtime is severe and the cause is understood.
• Small organisations: roles may be held by two or three people; the essentials are the inventory, per-agent credentials and a documented way to revoke them out of hours.
• Large organisations: centralise agent credentials behind a gateway or broker so one action revokes access across tools, and integrate agent alerts into the SOC.

Metrics

• Time from detection to agent stopped, and to all agent credentials revoked (target: [15] and [60] minutes for Critical).
• Percentage of production agents with a named owner and a unique identity.
• Percentage of agent tool calls captured in central logs.
• Percentage of agents with a tested kill switch in the last [6] months.
• Number of destructive actions executed without an enforced approval.

Common failure patterns

• Revoking the agent's runtime but not its OAuth refresh tokens or API keys, which keep working.
• Agents sharing a developer's credentials, so containment locks out a person and investigation cannot separate human from agent actions.
• Relying on instructions such as "do not change production" instead of permissions, as in the Replit and PocketOS incidents.
• Trace retention shorter than detection time, leaving no record of the injected input.
• Restoring the agent with the same over-broad credential and unpinned tools.

Framework mapping

Standards and references

• OWASP GenAI Security Project: Top 10 for Agentic Applications 2026
• OWASP: Top 10 for LLM Applications 2025
• MITRE ATLAS: AI Agent Tool Invocation
• NIST: SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management
• Model Context Protocol: Security Best Practices
• IETF: RFC 7009 OAuth 2.0 Token Revocation
• NIST: SP 800-53 Rev. 5

This template is provided for guidance and must be adapted to your organisation. Related NHI Mgmt Group resources: Agentic AI Security Guide · OWASP Top 10 for Agentic Applications Guide · Identity Incident Response Playbook · AI Agent Tool and Plugin Security Guide