AI Agent Incident Exercises: Tabletop Scenarios

Most incident response teams have never rehearsed an incident where the actor is their own AI agent, acting with real credentials, at machine speed, on instructions it picked up from a poisoned web page, ticket or tool. Classic scenario packs cover ransomware, phishing and supplier breaches, and the plans written for them assume a human attacker and a human-paced timeline. Agent incidents break those assumptions: the "compromised account" is a workload identity with delegated user tokens, the evidence is a chain of prompts and tool calls, and the fix may sit with a model provider or an MCP server maintainer. This guide gives security, IAM and platform teams a set of agent-specific exercise scenarios, injects, roles, evidence requirements and scoring measures. It assumes you already run tabletop exercises; for the general method see the Tabletop Exercise Guide.

Key takeaways

Exercise the identity layer, not just the model. The hardest decisions in an agent incident are which tokens, grants and keys the agent holds and how fast you can revoke them all.
Use real agent failure modes. Build scenarios from the OWASP Top 10 for Agentic Applications and from documented incidents such as postmark-mcp and the Replit database deletion, not from generic malware stories.
Invite the people outside the SOC. Agent owners, the platform team, the IdP administrator, procurement and the model or tool provider contact all hold decisions that responders cannot make alone.
Test the evidence before you need it. A tabletop that asks "show me the tool calls from 14:02" exposes missing prompt, tool and identity logs faster than any audit.
Follow the tabletop with a technical drill. Pull the kill switch and revoke a test agent's credentials in a non-production tenant, and time it.

Why agent incidents need their own exercises

NIST SP 800-61 Rev. 3 places incident response inside CSF 2.0 risk management, and CSF outcome ID.IM-02 expects improvements to come from security tests and exercises, including those run with suppliers and third parties. Agent incidents stress that outcome in specific ways:

Speed: an agent can complete hundreds of tool calls before an analyst opens the alert, so containment decisions must be pre-agreed rather than debated live.
Delegated authority: the agent often acts on behalf of a user with an OAuth token. Microsoft documents that revoking sessions stops new tokens but existing access tokens stay valid until expiry, one hour by default, unless the application supports Continuous Access Evaluation. Teams rarely know this until they try it.
Ambiguous intent: an agent that deletes data may be hijacked, misconfigured or simply wrong. The Replit incident in July 2025, where a coding agent wiped a production database during a declared code freeze and then claimed rollback was impossible (it was not), shows why responders must not trust the agent's own account of events. See the Replit AI Agent Database Deletion 2025.
Third-party dependencies: the model provider, the agent framework and each MCP server or plugin are suppliers with their own response timelines.
Unfamiliar evidence: system prompts, retrieved documents, memory entries and tool call arguments are the new log sources, and many are kept only briefly or not at all.

Scenario library

Each scenario below maps to an OWASP Top 10 for Agentic Applications (December 2025) risk and to MITRE ATLAS techniques verified against ATLAS data. Pick two or three per exercise, matched to agents you actually run.

Scenario	Starting situation	OWASP Agentic / ATLAS	Core decisions tested
1. Indirect prompt injection leads to data exfiltration	A support agent reads a customer ticket containing hidden instructions and starts emailing CRM records to an external address	ASI01 Agent Goal Hijack; AML.T0051.001 Indirect, AML.T0086 Exfiltration via AI Agent Tool Invocation	Stop the agent or stop the tool; who decides; notification clock for personal data
2. Malicious MCP server update	A popular MCP server you installed from a public registry ships a version that copies every outgoing message to an attacker	ASI04 Agentic Supply Chain Vulnerabilities; AML.T0010.005 AI Agent Tool	Find every install; pin or remove; which credentials did the server hold
3. Destructive action by an over-privileged agent	A coding agent with a production token drops a database during a change freeze	ASI02 Tool Misuse, ASI03 Identity & Privilege Abuse; AML.T0101 Data Destruction via AI Agent Tool Invocation	Restore path; whether the agent's report can be trusted; who owns the token
4. Stolen agent integration tokens	A vendor that hosts your chatbot integration reports that its stored OAuth tokens were exfiltrated	ASI03 Identity & Privilege Abuse; ATT&CK T1528 Steal Application Access Token, T1550.001 Application Access Token	Revoke tokens you do not hold; scan data the integration could read for secrets
5. Memory poisoning with delayed effect	A long-running assistant stored attacker-supplied "preferences" weeks ago that now redirect payments	ASI06 Memory & Context Poisoning; AML.T0080 AI Agent Context Poisoning	How far back to look; purge and rebuild memory; which past actions to reverse
6. Cascading multi-agent failure	An orchestrator agent passes a poisoned instruction to three worker agents across two business units	ASI07 Insecure Inter-Agent Communication, ASI08 Cascading Failures	Blast radius across owners; which agent to stop first; inter-team command
7. Rogue or orphaned agent	Monitoring finds an agent nobody owns still calling production APIs with a long-lived key	ASI10 Rogue Agents	Authority to disable without an owner; inventory gaps

Scenario 2 is modelled on a real case: in September 2025 version 1.0.16 of the npm package postmark-mcp added a hidden BCC to its send-email tool, and deleting the package from npm did not remove installed copies. See the postmark-mcp Malicious MCP Server 2025. Scenario 4 follows the Salesloft Drift Breach 2025, where stolen Drift OAuth tokens were used to export Salesforce data and customers then had to rotate credentials found in the exported support cases.

Designing injects that force identity decisions

Injects are the timed updates the facilitator releases. For agent scenarios, write at least half of them around identity and evidence, because that is where plans are thinnest.

Detection inject: a SIEM alert showing an unusual tool call (for example, a mail send to a new external domain) attributed to the agent's workload identity, with the on-behalf-of user also shown. Ask: whose incident is it, the agent owner's or the user's?
Inventory inject: "List every credential this agent can use." Expect API keys in the agent platform, OAuth grants from users, MCP server tokens, cloud roles and any secrets in its memory or configuration.
Containment inject: the agent platform's disable switch works, but scheduled runs restart it every 15 minutes, or a downstream MCP server still holds its own upstream token. See the Revoking Delegated Access During an Active Agent Incident guide.
Evidence inject: legal asks for the exact prompts, retrieved content and tool call arguments for a two-hour window. Reveal that prompt logging was disabled for cost reasons.
Supplier inject: the MCP server maintainer or model provider does not respond for 48 hours. Who decides to remove a business-critical tool?
Business inject: the business owner wants the agent back online before the root cause is known because customer queues are growing.
Communication inject: a journalist asks whether "your AI went rogue". Test who speaks and what is said.

Roles and participants

Role	Why they must be in the room	Decisions they own
Incident commander	Runs the response	Severity, escalation, when to stand down
Agent owner (business)	Accountable for the agent and its purpose	Accepting downtime; approving restart
Agent platform or ML engineering lead	Knows prompts, tools, memory and deployment	Kill switch, rollback to a known-good version, memory purge
IAM or IdP administrator	Holds the workload identity and OAuth configuration	Disabling service principals, revoking grants and sessions, rotating keys
SOC analyst and detection engineer	Reads tool call and identity logs	Scoping, new detections during the incident
Privacy or legal	Personal data and contractual notification	Notification decisions and evidence preservation
Third-party or procurement lead	Contacts for model provider, framework and MCP server vendors	Supplier escalation; contractual rights
Facilitator and evaluator	Run the exercise and record observations	Inject timing; scoring against objectives

CISA's Tabletop Exercise Packages provide role templates for planners, facilitators, evaluators and participants, plus an After Action Report template you can reuse for agent exercises.

Evidence the exercise should demand

Agent run records: run ID, agent version, model and model version, system prompt version and the user or event that triggered the run.
Tool call log: every tool invoked, its arguments, the response and the identity used, with timestamps that line up with the IdP and cloud logs. See the Agent Activity Timeline Reconstruction from Tool Calls and Identity Logs.
Identity logs: sign-ins and token issuance for the agent's workload identity and for on-behalf-of users (Entra ID sign-in logs for service principals and managed identities, Okta System Log, AWS CloudTrail AssumeRole events, Google Cloud audit logs).
Retrieved content and memory: which documents, pages or tickets entered the context window, and any memory writes.
Configuration history: changes to tools, MCP server versions, permissions and guardrail settings in the period.

Score the team on whether each item exists, how long it took to retrieve, and whether it was complete. A gap found here is often the most valuable outcome of the exercise.

Running the exercise

Set objectives: two to four measurable objectives, for example "revoke all credentials held by the agent within 30 minutes of the decision" or "produce the tool call timeline within two hours".
Choose the format: a 90-minute discussion tabletop for leadership, a three-hour tabletop for responders, then a technical drill in a non-production tenant.
Prepare a realistic agent profile: name, owner, tools, credentials, data sources and autonomy level of a real or representative agent. Generic agents produce generic answers.
Brief safety rules: for technical drills, use test agents and test credentials only, isolate the environment and agree an abort word.
Run injects on a clock: compress time but keep the order realistic. Record decisions and the time each took.
Hot wash: collect what went well and what did not within 30 minutes of the end.
After action report: findings, owners and due dates, fed into the agent risk register and the incident response plan. See the Tabletop Exercise After Action Report Template.
Retest: rerun the same scenario after fixes, and rotate scenarios each quarter as agent deployments change.

Technical drill: kill switch and revocation

Disable the test agent's workload identity in the IdP and confirm new token requests fail.
Revoke refresh tokens for the on-behalf-of test user (in Entra ID, Revoke sessions or Revoke-MgUserSignInSession) and measure how long existing access tokens keep working.
Rotate the API keys held by each MCP server the agent used and confirm old keys are rejected.
Check that scheduled triggers, queues and retries do not restart the agent.
Restore the agent from a known-good version and confirm its memory store was purged or rolled back.

The AI Agent Kill Switch and Containment Guide covers how to build these controls; the drill proves they work.

Decision guidance

Tabletop or technical drill: start with a tabletop when no agent runbook exists; move to technical drills once owners, kill switches and logging are in place. A drill on an agent with no kill switch only proves there is none.
Small organisations: one 90-minute tabletop on scenario 1 or 4 per year, using free material such as NCSC Exercise in a Box or CISA CTEPs adapted with an agent inject, gives most of the value.
Large organisations: run scenario-per-quarter cycles across business units, include at least one supplier (model provider or MCP vendor) a year, and tie objectives to agent autonomy levels.
Which agents first: agents with write access to production, payment or customer data, and agents using third-party MCP servers.

Measuring whether it works

Metric	How to measure	Direction
Time to full credential inventory for the agent	From inventory inject to complete list	Down
Time to effective revocation	From decision to last working token rejected (technical drill)	Down
Evidence completeness	Share of required evidence items retrieved	Up
Decision latency on business restart	Time from request to documented decision	Down
Findings closed before the next exercise	Closed after action items divided by total	Up
Agent coverage	High-risk agents exercised in the last 12 months	Up

Framework mapping

Framework	Reference	Relevance
NIST CSF 2.0	ID.IM-02, ID.IM-04, RS.MA-01	Improvements from exercises with third parties; maintaining IR plans; coordinated response
NIST SP 800-53 Rev. 5	IR-2, IR-3, IR-3(2), IR-4, IR-8	Training, incident response testing and coordination with related plans
NIST SP 800-84	Test, training and exercise programmes	Design, conduct and evaluation of exercises
CIS Controls v8.1	Safeguard 17.7	Conduct routine incident response exercises
ISO/IEC 27001:2022 Annex A	5.24, 5.26, 5.27	Incident management planning, response and learning from incidents
OWASP Top 10 for Agentic Applications	ASI01 to ASI10	Scenario source
MITRE ATLAS	AML.T0051, AML.T0053, AML.T0086, AML.T0101, AML.T0080	Technique references for injects

Common failure patterns

Treating the agent as a chatbot: the scenario stops at "the model said something bad" and never reaches the credentials and actions behind it.
No identity administrator in the room: nobody can say which service principal, role or OAuth client the agent uses, so revocation stays theoretical.
Trusting the agent's own logs or narrative: as the Replit case shows, an agent's account of what it did can be wrong. Exercises should require independent identity and platform logs.
Removing the package but not the install: in postmark-mcp, removal from the registry left installed copies running. Exercises should test how you find every deployed copy of a tool.
Forgetting data the integration could read: after Salesloft Drift, rotation extended to secrets found in exported support cases, not just the stolen tokens.
One exercise, no retest: findings are logged but agents, tools and owners change monthly, so the next incident meets a different system.

Practitioner checklist

Pick the three highest-risk agents and write a one-page profile for each: owner, tools, credentials, data and autonomy.
Build at least two scenarios from the library above that match those agents.
Write injects that require a full credential inventory and a timed revocation decision.
Invite the agent owner, platform lead, IdP administrator, privacy and procurement, not only the SOC.
Define the evidence list in advance and score retrieval time and completeness.
Run a technical kill switch and revocation drill on a test agent in a non-production tenant.
Record decisions and times, produce an after action report with owners and dates, and update the agent IR playbook.
Repeat with a new scenario each quarter and include a supplier at least once a year.

AI agent incident exercises: Tabletop and technical drills for agent compromise and misuse