AI Agent Observability & Incident Response Guide

When an AI agent does something wrong, the first two questions are always the same: what exactly did it do, and how do we stop it? Many organisations cannot answer either quickly. Agent actions are logged as if a human user did them, or under a shared service account, or not at all. Stopping the agent means finding every token, key and grant it holds across several systems. This guide covers what to log for AI agents, how to attribute actions to the right agent and the right person, which signals indicate an agent has gone wrong, and a practical incident response runbook, including the kill switch you should build before you need it.

Key takeaways

Every agent action should be attributable to the agent, the human or system it acted for, the tool it used, and the task that triggered it.
Log the decision trail (inputs, plan and tool calls) as well as the outcome, so you can tell whether an agent was manipulated, malfunctioned or did what it was told.
Detect on behaviour change: new tools, new data, unusual volumes, repeated permission errors and attempts to find other credentials.
Build a tested kill switch: the ability to stop an agent and revoke all of its credentials and grants in minutes, without stopping everything else.
Treat logs of prompts and agent context as sensitive data: they may contain personal data, confidential content and occasionally secrets.

Why agent activity is hard to see

Borrowed identities. Agents that act with a user's token appear in logs as the user. Agents that share a service account are indistinguishable from each other.
Distributed actions. One task can touch a model provider, several MCP servers, SaaS APIs and cloud services, each with its own logs and identifiers.
Missing intent. Resource logs show what happened, not why. Without the agent's inputs and plan, you cannot tell prompt injection from a bug from an instruction.
Volume and speed. Agents act faster and more often than people, so both noise and damage accumulate quickly.

What to log

Field	Why it matters
Agent identifier (unique, from the registry)	Attributes the action to a specific agent, not a shared account
Agent version: model, prompt or configuration version, toolset	Links behaviour to what was deployed; supports rollback
Principal: the user or system the agent acted for	Accountability and access checks; separates human from agent activity
Delegation chain (for multi-agent systems)	Shows which agents passed the task along
Correlation or trace ID	Joins model calls, tool calls and downstream API logs for one task
Task or session identifier and the triggering request	Explains why the agent was acting
Inputs and retrieved context (or references and hashes of them)	Reveals injected instructions and poisoned data
Planned action and reasoning summary, where available	Shows what the agent intended before it acted
Tool or API called, with arguments (redacted as needed)	The actual action taken
Credential used (token ID, key ID or role, never the secret)	Enables precise revocation
Authorisation decision and policy that made it	Shows whether controls allowed or blocked the action
Human approvals: approver, time, what was shown	Evidence for high-impact actions
Outcome and errors	Detects failures and repeated permission errors
Memory writes	Detects memory poisoning (OWASP ASI06)

Where possible, use standard formats so agent telemetry can be joined with the rest of your observability data. OpenTelemetry has published semantic conventions for generative AI, still evolving at the time of writing, which cover model and agent spans and are supported by a growing number of frameworks.

Also make sure the systems agents act on log the agent's own identity. That is only possible if agents have their own identities or carry delegated tokens that name them, which is why attribution starts with the identity model.

Handling logs safely

Classify agent logs according to the most sensitive data the agent handles.
Redact or tokenise personal and confidential data where full content is not needed; keep references to source documents instead of copies.
Scan logs for secrets; if a secret appears, rotate it and fix the source.
Protect logs against tampering (write-once storage, integrity controls), as OWASP recommends for agent audit trails.
Set retention to meet investigation and regulatory needs without keeping prompts indefinitely.

Detection: signals that an agent has gone wrong

New tools or APIs: the agent calls something it has never used, or that is outside its registered purpose.
New data: access to data sets, repositories or records outside its normal pattern, especially sensitive ones.
Volume and rate spikes: sudden increases in actions, records read, messages sent or tokens consumed.
Repeated permission errors: often the first sign an agent is trying to do something it should not, or is searching for a way round a control.
Credential discovery: reading credential files, secret stores or environment variables, or using a credential not issued to it.
Destructive or irreversible actions: deletes, permission changes, external sharing and payments, particularly outside business hours.
Exfiltration patterns: reading sensitive data followed by calls to external endpoints, email or file-sharing tools.
Injection indicators: retrieved content containing instruction-like text, followed by a change in the agent's plan.
Memory changes: writes that alter the agent's stored rules or goals.
Spend anomalies: unexpected model usage costs, a common sign of stolen AI credentials being used for LLM hijacking (see the AI LLM hijack breach).

Baseline each agent individually. What is normal for a data analysis agent is suspicious for a customer support agent.

The kill switch

Design, build and test the ability to stop an agent before an incident. A kill switch should:

Stop execution: halt running tasks and prevent new ones, in the orchestrator or agent platform.
Revoke credentials: disable the agent's identity and revoke its access tokens, refresh tokens, OAuth grants, API keys, certificates and cloud role sessions.
Cut tool access: block the agent at the MCP or API gateway, and remove it from agent discovery so other agents stop calling it.
Preserve evidence: snapshot logs, memory and configuration before anything is deleted.
Work per agent: stop one agent without stopping the whole platform, and also support stopping a class of agents (for example, everything using a compromised MCP server).

Revocation only works if you know every credential the agent holds, which is another reason for a registry and for platform-issued, short-lived credentials. Where supported, use shared signals, such as the OpenID Shared Signals Framework with the Continuous Access Evaluation Profile (CAEP), to push revocation events to relying services rather than waiting for tokens to expire.

Incident response runbook for AI agents

1. Detect and triage

Identify the agent, its owner, its version and the principal it acted for.
Assess impact: what did it read, change, send or delete, and is it still running?

2. Contain

Trigger the kill switch for the agent (and related agents if the cause may be shared).
If a user's delegated access was involved, consider revoking that user's grants to the agent platform as well.
Block implicated tools, MCP servers, packages or external endpoints.

3. Investigate

Reconstruct the task: inputs, retrieved content, plan, tool calls and outcomes, using correlation IDs.
Determine the cause: prompt injection, poisoned memory, compromised tool or dependency, stolen credentials, misconfiguration, model error or a legitimate but harmful instruction.
Check for persistence: memory entries, new credentials, new service accounts, OAuth grants, scheduled tasks or code changes the agent created.

4. Eradicate and recover

Remove poisoned memory and malicious content from retrieval sources.
Rotate every credential the agent could reach, not only those it used.
Restore affected data from backups that the agent could not delete.
Fix the root cause before re-enabling: narrower permissions, new approval gates, input filtering or tool changes.

5. Learn

Update the agent's threat model, policies and detections. See the Threat Modelling AI Agents guide.
Consider reporting obligations: personal data breaches, sector regulators and, for some AI systems in the EU, serious incident reporting under the AI Act. See the Agentic AI Compliance Guide.

Lessons from real incidents

In the PocketOS incident, a coding agent deleted a production volume and its backups; recovery depended on the hosting provider restoring data. Keep backups out of reach of the identities agents use.
In the Replit incident, an AI assistant deleted a live database and generated fake records, showing the need to detect both destructive and fabricating behaviour.
In OAuth token theft cases such as Salesloft Drift, containment depended on revoking tokens across many connected customer environments, so revocation runbooks for integrations are essential.

Practitioner checklist

Every agent action is logged with agent ID, version, principal, correlation ID, tool, credential ID, decision and outcome.
Downstream systems see and log the agent's own identity.
Agent logs are classified, redacted where needed, tamper-protected and retained appropriately.
Per-agent baselines and detections cover new tools, new data, volume, permission errors, credential discovery, destructive actions and spend.
A kill switch per agent stops execution, revokes all credentials and grants, blocks tools and preserves evidence, and has been tested.
Backups are out of reach of agent identities.
An AI agent incident response runbook exists and has been rehearsed.

Standards and references

OWASP Top 10 for Agentic Applications for 2026: ASI06, ASI08, ASI10
OpenTelemetry semantic conventions for generative AI
OpenID Shared Signals Working Group (SSF, CAEP, RISC)
NIST SP 800-61 Rev. 3: Incident Response Recommendations
NIST NCCoE: Software and AI Agent Identity and Authorization (auditing and non-repudiation)

AI Agent Observability, Audit and Incident Response Guide