AI Agent Observability & Incident Response Guide

When an AI agent does something wrong, the first two questions are always the same: what exactly did it do, and how do we stop it? Many organisations cannot answer either quickly. Agent actions are logged as if a human user did them, or under a shared service account, or not at all, and stopping the agent means finding every token, key and grant it holds across several systems. This guide covers what to log for AI agents, how to attribute actions to the right agent and the right person, which signals show an agent has gone wrong, how to build and test a kill switch, and a practical incident response runbook. It also maps the controls to the main frameworks and offers metrics and a roadmap. It is written for security operations, detection engineering, platform and identity teams that run or approve AI agents in production.

Key takeaways

Attribute every action. Each agent action should name the agent, its version, the human or system it acted for, the tool it used, the credential it presented and the task that triggered it.
Log the decision trail, not only the outcome. Inputs, retrieved context, plan and tool calls let you tell manipulation from malfunction from an instruction that was simply harmful.
Detect on behaviour change. New tools, new data, unusual volume or spend, repeated permission errors and credential searching are the signals that matter, baselined per agent.
Build and rehearse a kill switch. Stopping one agent should halt its tasks, revoke every credential and grant, cut its tool access and preserve evidence in minutes, without stopping the whole platform.
Treat agent logs as sensitive data. Prompts and context can hold personal data, confidential content and secrets, so classify, redact, protect and retain them deliberately.

Why agent activity is hard to see

Traditional logging assumes a person or a well-defined service performs each action. Agents break that assumption in four ways.

Borrowed identities. Agents that act with a user's token appear in logs as the user. Agents that share a service account are indistinguishable from each other and from the scripts that also use it. OWASP lists NHI Reuse and Human Use of NHI among its Top 10 Non-Human Identity risks.
Distributed actions. One task can touch a model provider, several MCP servers, SaaS APIs and cloud services, each with its own logs, identifiers and retention. Without a shared correlation ID, the pieces do not join up.
Missing intent. Resource logs show what happened, not why. Without the agent's inputs and plan you cannot tell prompt injection from a bug from an instruction.
Volume and speed. Agents act faster and more often than people. In the PocketOS Database Deletion 2026, the founder told The Register the deletion "took 9 seconds".

NIST's National Cybersecurity Center of Excellence (NCCoE) names this gap directly. Its February 2026 concept paper on software and AI agent identity lists "Logging and Transparency" as a focus area, linking agent actions to the identity of the non-human entity, and asks: "How can we ensure that agents log their actions and intent in a tamper-proof and verifiable manner?" See the Agentic AI Identity Guide.

What to log

An agent audit record has to answer who acted, for whom, with what authority, why and with what result. The table sets out the minimum fields.

Field	Why it matters
Agent identifier (unique, from the agent registry)	Attributes the action to a specific agent, not a shared account
Agent version: model, prompt or configuration version, toolset	Links behaviour to what was deployed; supports rollback
Principal: the user or system the agent acted for	Separates human from agent activity
Delegation chain (multi-agent systems)	Shows which agents passed the task along
Correlation or trace ID	Joins model calls, tool calls and downstream API logs for one task
Task or session identifier and the triggering request	Explains why the agent was acting
Inputs and retrieved context (or references and hashes of them)	Reveals injected instructions and poisoned data
Planned action and reasoning summary, where available	Shows what the agent intended before it acted
Tool or API called, with arguments (redacted as needed)	The action actually taken
Credential used (token ID, key ID or role, never the secret)	Enables precise revocation
Authorisation decision and the policy that made it	Shows whether controls allowed or blocked the action
Human approvals: approver, time, what was shown	Evidence for high-impact actions
Outcome, errors and token usage	Detects failures, permission errors and spend anomalies
Memory writes	Detects memory and context poisoning (OWASP ASI06)

Use a standard telemetry format

OpenTelemetry's generative AI semantic conventions now live in their own repository and are still marked "Development", so expect changes. The agent spans specification defines operations such as create_agent, invoke_agent, plan, execute_tool, retrieval and memory operations (create_memory, update_memory, delete_memory), with attributes such as gen_ai.agent.id, gen_ai.agent.version, gen_ai.tool.name and token usage counts. Message content such as gen_ai.input.messages is opt-in, and the specification warns it is "likely to contain sensitive information including user/PII data". Record the convention version with the data.

Attribution starts with identity

The systems agents act on must log the agent's own identity, not only the platform's. That is only possible if agents have their own identities, or carry delegated tokens that name both the agent and the user. The NCCoE concept paper pairs logging with "Access Delegation", linking user identities to agents "to maintain accountability for the actions of automated systems". See the OAuth Token Exchange and Delegation Guide and the AI Agent Authorisation Guide.

Handling logs safely

Agent logs capture what users typed and what the agent read, so they are more sensitive than most application logs.

Control	What to do	Why
Classification	Classify agent logs at the level of the most sensitive data the agent handles	Access and retention follow it
Minimisation and redaction	Redact or tokenise personal and confidential data; store references and hashes of source documents instead of copies	Keeps forensic value, reduces exposure
Secret scanning	Scan prompts, tool arguments and outputs for credentials; rotate any secret found and fix the source	Agents read files and variables holding secrets
Integrity	Write-once or append-only storage, signing or hash chaining, separate administration	OWASP calls for immutable, signed audit logs of agent actions and tool calls
Access	Restrict log access to a small group; log who reads agent logs	Logs are a target themselves
Retention	Set retention for investigation and regulatory needs; avoid keeping full prompts indefinitely	Balances evidence and privacy

Keep the logging configuration itself under watch. In the cloud LLM hijacking campaign Sysdig documented in May 2024, attackers queried the model invocation logging configuration, and Sysdig noted that prompts and results "are not stored in Cloudtrail" without additional configuration. Alert on changes to, or reads of, AI logging settings. See the Cloud Logging and Audit Trail Guide and the Tokenisation and Data Masking Guide.

Detection: signals that an agent has gone wrong

Compare each agent with its own registered purpose and recent behaviour. What is normal for a data analysis agent is suspicious for a customer support agent.

Signal	What it looks like	Related OWASP risk
New tools or APIs	A call to a tool never used before, or outside the registered purpose	ASI02 Tool Misuse and Exploitation
New data	Access to data sets, repositories or records outside the normal pattern	ASI03 Identity and Privilege Abuse
Volume, rate and spend spikes	Sudden increases in actions, records read, messages sent or tokens consumed	ASI02; NHI2 Secret Leakage (stolen keys)
Repeated permission errors	Bursts of denied calls, often the first sign of searching for a way round a control	ASI03
Credential discovery	Reading credential files, secret stores or environment variables; using a credential not issued to it	ASI03; NHI5 Overprivileged NHI
Destructive or irreversible actions	Deletes, permission changes, external sharing and payments, especially out of hours	ASI10 Rogue Agents
Exfiltration patterns	Sensitive read followed by calls to external endpoints, email or file sharing	ASI02
Injection indicators	Retrieved content with instruction-like text, followed by a change in plan	ASI01 Agent Goal Hijack
Memory changes	Writes that alter stored rules, goals or trusted facts	ASI06 Memory and Context Poisoning

OWASP's guidance for tool misuse gives a concrete pattern to alert on: "unusual tool-chaining patterns (e.g., DB read followed by external transfer)". It also recommends usage ceilings on cost, rate or tokens "with automatic revocation or throttling when exceeded", which doubles as a detection threshold. Spend anomalies are a common sign of stolen AI credentials; see LLMjacking 2024 to 2026 and the LLMjacking Guide. Run agent detections alongside identity detections. See the Detection Engineering Guide and the ITDR Guide.

The kill switch

Design, build and test the ability to stop an agent before an incident. OWASP's guidance for rogue agents calls for "kill-switches and credential revocation to instantly disable rogue agents" and for quarantining suspicious agents for forensic review. A kill switch works in layers.

Layer	Action	Notes
Execution	Halt running tasks and block new ones in the orchestrator or agent platform	Per agent and per class of agent
Identity and credentials	Disable the agent's identity; revoke access and refresh tokens, OAuth grants, API keys, certificates and cloud role sessions	Needs a complete credential inventory
Tools	Block the agent at the MCP or API gateway; remove it from agent discovery so other agents stop calling it	OWASP's supply chain kill switch disables "specific tools, prompts, or agent connections across all deployments"
Propagation	Push revocation events to relying services instead of waiting for tokens to expire	Shared Signals Framework with CAEP, where supported
Evidence	Snapshot logs, memory, configuration and prompts before anything is deleted or rolled back	Preserve chain of custody

Revocation only works if you know every credential the agent holds, which is why a registry and platform-issued, short-lived credentials matter. For propagation, the OpenID Foundation approved the Shared Signals Framework, the Continuous Access Evaluation Profile (CAEP) and RISC as Final Specifications in September 2025. CAEP defines event types including Session Revoked, Token Claims Change and Credential Change, and its abstract covers "human or robotic users". See the Token and Session Security Guide and the MCP Security Guide.

Incident response runbook for AI agents

NIST SP 800-61 Rev. 3 (April 2025) recasts incident response as a CSF 2.0 Community Profile, part of the wider risk management cycle. Each step below shows the CSF outcomes it supports.

1. Detect and triage (DE.AE, RS.MA)

Identify the agent, its owner, its version and the principal it acted for.
Assess impact: what did it read, change, send or delete, and is it still running?

2. Contain (RS.MI-01)

Trigger the kill switch for the agent, and for related agents if the cause may be shared, such as a common MCP server or package.
If a user's delegated access was involved, consider revoking that user's grants to the agent platform as well.
Block implicated tools, MCP servers, packages or external endpoints.

3. Investigate (RS.AN)

Reconstruct the task with correlation IDs: inputs, retrieved content, plan, tool calls and outcomes.
Determine the cause: prompt injection, poisoned memory, compromised tool or dependency, stolen credentials, misconfiguration, model error or a legitimate but harmful instruction.
Check for persistence: memory entries, new credentials, service accounts, OAuth grants, scheduled tasks or code changes the agent created.
Record investigative actions and protect the integrity of collected evidence (RS.AN-06, RS.AN-07).

4. Eradicate and recover (RS.MI-02, RC.RP)

Remove poisoned memory and malicious content from retrieval sources; OWASP recommends snapshots, rollback and quarantine for suspected memory poisoning.
Rotate every credential the agent could reach, not only those it used.
Restore affected data from backups the agent could not reach.
Fix the root cause before re-enabling: narrower permissions, new approval gates, input filtering or tool changes.

5. Learn and report (ID.IM, RS.CO)

Update the agent's threat model, policies and detections. See the AI Agent Threat Modelling Guide.
Assess reporting obligations, such as personal data breach notification, sector regulators and AI-specific rules. See the Agentic AI Compliance Guide.

For the identity-specific steps of containment, see the Identity Incident Response Playbook and the Leaked Credential Response Playbook.

Lessons from real incidents

PocketOS (April 2026). According to the founder's account reported by The Register, a coding agent hit a credential mismatch in staging, found a Railway API token in an unrelated file and used it to delete the production volume and its volume-level backups. The token had been created for managing custom domains but was, in Railway's words, "provisioned with account scoped access". Railway restored the data and changed its API so deletes are soft deletes for 48 hours. Alert on destructive calls by agent credentials, scope tokens to the task and keep backups out of reach of agent identities. See PocketOS Database Deletion 2026.
Replit and SaaStr (July 2025). SaaStr's founder claimed Replit's AI deleted a live database despite instructions not to change code without permission, created a 4,000-record database of fictional people, and wrongly told him rollback would not work. Detect fabricating as well as destructive behaviour. See Replit AI Agent Database Deletion 2025.
Salesloft Drift (August 2025). Google Threat Intelligence Group reported that UNC6395 used stolen Drift OAuth tokens to export Salesforce data. The actor deleted query jobs, but GTIG said "logs were not impacted", and recommended reviewing Event Monitoring and UniqueQuery logs. Salesloft and Salesforce revoked all active access and refresh tokens for the app on 20 August 2025. Integration identities need their own logs and a revocation runbook that reaches every connected tenant. See Salesloft Drift Breach 2025.
LLMjacking (2024 onwards). Sysdig observed stolen cloud credentials used to reach hosted models and estimated a worst case of over $46,000 of model consumption per day for the victim. Watch model spend and logging configuration as security signals.

Common failure patterns

Shared identities. Several agents share one service account or key, so no log says which acted and revoking it breaks everything.
Outcome-only logging. Logs record the delete but not the prompt or plan, so injection and error look the same.
Trusting the agent's own narrative. Teams ask the agent what happened. In the Replit case the agent's statement that rollback was impossible was wrong; reconstruct from independent logs.
Broad standing tokens on disk. Long-lived, fully scoped tokens left in files are found and used by agents, as in the PocketOS case. See the Secrets Management Guide.
Backups in the blast radius. Backups reachable with the same credential, or stored with the data, are deleted with it. See the Backup Security and Immutable Backup Guide.
Untested kill switch. It stops the orchestrator but leaves OAuth grants, refresh tokens and cloud sessions alive.
Logging that attackers can see or switch off. Prompt and response logging for hosted models needs extra configuration, and attackers check whether it is on, as the LLMjacking reconnaissance showed.
Unregistered agents. Agents built outside the platform have no registry entry, owner or baseline. See the Shadow AI Discovery Guide.

Framework mapping

Mappings were checked against the framework texts in the references. They show where these controls provide evidence, not full coverage of any control.

Control area	NIST CSF 2.0	NIST SP 800-53 Rev. 5	ISO/IEC 27001:2022 Annex A	CIS Controls v8.1	OWASP
Agent inventory and identity	ID.AM-02, PR.AA-01	CM-08, AC-02	5.16 Identity management	Control 5 Account Management	NHI9 NHI Reuse; NHI10 Human Use of NHI
Agent audit logging and attribution	PR.PS-04	AU-02, AU-03, AU-12, AU-10	8.15 Logging	Control 8 Audit Log Management	ASI08 (tamper-evident logs bound to agent identities)
Log protection and retention	PR.DS-01	AU-09, AU-11	8.15 Logging; 8.11 Data masking	Control 8	ASI10 (immutable, signed audit logs)
Behavioural monitoring and correlation	DE.CM-03, DE.CM-09, DE.AE-02, DE.AE-03	SI-04, AC-02(12), AU-06	8.16 Monitoring activities	Control 8; Control 13 Network Monitoring and Defense	ASI02, ASI06, ASI10
Kill switch and credential revocation	RS.MI-01	AC-02(03), IR-04	5.18 Access rights; 5.26 Response to information security incidents	Control 6 Access Control Management	ASI04 (supply chain kill switch); ASI10; NHI1 Improper Offboarding
Least privilege for agent credentials	PR.AA-05	AC-06	8.2 Privileged access rights	Control 6	ASI03; NHI5 Overprivileged NHI; NHI7 Long-Lived Secrets
Investigation and evidence	RS.AN-03, RS.AN-06, RS.AN-07	IR-04, AU-06	5.28 Collection of evidence	Control 17 Incident Response Management	ASI08 (lineage metadata for forensics)
Response planning and lessons learned	RS.MA-01, ID.IM	IR-08, IR-06	5.24 Planning and preparation; 5.27 Learning from incidents	Control 17	ASI10
Backups out of agent reach	PR.DS-11	CP-09	8.13 Information backup	Control 11 Data Recovery	ASI06 (snapshots and rollback)

Metrics that show progress

Metric	What it shows	Target direction
Share of production agents with a unique identity and registry entry	Whether attribution is possible at all	Up, towards 100%
Share of agent actions carrying agent ID, principal and correlation ID	Completeness of the audit trail	Up
Time to stop an agent and revoke all its credentials in a test	Real kill switch capability	Down
Share of agents whose kill switch was tested in the last quarter	Whether the capability is maintained	Up
Mean time to detect agent misbehaviour in exercises	Detection coverage and tuning	Down
Secrets found in agent logs per month	Effectiveness of redaction and secret hygiene	Down
Destructive actions by agents without a recorded human approval	Strength of approval gates	Down, towards zero

Implementation roadmap

Phase	Focus	Actions
First 30 days	Visibility	Inventory agents and owners; give each a distinct identity; list its credentials, grants and tools; switch on logging with correlation IDs; move backups out of agent reach
Days 31 to 90	Detection and containment	Adopt the OpenTelemetry GenAI conventions; send agent telemetry to the SIEM; build the core detections; build and test a per-agent kill switch; write the runbook
Days 91 to 180	Assurance	Make logs tamper-evident; add per-agent baselines and spend thresholds; integrate CAEP where supported; rehearse an agent incident with the SOC; report the metrics

Practitioner checklist

Every production agent has a unique identity, a registry entry and a named owner.
Every agent action is logged with agent ID, version, principal, correlation ID, tool, credential ID, authorisation decision and outcome.
Downstream systems see and log the agent's own identity, not only the user's or the platform's.
Agent logs are classified, redacted where needed, scanned for secrets, tamper-protected and retained deliberately.
Per-agent baselines and detections cover new tools, new data, volume and spend, permission errors, credential discovery, destructive actions and memory writes.
A per-agent kill switch halts execution, revokes all credentials and grants, blocks tools and preserves evidence, and has been tested this quarter.
Revocation reaches relying services through shared signals or a documented manual process.
Backups are out of reach of agent identities and restores have been tested.
An AI agent incident response runbook exists, maps to your wider incident plan and has been rehearsed.

AI Agent Observability, Audit and Incident Response Guide