Agent Observability vs Audit Logging for AI Agents

Most agent platforms send every trace, prompt and tool call into one observability backend, and then treat that backend as the audit trail. It works until someone asks for proof: a regulator, an auditor, a court or your own incident team. Traces are sampled, editable, full of sensitive content and often missing the human who delegated the task, so they fail exactly when they are needed as evidence. This guide compares agent observability and agent audit logging as two separate systems with different threat models, and shows how to split the pipeline, protect the evidentiary record, redact the debug record and set different retention for each. It is written for platform, security and compliance engineers who already run agent tracing and now need records that stand up to scrutiny.

Key takeaways

Two jobs, two systems. Observability answers "why did the agent behave like this?" for engineers; audit logging answers "who authorised this action and can we prove it?" for investigators and regulators.
Sampling and evidence do not mix. OpenTelemetry's own guidance says sampling may be inappropriate where regulations prohibit dropping data, so audit events must bypass every sampler.
Integrity is the dividing line. An audit record needs write-once storage, hash chaining or signing, trusted time and separate administrators; a trace store needs none of these and usually has none.
The delegator is the field most often missing. Traces name the agent and the model; evidence must also name the human or job that started the task and every hop in between.
Redact traces hard, keep audit records lean. Prompt and tool content belongs in a short-lived, restricted debug store; the audit record keeps identities, decisions and hashes, not raw text.

The short answer

Agent observability is engineering telemetry: spans, metrics, prompts, retrieved context and token counts, used to debug quality, latency, cost and failures. Agent audit logging is an evidentiary record: a complete, attributable, tamper-evident account of what each agent did, under whose authority and with what result. They may share instrumentation, but never a store, sampler, access model or retention schedule. The wider logging design for AI systems is in the Secure AI logging guide; this page goes deeper on the split for agents.

Property	Agent observability (debug traces)	Agent audit logging (evidence)
Primary reader	Developers, ML engineers, SRE	Security investigators, auditors, regulators, legal
Question answered	Why did the run fail, loop, cost too much or answer badly?	Who acted, for whom, under which policy, and what changed?
Completeness	Sampled; a representative subset is enough	Every in-scope event, no sampling, gaps are themselves recorded
Content	Prompts, completions, retrieved chunks, tool arguments (often opt-in)	Identities, decisions, targets, outcomes, hashes or references to content
Mutability	Editable, re-indexed, deleted by retention jobs and admins	Write-once, append-only, deletion only by policy after retention
Integrity proof	None expected	Hash chain or signatures, trusted timestamps, verification runs
Access	Broad engineering access	Small named group, separate from the people whose actions are recorded
Retention	Days to weeks	Set by law and policy: months to years

Two threat models

What threatens the observability store

Data exposure: traces carry what users typed, what the agent retrieved and what tools returned. The OpenTelemetry GenAI conventions mark the content attributes gen_ai.system_instructions, gen_ai.input.messages and gen_ai.output.messages as opt-in for this reason. Once enabled, the trace store holds secrets and personal data with broad read access.
Injection on read: trace content is attacker-influenced text. When an engineer or an AI assistant reads it back, it can carry prompt injection or forged log lines. OWASP's Logging Cheat Sheet describes injection through carriage return and line feed characters and recommends structured formats such as JSON.
Platform compromise: see Securing the LLM observability platform itself.

What threatens the audit record

Repudiation: a user or operator denies having started a task, approved an action or changed a policy. Without the delegator and approval in the record, nobody can prove otherwise.
Tampering and deletion: an insider, a compromised administrator or the agent's own credential edits or removes the record. OWASP lists blocking writes, damaging logs and recording the wrong identity as attacks on accountability.
Self-reporting: the agent's own narrative of what it did is not evidence. In the PocketOS Database Deletion 2026, the clearest account of the deletion came from the founder, the platform provider and the agent's own after-the-fact explanation; an authoritative record written by the platform outside the agent's control is what makes such a reconstruction provable.

Where the pipelines split

Split as early as possible, ideally where the event is emitted, so nothing in the debug path can drop, edit or delay the evidentiary copy.

Stage	Observability path	Audit path
Emitter	Agent SDK and framework instrumentation (OpenTelemetry spans)	Tool gateway, policy engine, identity provider and target systems, not the agent's own code
Transport	OTLP to a collector, batching, best effort	Authenticated, acknowledged delivery with a local durable buffer; TLS in transit
Processing	Sampling (head or tail), redaction processors, enrichment	Schema validation, canonical serialisation, hash chaining, timestamping; no sampling
Store	Trace backend or observability platform	Write-once object storage or a SIEM with immutable retention
Readers	Engineering teams, dashboards, evaluation jobs	Security, audit, legal; every read itself logged
Link between them	Shared trace ID, conversation ID and tool call ID, so an investigator can move from an audit event to the restricted debug context when it still exists

Emit audit events from the gateway and policy engine, not the agent: the agent process is the component most exposed to prompt injection and must not be able to suppress the record of its own actions. See the AI agent runtime reference architecture.

Sampling and completeness

OpenTelemetry describes head sampling as a decision made as early as possible, before the whole trace is known, and tail sampling as a decision made after considering all or most spans in the trace. Both deliberately discard data. The OpenTelemetry sampling page lists cases where sampling may be inappropriate, including where regulations prohibit dropping data and unsampled data cannot be routed to low-cost storage.

Never route audit events through a sampler. If both records share a collector, use a separate pipeline with no sampling processor for audit events, and test that a load spike does not drop them.
Do not trust the sampled flag from callers. W3C Trace Context treats the sampled bit in traceparent as a caller recommendation and warns that callers can forge the headers.
Record gaps as events. Exporter failures, buffer overflows and pipeline restarts belong in the audit stream, so gaps are provable.
Decide fail-open or fail-closed per action. For high-impact tools such as deletion, payments or permission changes, block the action if the audit event cannot be written. For read-only tools, fail open and alert.

Integrity: turning a log into evidence

US SEC Rule 17a-4(f)(2)(i) gives a useful model even outside financial services: an electronic recordkeeping system must either keep a complete time-stamped audit trail of modifications and deletions, including when and by whom they were made, or preserve records exclusively in non-rewriteable, non-erasable format.

Write-once storage: retention locks stop deletion and overwrite for a set period, and legal holds protect records with no end date until released.
Hash chaining: each record or batch carries the hash of the previous one, so a removed or edited record breaks the chain. Sign each batch with a key the logging pipeline holds and the agent platform cannot reach.
Trusted time: RFC 3161 time-stamp tokens bind a hash of the data to a signed time from a time-stamping authority, which gives evidence that the record existed before that time. Synchronise all emitters to a common time source; see the NTP Security Guide.
Separation of duties: agent platform administrators should not administer the audit store.
Verification: integrity controls that are never checked prove nothing. Run scheduled chain and signature verification and record the result.

The major cloud object stores each offer write-once controls that can hold the audit tier. Platforms are listed alphabetically, and the order is not a ranking.

Platform	Write-once feature	What the documentation describes
Amazon S3	S3 Object Lock	WORM retention per object version or bucket default, in governance or compliance mode, plus legal holds; requires S3 Versioning. In compliance mode no user, including the root user, can overwrite or delete a protected version or shorten its retention.
Google Cloud Storage	Bucket Lock and object holds	Retention policies that block deletion or replacement until objects reach the retention period; locking a policy is irreversible and it can then only be extended. Event-based holds protect individual objects.
Microsoft Azure Blob Storage	Immutable storage	Time-based retention policies and legal holds at container or version level; a policy must be locked to be in a compliant state for SEC 17a-4(f), and a locked policy can be extended but not shortened or deleted.

Each provider describes test modes before locking: governance mode for Amazon S3 Object Lock, unlocked policies for Google Cloud Storage and Microsoft Azure. Use them in a non-production bucket first, because a locked retention period cannot be undone.

Identity attribution and the missing delegator

An observability span usually knows the agent: the GenAI agent span conventions define gen_ai.agent.id, gen_ai.agent.name and gen_ai.agent.version, with gen_ai.conversation.id for the session. They do not, on their own, carry the human principal, the delegation chain or the policy decision. That is the information an investigator needs first.

Record both identities on every action: the agent identity from the AI agent registry and the user, service or scheduled job that started the task.
Carry the chain, not just the last hop: when tokens are exchanged under RFC 8693, the act claim identifies the current actor and nested act claims record prior actors, forming a history from the original subject through each delegation step. RFC 8693 says prior actors are informational only for access control, which is exactly why the audit record must capture them: the authorisation decision will not. See Delegation Chains and Attenuated Authority for AI Agents.
Log the decision, not just the call: policy name and version, permit or deny, scopes requested and granted, and the human approval with approver and time.

Redaction and sensitive data in traces

OpenTelemetry's guidance on sensitive data puts the responsibility on the operator: collect only data that serves an observability purpose, review what instrumentation libraries emit, and use Collector processors to remove it. The documented options are the attributes processor to remove or modify attributes, the redaction processor to delete attributes not on an allow list, the transform processor to rewrite values, and the filter processor to drop whole spans. Hashing small, predictable values such as numeric IDs may not anonymise them.

Data item	Debug trace store	Audit record
Prompt and completion text	Opt-in, redacted, short retention, restricted project	Never raw; hash or reference to the restricted store
Retrieved documents and tool results	Document IDs and sensitivity labels; text only for debugging windows	Document IDs, labels and the identity used for the permission check
Tool arguments	Redacted arguments	Target resource, operation and argument hash; explicit fields for high-impact parameters such as amount or record count
Credentials and tokens	Never	Never; token or key identifiers only
User identifiers	Pseudonymised or truncated	Real identifier, because attribution is the purpose

The asymmetry is deliberate: the audit record needs real identities but not conversation content; the debug record needs content briefly but not names.

Retention: split by purpose

For high-risk AI systems, EU AI Act Article 12 requires automatic recording of events over the system's lifetime, and Article 19 requires providers to keep the logs under their control for a period appropriate to the intended purpose of at least six months, unless Union or national law, in particular data protection law, provides otherwise. The AI Agent Audit Trails for Regulators Guide covers the full set of obligations and timing. One retention period cannot satisfy both stores.

Tier	Contents	Typical retention driver	Access
Debug content	Prompts, completions, retrieved text	Data minimisation; days to a few weeks	Named engineers, time-boxed, logged
Operational traces	Spans without content, metrics, costs	Engineering need; weeks to months	Engineering
Audit events	Identity, delegation, decision, action, outcome, hashes	Law, contract, internal policy; at least six months for EU high-risk systems, longer where sector rules apply	Security, audit, legal
Preserved evidence	Audit events and debug content for a specific incident	Legal hold until released	Incident lead and counsel

When an incident opens, place a hold on the matching debug content before it expires. For SaaS-side logs, see the SaaS Audit Log Retention and Licensing Guide.

Minimum audit event schema

A field-by-field template is in the AI Agent Audit Log Schema Template. The minimum set that separates an audit event from a trace span is short.

Field group	Fields	Why a trace is not enough
Event	Event ID, type, emitter, schema version, event time and receipt time	Spans have no stable event identity across re-indexing
Actor	Agent ID and version, credential reference	Agent ID is often optional on spans
Principal and chain	Delegating user or job, full delegation chain, tenant	Usually absent from spans
Authority	Policy ID and version, decision, scopes, approval record	Policy engines rarely emit spans into the agent trace
Action	Tool name, operation, target resource, argument hash	Arguments in spans are either raw or redacted away
Outcome	Result status, error class, records affected	Spans record errors, not business effect
Correlation	Trace ID, conversation ID, tool call ID	Shared with traces by design
Integrity	Previous-record hash, batch signature reference, timestamp token reference	No equivalent in trace stores

Worked scenario

A support agent is asked by a customer service lead to clean up duplicate customer records. It calls a CRM tool through the gateway and deletes 4,000 records, including live accounts. The next morning the customer service director asks what happened.

With only observability: successful runs were tail sampled at 10 percent, so the trace may not exist. If it does, the CRM call shows only the agent's service account, and nobody can show who started the task or whether a bulk delete was approved.
With a separate audit stream: the gateway's audit event names the agent, the lead who delegated the task, the policy that permitted delete without approval below 5,000 records, the target object, the record count and a hash of the arguments. The hash chain verifies, so the record has not been edited.
Joining the two: using the conversation ID, the team places a hold on the restricted debug content before it expires and finds the duplicate list came from a stale export.
Outcome: a provable policy threshold problem, not a rogue agent, fixed with an approval rule for bulk deletes. See the AI Agent Kill Switch and Containment Guide.

Decision guidance

Read-only prototypes: observability alone is acceptable, with content capture off.
Any agent that changes data, spends money, sends messages or touches personal data: a separate audit stream from the gateway and policy engine is required, even if the trace backend is excellent.
Regulated sectors or EU high-risk systems: write-once storage, hash chaining, trusted time and documented retention are the baseline, not an upgrade.
Small organisations: use a SIEM or log platform with immutable retention as the audit store and a hosted observability tool with content capture off; the split matters more than the tooling.
Large organisations: one audit schema across all agent platforms; debug stores stay with owning teams.

How to implement

Inventory emitters: every agent runtime, gateway, MCP proxy, policy engine and target system that produces agent events.
Define the audit event classes: start with tool calls that change state, authorisation decisions, human approvals, credential issuance and configuration changes.
Emit from outside the agent: add audit emission at the gateway and policy engine first, with the delegator and chain populated from the token.
Build the separate pipeline: durable buffer, authenticated transport, no sampler, schema validation and hash chaining before the write.
Stand up the write-once store in test mode: run governance or unlocked retention for a pilot period, confirm deletes fail and verification passes, then lock.
Turn content capture down in traces: switch prompt and output capture to opt-in per project, add redaction and filter processors, and shorten debug content retention.
Join the records: make sure trace ID, conversation ID and tool call ID appear in both, and add a legal hold step to the incident runbook.
Rehearse and roll back safely: prove who started a destructive agent action using only the audit store; keep the old trace pipeline unchanged until the audit stream passes.

Detection and measurement

Audit coverage: share of state-changing tool calls at the gateway with a matching audit event (target: 100 percent).
Delegator completeness: share of audit events with a non-empty principal and chain.
Chain verification: verification runs completed and failures found per month; any failure is an incident.
Pipeline gaps: minutes per month where the audit pipeline reported dropped or delayed events.
Content exposure: number of trace projects with prompt or output capture enabled, and readers with access to them.

Alert on audit events with a missing principal, on any delete or retention change attempted against the audit store, and on a gap between gateway call counts and audit event counts.

Framework mapping

Requirement	NIST SP 800-53 Rev. 5	NIST CSF 2.0	ISO/IEC 27001:2022 Annex A
Define and generate audit events	AU-2, AU-12	PR.PS-04	A.8.15 Logging
Record content and attribution	AU-3, AU-3(1), AU-10	PR.AA-05	A.8.15 Logging
Protect audit information	AU-9, AU-9(2), AU-9(3), AU-9(4)	PR.DS-01	A.8.15 Logging
Trusted time	AU-8	PR.PS-04	A.8.17 Clock synchronisation
Retention and disposal	AU-11, SI-12	PR.DS-01	A.5.33 Protection of records
Review and correlation	AU-6	DE.AE-03	A.8.16 Monitoring activities
Evidence for incidents	IR-4	RS.AN-07	A.5.28 Collection of evidence

Common failure patterns

The trace backend is the audit trail. It is sampled, editable by administrators and purged on a short schedule, so the run under investigation is often gone.
The agent writes its own audit log. A confused or injected agent can omit, alter or invent entries; the record must come from the gateway, policy engine and target systems.
Every action shows the service account. The delegator never reaches the downstream logs, so a destructive action cannot be tied to a person or job.
Full prompts in the audit store. Teams copy content into the long-retention tier "to be safe", creating years of personal data and secrets with legal-hold reach.
Integrity controls nobody verifies. Hashes and signatures are written but never checked, and the signing key sits on the same host as the agent platform.

Practitioner checklist

Write down which system is the audit record for each agent platform; if the answer is the trace backend, start the split.
Emit audit events from the tool gateway and policy engine, not from agent code.
Exclude audit events from every sampler and test them under load.
Populate agent ID, delegator and full delegation chain on every state-changing action.
Store audit events in write-once storage with hash chaining, signing and trusted timestamps, and verify them on a schedule.
Turn prompt and output capture to opt-in, add redaction processors and shorten debug content retention.
Keep shared trace, conversation and tool call IDs in both records, and add a legal hold step to the incident runbook.
Rehearse producing verified evidence of one agent action within an agreed time.

Agent Observability vs Audit Logging: Debug Traces Are Not Evidence