Find Secrets in AI Agent Transcripts and Traces

Every API key, token and connection string an AI agent has seen is probably still sitting in a transcript, a trace or an export somewhere, long after the session ended. Coding agents and chat assistants write session histories to local files, sync conversations to the provider account, and send prompts, tool arguments and tool results to tracing and observability stores. A developer who pasted a database URL into a chat, or an agent that read a .env file, leaves a copy in each of those places. This guide is one task done well: finding secrets already at rest in agent transcripts, chat exports and LLM traces, triaging and rotating what you find, and cleaning the stores so the copy does not outlive the rotation. It is for security engineers, platform teams and developers. Keeping secrets out of the context window in the first place is covered in Runtime Secrets Injection for AI Agents Without Prompt Exposure, and the same task for pipelines in How to find secrets in CI logs and build artefacts.

Key takeaways

Treat any secret found in a transcript as exposed. It has been sent to a model provider, possibly to a tracing backend, and often to backups; rotation comes before clean-up.
Normalise before you scan. Transcripts are JSON, so multi-line keys and quoted values are escaped; decode string values first or scanners miss them.
Scan in place, not by copying. Gathering every transcript into one share to scan it creates a new, richer target.
Deleting the local file is not the end. Synced history, shared links, exports, trace stores and backups each need their own clean-up.
Turn content capture off by default. OpenTelemetry's GenAI conventions say instrumentations should not capture prompts and outputs by default; keep it that way in production unless you redact first.

Where secrets come to rest

Agent tools store sessions in different places and formats, and these change between releases. Use the table as a map of store types, then confirm the exact paths and retention settings in each tool's own documentation.

Store	Typical form	Who can read it	Why secrets end up there
Local session transcripts of CLI coding agents	JSON or JSONL files under a hidden folder in the user's home directory, per project or session	The user, local malware, endpoint backup	Tool results include file contents, command output and environment values the agent read
IDE assistant chat history	Editor workspace or global storage, often a local database or JSON	The user, extensions, sync services	Pasted snippets, error messages with connection strings
Agent memory and notes files	Markdown or JSON memory files, vector stores	The agent in later sessions, repository collaborators if committed	Agents record "useful facts", sometimes including keys
Provider-side conversation history	Synced to the user's or organisation's account	Account holders, workspace admins, provider under its terms	Everything typed or pasted into chat
Shared conversation links	Web pages generated from a conversation	Anyone with the link, possibly search engines if public	Users share a debugging session without reviewing it
Data exports	Archive of JSON or HTML conversation files	Whoever holds the download	Exports for backup, migration or records requests
LLM traces and observability stores	Span attributes or external content blobs	Observability users, often a broad engineering group	Prompts, outputs, tool arguments and results captured for debugging
Gateway and proxy logs	Request and response logs at an AI gateway	Platform and network teams	Full request bodies logged for cost or audit
Evaluation and fine-tuning datasets	Samples copied from traces	Data science teams, training pipelines	Production traces reused as test or training data

The OpenTelemetry GenAI semantic conventions define opt-in attributes that carry content, including gen_ai.system_instructions, gen_ai.input.messages, gen_ai.output.messages, gen_ai.tool.call.arguments, gen_ai.tool.call.result and gen_ai.memory.records. These conventions are marked as in development. If your traces follow them, these are the attributes to scan. OWASP's LLM02:2025 Sensitive Information Disclosure lists security credentials among the sensitive information an LLM application can expose; MITRE ATLAS tracks the same risk as AML.T0057 LLM Data Leakage.

Before you scan: permission and data handling

Get written scope: transcripts contain personal data, customer data and confidential code as well as secrets. Agree with privacy, legal and HR which stores you may scan, and whose.
Scan where the data lives: run the scanner on the endpoint, in the observability platform's own environment, or through its export API into memory, rather than copying everything to a central location.
Redact scanner output: reports should show the file, line or span, the rule and a partial value, never the full secret. Gitleaks has a --redact option for its logs and output.
Decide on live verification: some scanners can check whether a found credential is valid by calling the issuer's API; the TruffleHog README describes this as checking the secret against the API it appears to belong to. That gives better triage, but it uses the credential and sends it to the issuer; decide in advance which secret types you will verify this way.

Step by step: find secrets at rest

Build the target list: list the agent tools in use (from software inventory, extension inventory and proxy logs), the folders they write to, the observability projects that receive GenAI spans, gateway log locations, and any export or evaluation datasets. Include shared developer machines, remote development environments and CI runners where agents run.
Normalise the content: transcripts and spans store text inside JSON strings, so a private key appears as one long line with escaped newline sequences, and quotes are escaped. Parse each JSON record and write the decoded string values to a temporary, access-restricted stream for scanning, or pipe them straight into the scanner's standard input. Decode base64 tool results the same way.
Scan with more than one method: OWASP's Secrets Management Cheat Sheet (section 8.1) recommends using more than one detection tool and correlating results. Use a pattern-based scanner for known key formats plus entropy checks for generic secrets. Open-source options include Gitleaks and TruffleHog (listed alphabetically): Gitleaks has dir and stdin modes and report formats including JSON and SARIF; TruffleHog has a filesystem mode and can filter results by verification status. Confirm current options in each project's documentation.
Scan trace stores through their APIs: export spans for a time window, filtered to GenAI operations, and scan the content attributes listed above. If content is stored externally with references on the spans, as the OpenTelemetry conventions recommend for production, scan that external store too.
Use a baseline for repeat runs: after the first clean-up, record known and accepted findings so later scans only show new ones. Gitleaks supports a baseline file for this.
Prove the scanner works: seed a tagged dummy credential into a test session (pasted, read from a file by the agent, and returned by a tool) and confirm every scan finds all three copies. The canary approach is described in the runtime injection guide.

Triage and rotation

OWASP (section 9.2) treats revoking exposed keys as an immediate containment step and removing them from systems, including logs, as part of remediation. In transcripts the order matters: rotate first, then clean, because the copy has already left your control.

Finding	Priority (our recommendation)	Action
Cloud provider keys, CI/CD tokens, source control tokens with write access	Rotate within hours	Revoke, reissue through the secrets manager, review the credential's activity logs for use from unknown sources
Database connection strings and service passwords for production	Rotate within one working day	Rotate at the database, update the secrets manager, check access logs
Model provider API keys	Rotate within one working day	Revoke and reissue; check usage and spend for unexplained calls
Personal tokens of a developer	Rotate within one working day	User revokes and recreates; move the workflow to short-lived credentials
Test, sandbox or expired credentials	Confirm and record	Confirm scope and expiry, revoke anyway if cheap, clean the store
Private keys and signing keys	Treat as compromise	Revoke certificates or rotate key pairs; follow the key compromise process

Find the source as well as the copy: if the agent read a secret from a file, that file still holds it. Fix the source (move it to a secrets manager) or the next session leaks it again.
Check where else the secret lives: a key found in a transcript was often also in a repository, a shell history or a ticket. The Developer Secret Scanning Triage guide covers the wider process.
Record each finding: store, secret type, owner, rotation time and clean-up status, so you can report time to rotate.

Cleaning the stores

Local transcripts: delete affected session files, or the whole history for the period, after evidence is captured if an investigation is open. Set the tool's retention or clean-up period to the shortest that still meets developer needs, where the tool offers one.
Endpoint backups and sync: home directory backups and file sync services hold earlier copies; apply their deletion or version purge features.
Provider-side history: delete the conversations through the provider's controls, and use the organisation's admin and retention settings for workspace accounts. Retention and deletion behaviour differ between providers and plans, so confirm them in each provider's documentation. See the Zero Data Retention for AI Guide for the options.
Shared links: revoke every shared link created from an affected conversation. Public save-and-share features have exposed secrets at scale before: the JSONFormatter and CodeBeautify leak exposed cloud keys and tokens through saved, shareable pastes.
Exports and datasets: find and delete exports; remove affected samples from evaluation and fine-tuning datasets before they are reused.
Trace stores: delete or redact affected spans or content blobs using the platform's deletion features, and shorten content retention. Restrict who can view raw content.

Stop it recurring

Keep content capture off in production by default: the OpenTelemetry conventions state that instrumentations should not capture instructions, inputs or outputs by default and should offer an opt-in; for production they recommend storing content externally with references on spans, which allows separate access controls.
Redact in the pipeline: remove credential patterns in the telemetry collector before export so every backend benefits. See the LLM Observability Guide.
Keep secrets out of the agent's reach: agent ignore rules for credential files, no secrets in environment variables the agent can print, and tool-layer injection.
Schedule the scan: run the endpoint scan weekly through endpoint management and the trace scan daily (our recommendation), with alerts for new verified findings.

Measuring it

New secrets found per week in transcripts and traces: should fall as prevention takes hold.
Median time from finding to rotation: by priority class.
Share of findings with the source fixed: not just the copy rotated.
Observability projects capturing content without redaction: target zero in production.
Canary detection rate: all seeded test credentials found on every run.

Common failure patterns

Scanning raw JSON: escaped private keys and quoted values slip past pattern rules.
Cleaning before rotating: the transcript is deleted but the key, already sent to a provider and a trace store, still works.
Central copy for scanning: all transcripts copied to a shared drive, which then becomes the biggest secret store in the company.
Local only: laptops are scanned but synced history, shared links and observability stores are not.
Tool results ignored: teams scan what users typed, while most secrets arrive in what tools returned.
Debug capture left on: full content capture enabled for an incident and never turned off.

Framework mapping

Area	Reference	How it applies
Adversary behaviour	MITRE ATT&CK T1552.001 Credentials In Files; T1005 Data from Local System; T1213 Data from Information Repositories	Local transcripts and memory files are credential files; trace stores are information repositories
AI-specific behaviour	MITRE ATLAS AML.T0055 Unsecured Credentials; AML.T0057 LLM Data Leakage	Credentials stored insecurely around AI systems and leaked through LLM data
AI risk	OWASP LLM02:2025 Sensitive Information Disclosure	Security credentials as sensitive information in LLM applications
Controls	NIST SP 800-53 Rev 5 IA-5, SI-12, AU-9	Authenticator management, information retention, protection of logs that hold content
Outcomes	NIST CSF 2.0 PR.DS-01, DE.CM-09	Protect data at rest; monitor runtime environments and their data

Practitioner checklist

List every agent tool, its local storage, synced history, shared links, exports and trace destinations.
Agree scope with privacy and legal before scanning transcripts.
Decode JSON and base64 content before scanning; scan in place with redacted output.
Use at least two detection methods and a seeded canary to prove coverage.
Rotate found secrets by priority before cleaning the stores.
Fix the source file or configuration the agent read the secret from.
Clean local files, backups, provider history, shared links, exports, datasets and trace stores.
Keep GenAI content capture off in production unless redacted, and redact in the collector.
Schedule recurring scans with a baseline and alert on new findings.

How to Find Secrets in AI Agent Transcripts and Traces: scan, rotate and clean up stored sessions