Sensitive Data in Logs Guide: Finding and Preventing PII, Tokens and Secrets in Application Logs

Application logs quietly become one of the largest unprotected copies of passwords, tokens and personal data in an organisation, because they are written by every service, copied to many systems and read by many people. A single debug statement that prints a request object can put bearer tokens and customer records into a log platform, its backups, a vendor's support bundle and an engineer's laptop within minutes. This guide covers how sensitive data gets into application logs, what must never be logged, how to stop it in code and in the pipeline, how to find what is already there, and what to do when you do. It is written for developers, platform engineers and security teams who own logging. Retention, legal basis and detection content are covered in other guides; this one is about leakage.

Key takeaways

• Decide by data class, not by developer judgement. Credentials and payment data are never logged; session tokens and identifiers are logged only hashed or masked, as OWASP ASVS 5.0 requirement V16.2.5 describes.
• Allow-list fields at the source. Structured logging with named, approved fields stops most leaks; logging whole objects, requests or exceptions causes most of them.
• Pipeline redaction is a safety net, not the control. Collector and agent redaction catches known patterns and unknown keys, but regular expressions miss secrets they were not written for.
• Assume it is already there. Scan existing log indexes, archives and support bundles for token formats and personal data before declaring the problem fixed.
• A secret in a log is a leaked secret. Rotate it first, then purge every copy, then fix the code path.

How sensitive data gets into application logs

MITRE classifies the weakness as CWE-532 Insertion of Sensitive Information into Log File, a child of CWE-200 Exposure of Sensitive Information to an Unauthorized Actor. Its examples are ordinary: a logger.info call that includes a card number, a full user object written during exception handling, and a database error that includes the query text. Almost every leak follows one of a small number of paths.

What must never be logged, and what must be masked

The OWASP Logging Cheat Sheet lists data that should not be recorded directly: application source code, session identifiers (a hashed value may be used to track session events), access tokens, sensitive personal data and some PII such as health data and government identifiers, authentication passwords, database connection strings, encryption keys and other primary secrets, bank account or payment card holder data, data classified higher than the logging system may store, commercially sensitive information, and data the user has not consented to. It adds a second list needing special treatment: file paths, internal network names and addresses, and non-sensitive personal data such as names, phone numbers and email addresses.

ASVS 5.0 turns this into requirements. V14.1.1 asks that all sensitive data be identified and classified into protection levels, including data that is only encoded, such as Base64 strings or the payload of a JWT. V14.1.2 asks that each protection level documents how the data is to be logged and who can access it in logs. V16.2.5 asks the application to enforce logging by protection level: some data, such as credentials or payment details, may not be logged at all, and other data, such as session tokens, only hashed or masked.

A plain hash is not anonymisation. The OpenTelemetry security guidance warns that hashes can be reversed in practice when the input space is small and predictable, such as numeric user IDs, and suggests truncation or generalisation as alternatives. Use a keyed hash held outside the log platform when correlation is needed. For the privacy side of security logging (legal basis, pseudonymisation and retention), see the Privacy in Security Logging and Monitoring Guide.

Preventing leaks in application code

The control that works is a logging interface that makes the safe path the easy path.

• Structured logging with constant messages: emit JSON events with a fixed message template and named fields. The OWASP cheat sheet recommends structured formats over unstructured text, constant message templates and no string concatenation; it also notes that parameterised logging alone does not stop CR and LF injection, so use an encoder that escapes values (ASVS V16.4.1).
• A field allow-list per event type: define the schema for each event (for example auth.login.failed has user_id, source_ip, reason) and reject unknown fields in tests. The Secure Logging Vocabulary and Application Security Events Guide gives a vocabulary to start from.
• Redacting types: wrap secrets in a type whose string, JSON and debug representations print [REDACTED] (for example a Secret or SensitiveString class, Pydantic SecretStr in Python). The value cannot leak through a careless log call because the object never renders it.
• No whole-object logging: ban passing request, response, user, session and exception context objects directly to the logger. Log the exception class and a sanitised message; keep stack traces, but check driver messages for connection strings.
• Library-level filters as a backstop: in Python, attach a redaction logging.Filter to handlers rather than loggers, because filters on a logger do not apply to records propagated from its child loggers. Java logging frameworks offer pattern replacement in layouts; use it for known token formats only.
• Safe defaults in frameworks: turn off SQL parameter logging in ORMs, HTTP wire logging in clients and SDKs, and request body logging in web frameworks for production profiles. Fail the build if a production configuration sets the root log level to debug or trace.
• Size limits: cap field lengths. The cheat sheet advises limiting the size of user input used in log messages, which also limits how much pasted data can land in a single event.

Error responses deserve the same care. ASVS V16.5.1 requires a generic message to the consumer for unexpected or security-sensitive errors, with no stack traces, queries, secret keys or tokens. Detail belongs in the server log, after redaction.

Redaction in the logging pipeline

Pipeline redaction catches what code reviews miss, from every service, without a redeploy. It works best as an allow-list of attribute keys plus a deny-list of value patterns.

The OpenTelemetry Collector redaction processor, read for this guide in November 2026, is a useful model. It deletes span, log and metric attributes that are not on an allowed_keys list, and is designed to fail closed: if allowed_keys is empty, no attributes are allowed unless allow_all_keys is set to true. It masks values matching blocked_values regular expressions, supports blocked_key_patterns and allowed_values exceptions, can hash instead of masking through hash_function, and can add summary attributes that count what was redacted. For log records whose body is a map, it also reports redacted and masked body keys. Its README lists stability as alpha for logs and metrics and beta for traces, so test upgrades. Other collector processors (attributes, transform and filter) can delete, hash or drop records, as the OpenTelemetry sensitive data page describes.

Place pipeline redaction before the first durable store and before any fan-out to third parties. Redacting in the SIEM after indexing leaves clear copies in the agent buffer, the message queue and the raw archive. The OWASP cheat sheet also asks for due diligence before sending event data to third parties, which includes knowing what redaction runs first.

Finding sensitive data already in logs

Most organisations that start this work find historic leaks. Search in this order:

1. Known token formats: search indexes for prefixes and shapes that are distinctive, for example AWS access key IDs beginning AKIA or ASIA, GitHub tokens beginning ghp_ or github_pat_, Slack tokens beginning xoxb-, private key headers (-----BEGIN) and JWTs beginning eyJ. Secret scanning tools used on repositories can be pointed at exported log samples; see the Git Secret Scanning Guide for tool categories.
2. Sensitive field names: query for keys such as password, passwd, secret, token, authorization, cookie, set-cookie, api_key, ssn and card, in both structured fields and message text.
3. Personal data patterns: email addresses, phone numbers and national identifier formats relevant to your users, checked with validation (for example Luhn checks for card numbers) to reduce false positives. Data classification and DLP tools can scan log archives; see the Data Loss Prevention (DLP) Guide.
4. Seeded test data: run synthetic transactions with marked test credentials and test personal data through each service, then search the log platform for them. This finds leak paths before real data takes them.
5. Every copy: repeat the search in cold archives, backups, crash and support bundles, error tracking tools, APM and tracing backends, and any exports sent to vendors.

Responding when you find secrets or personal data

1. Rotate first: treat every credential found in a log as compromised. Revoke and reissue keys and tokens, and invalidate sessions, before cleaning up. For tokens, see the JWT and Session Tokens in Code and Logs Guide.
2. Check for use: review the issuer's audit logs for use of the exposed credential from unexpected sources during the exposure window, and review who accessed the log indexes that held it.
3. Purge copies: delete or re-index affected data in the log platform, archives and third-party tools, and record what could not be purged (for example immutable backups) with its expiry date.
4. Assess notification: personal data exposed to people who should not see it can be a personal data breach. Under GDPR Article 33, controllers notify the supervisory authority without undue delay and, where feasible, within 72 hours of becoming aware, unless the breach is unlikely to result in a risk to individuals. Involve your privacy team early.
5. Fix the path: correct the code or configuration, add a test that fails if the field appears again, and add a pipeline pattern for the format as a backstop.

Where identity fits

Most secrets found in application logs are non-human identity credentials: service account keys, API tokens, OAuth client secrets, webhook signing keys and database passwords. They usually have broad permissions, no MFA and long lifetimes, which makes a log entry a usable entry point long after it was written. Two measures limit the damage. Short-lived, workload-issued credentials reduce the value of anything that leaks; see the Secrets Management Guide. Log readers are themselves a privileged population: restrict read access to production log indexes, review it periodically and record access to logs, as the OWASP cheat sheet and ASVS V16.4.2 require.

How to implement: a phased approach

1. Inventory (weeks 1 to 2): build the logging inventory ASVS V16.1.1 asks for: what each layer logs, formats, where logs are stored, who can read them and for how long. Include tracing, error tracking and vendor tools.
2. Classify and set rules: map data classes to the never, masked and allowed rules above, and publish them in the logging standard. See the Logging and Monitoring Policy Template.
3. Baseline scan: run the discovery searches across current indexes and archives. Rotate and purge what you find before changing anything else.
4. Pipeline backstop: deploy collector or agent redaction in report-only mode first (count matches without masking, where the tool supports it, or mirror a sample to a test index), tune false positives, then enforce. Keep the previous configuration for rollback; a redaction rule that drops too much is rolled back in minutes, a leak is not.
5. Fix at source: introduce the shared logging library, redacting types and framework defaults service by service, starting with authentication, payments and customer data services.
6. Prevent regressions: add CI tests that run seeded transactions and fail on test secrets in captured logs, lint rules against logging whole objects, and code review prompts for logging changes.

Decision guidance

• Small teams with one log platform: start with the shared library and the platform's own masking features; schedule a monthly token-format search.
• Large estates with many languages: invest in the collector layer first for coverage, then fix services in order of data sensitivity.
• Mask or drop: mask when the event is still useful without the value; drop when the event exists only to show the sensitive payload.
• Hash or omit identifiers: use a keyed hash when investigators must correlate events across systems; omit when they do not.

Detection and measurement

Run the token-format and field-name searches as scheduled detections on production indexes, alerting the owning team with the service name and event type, not the value. Monitor the redaction layer itself: a sudden fall in redaction counts often means a service changed its field names and is bypassing the rule.

• Secrets found in logs: credentials detected per month, by service; the target trend is towards zero, and every hit is rotated.
• Time to rotate: median hours from detection of a credential in logs to its revocation.
• Allow-list coverage: percentage of services emitting structured logs through the shared library with an approved field schema.
• Pipeline redaction hits: masked values per million events, per service; persistent hits identify code to fix at source.
• Seeded data escapes: number of test credentials or test personal data values found in log stores after CI and synthetic runs (target zero).

Framework mapping

In ATT&CK, T1552 Unsecured Credentials describes adversaries searching compromised systems for insecurely stored credentials, which includes credentials written to log files. CIS Control 8 covers audit log management; see the CIS Control 8 Audit Log Management Guide.

Common failure patterns

• Logs stored where anyone can reach them: in the DeepSeek Database Exposure 2025, an internet-reachable database without authentication held a log table with plaintext chat history, API keys and backend details. Leaked log content and weak access controls compound each other.
• Debug left on after an incident: wire-level logging enabled to troubleshoot, never reverted, quietly capturing Authorization headers for months.
• Redacting only in the SIEM: the search interface shows masked values while the raw archive, agent buffers and vendor exports hold clear text.
• Regex-only programmes: pattern rules for a handful of token formats, with no field allow-list, so passwords in a field called pwd or a Base64-encoded credential pass straight through.
• Cleaning up without rotating: deleting log lines that contained a key while leaving the key valid.
• Forgetting telemetry and AI logs: tracing spans with full SQL and URLs, or prompt logs holding pasted customer data, outside the scope of the logging standard.

Practitioner checklist

• A logging inventory lists every layer, store, reader group and retention period, including tracing, error tracking and vendor tools.
• Data classes map to never, masked or allowed rules in a published logging standard.
• Services log structured events with constant messages, an approved field schema and an escaping encoder.
• Credentials and tokens are wrapped in redacting types; whole-object, request and response logging is banned in production.
• Production configurations block debug level, SQL parameter logging and HTTP wire logging.
• Pipeline redaction runs before the first durable store with a fail-closed attribute allow-list and value patterns for known token formats.
• Scheduled searches for token formats and sensitive field names run across indexes and archives, alerting owners without showing values.
• Every credential found in logs is rotated before purge, and every copy is tracked to deletion.
• CI runs seeded transactions and fails if test secrets or test personal data appear in captured logs.
• Read access to production logs is restricted, reviewed and itself logged.

Standards and references

• OWASP: Application Security Verification Standard 5.0 (V14 Data Protection, V16 Security Logging and Error Handling)
• OWASP: Logging Cheat Sheet
• MITRE: CWE-532 Insertion of Sensitive Information into Log File
• NIST: SP 800-122 Guide to Protecting the Confidentiality of Personally Identifiable Information
• OpenTelemetry: Handling sensitive data
• OpenTelemetry: Collector redaction processor
• Python Software Foundation: Logging Cookbook

Related NHI Mgmt Group resources: Log Management and Retention Guide · Privacy in Security Logging and Monitoring Guide · Developer Secret Scanning Triage · Data Classification Guide