Data Masking vs Data Redaction: Choosing for Logs, Prompts and Traces

Teams say "we mask it" when the sensitive value is in fact still stored, still searchable by anyone with an unmask permission, and still recoverable from a backup. Masking and redaction both hide data, but they make different promises: masking changes what a reader sees or what a copy contains while often keeping the original somewhere, and redaction removes the value so it cannot be recovered from that record. The difference matters most in the places sensitive data now leaks fastest: application logs, LLM prompts and completions, and distributed traces. This guide is for engineers, platform teams and privacy leads choosing between the two for those pipelines. It does not re-explain masking types, which are covered in the Static vs Dynamic Data Masking guide.

Key takeaways

• Ask where the original lives. If an unmasked copy exists anywhere (the log store, a key, a vault, a backup), you have masking or pseudonymisation, not redaction, and the original still needs protecting.
• Redaction is the default for telemetry. Logs, prompts and traces rarely need the raw value, so removing it at the earliest hop is cheaper than guarding it everywhere downstream.
• Masking earns its place when utility needs shape. Test data, support screens and analytics that need format, length or joinability justify masking or reversible tokens, with the key or vault treated as a crown jewel.
• Hashes and partial masks are not redaction. Hashing small, predictable inputs is reversible in practice, and partial masks can combine with other fields to re-identify a person.
• Measure the leak, not the policy. Track findings that reach storage, unmask events and residual detections in sampled records, because detection-based redaction always misses something.

The short answer

Use redaction when nobody downstream needs the value: secrets, credentials, full payment card numbers, free-text personal data in logs, and personal data in prompts sent to a model provider. Use masking when the consumer needs something that looks like the value (a test database, a support agent who needs the last digits, an analyst who needs to join records), and accept that you are now running a reversibility or re-identification control that must itself be governed.

The test is reversibility from the protected record plus anything else your organisation holds. The Encryption vs Hashing vs Masking guide covers the cryptographic side; this page applies the question to pipelines.

Reversibility is a spectrum, and vendors use the words differently

Product documentation does not use "mask" and "redact" consistently, so read what the transformation does rather than its name. Three widely used toolsets illustrate the range (listed alphabetically; examples, not recommendations).

• Amazon CloudWatch Logs data protection: a data protection policy detects sensitive data at ingestion and masks it at egress points, including Logs Insights, metric filters and subscription filters. Principals with the logs:Unmask IAM permission can view unmasked data, so this is view-time masking with the original retained, not redaction. Events ingested before the policy was applied are not masked.
• Google Cloud Sensitive Data Protection: the transformation reference lists redactConfig, replaceConfig, characterMaskConfig and cryptoHashConfig as non-reversible, and cryptoReplaceFfxFpeConfig (format-preserving encryption) and cryptoDeterministicConfig as reversible through the content.reidentify method. Date shifting and bucketing generalise values.
• Presidio (open source): anonymiser operators are replace, redact, mask, hash, encrypt, custom and keep. Only encrypt can be reversed, with the decrypt deanonymiser.

OpenTelemetry adds a fourth vocabulary: its Collector redaction processor deletes attributes that are not on an allow list, and masks values matching blocked_values patterns with a fixed run of asterisks (or hashes them when hash_function is set). So a "redaction processor" can perform masking. Write your internal standard in terms of outcomes, for example "irreversible removal", "irreversible substitution", "reversible with key K held by team T", "view-time masking with original retained", then map each product setting to one of those.

Threat model for each option

Masking

• Unmask right sprawl: view-time masking is an authorisation control. Its strength equals the hygiene of whoever holds logs:Unmask or the database unmask role, including automation roles and break-glass accounts.
• Key or vault compromise: reversible masking concentrates risk in one key or token vault. Steal the key and every masked copy across test, analytics and logs is readable at once.
• Re-identification: irreversible masks still leak. Last four digits of a card, a truncated postcode, a birth year and a job title together can single out a person. The Anonymisation and Re-identification Risk Guide covers the tests.
• Guessable hashes: OpenTelemetry's own guidance warns that hashes of small, predictable inputs such as numeric user IDs are reversible in practice. Unsalted hashes of email addresses or phone numbers fall to a dictionary in minutes.

Redaction

• False negatives: detection-based redaction relies on patterns, checksums, keywords and machine learning. Novel token formats, secrets split across lines, personal data in non-English text and values inside encoded blobs slip through.
• Late redaction: redacting in the SIEM or at query time means the raw value already crossed the network, sat in buffers and was written to disk, dead-letter queues and backups.
• Over-redaction: blanking every number or identifier destroys correlation, so engineers turn on verbose debug logging that bypasses the control.
• Cosmetic document redaction: drawing a black box over text in a PDF or image layer leaves the underlying text extractable. True document redaction removes the content and its metadata.

Architecture in words: where each control sits in a pipeline

The earlier the hop, the fewer systems hold the raw value. A practical order of preference for telemetry:

1. Do not emit it: structured logging with typed fields lets code exclude secrets and personal data before serialisation. The OWASP Logging Cheat Sheet lists data that should usually not be recorded directly, including session identifiers, access tokens, passwords, connection strings, encryption keys and payment card data, and suggests a hashed stand-in where a session must be tracked. ASVS 5.0 requirement V16.2.5 asks that logging follows the data's protection level, for example never logging credentials or payment details and only logging session tokens hashed or masked.
2. Redact in the process or sidecar: logging library filters and SDK hooks redact before the record leaves the host.
3. Redact at the collector: OpenTelemetry Collector processors (attributes, filter, redaction, transform) apply policy centrally before export. Note the README lists the redaction processor as beta for traces and alpha for logs and metrics, so pin versions and test upgrades.
4. Mask at the store: view-time masking in the log platform protects human readers but leaves the original stored. Use it as a second layer, not the first.

For LLM traffic the equivalent chain is: client or agent, then an AI gateway that redacts or substitutes before the model call, then the provider, then the trace and evaluation store. The AI Gateway and LLM Gateway Guide covers where that gateway sits.

LLM prompts, completions and traces

Prompts concentrate sensitive data: users paste customer records, retrieval pulls documents into context, and agents pass tool outputs (often containing tokens and personal data) back into the next call. OWASP LLM02:2025 Sensitive Information Disclosure recommends sanitisation, including scrubbing or masking sensitive content, plus strict access control and restricted data sources. MITRE ATLAS tracks the resulting leak as AML.T0057 LLM Data Leakage. The OWASP LLM Top 10 LLM02 Sensitive Information Disclosure Guide covers the risk itself; the decision here is which transformation to apply where.

OpenTelemetry's GenAI semantic conventions (Development status as checked in November 2026) mark gen_ai.input.messages, gen_ai.output.messages and gen_ai.system_instructions as Opt-In attributes, with _ref variants that point to content held in external storage. That design supports the right default: keep content out of spans, and if you must keep it, store it separately with its own redaction, access control and retention. The Secure AI logging guide covers what else belongs in AI logs.

Reversible substitution needs care. The placeholder map is a re-identification key. Hold it in memory or a short-lived store scoped to the session, never in the trace, and make sure restored responses are not written back to logs unredacted.

Logs and test data: the two classic cases

Application and security logs: redaction is the default because responders need the event, the actor and the outcome, not the card number. Where correlation across events matters (the same customer or session in many records), use a keyed hash (HMAC with a secret held outside the log platform) rather than a plain hash, and rotate the key on a schedule that matches retention. The Sensitive Data in Logs Guide covers finding and cleaning data already in logs, and the Privacy in Security Logging and Monitoring Guide covers lawful basis and controlled re-identification for investigations.

Test and development data: masking is usually the right tool because realistic format, referential integrity and edge cases matter to testing. Redaction alone produces empty columns that break validation. Use irreversible, consistent masking (same input gives the same output across tables) so joins survive, and keep the masking key out of the non-production environment. See Production Data in Non-Production.

Decision criteria

Small organisations get most of the value from two moves: library-level filters that drop known secret fields, and a collector or gateway rule set with deny-by-default attribute allow lists. Large organisations need a central transformation catalogue (outcome classes as above), a named owner for each reversible key, and detection on unmask events across platforms.

Worked scenarios

• Payment service logs: a checkout service writes request bodies at debug level. Decision: drop body logging in production, redact card numbers and CVV at the collector as a backstop, keep a keyed hash of the customer ID for fraud correlation. View-time masking in the log platform stays as a third layer for anything missed.
• Support copilot: agents paste ticket text into an internal assistant backed by an external model. Decision: the gateway replaces names, emails and account numbers with placeholders, restores them in the reply shown to the agent, and drops the map at session end. Traces record token counts and latency but not message content.
• Agent with CRM tool access: an AI agent calls a CRM API whose responses include contact data and an OAuth refresh token in a nested field. Decision: redact the token at the tool adapter before the result is returned to the model, mask contact identifiers that the next tool call needs as consistent tokens, and treat any trace that captured the token as a secret exposure requiring rotation.
• Regression test database: QA needs production-like data. Decision: irreversible deterministic masking of names, emails and identifiers so joins work; the masking secret stays in the production pipeline only; free-text notes are redacted entirely because masking cannot make them safe.

Where identity fits

Credentials are the sensitive data most often found in logs and traces: API keys, bearer tokens, cookies, signed URLs and service account keys emitted by HTTP clients, SDK debug modes and agent tool outputs. ATT&CK T1552.001 Credentials In Files describes attackers harvesting exactly this. For a secret, masking is never sufficient: redact it, and if it was ever written in clear, rotate it, because you cannot prove every copy is gone. Reversible masking keys and unmask permissions are themselves privileged non-human access: assign them to named workload identities, keep them out of CI variables shared across environments, and log every use. The Pseudonymisation Techniques Guide covers key separation in more depth.

How to implement: phased approach

1. Inventory flows: list every log source, trace exporter, AI gateway, evaluation store and test data job, and record what sensitive fields each can carry and who consumes them.
2. Classify outcomes: for each field and flow, choose an outcome class (removal, irreversible substitution, keyed one-way, reversible with key, view-time masking) using the decision table. Record the key owner for every reversible choice.
3. Start in detect-only mode: run detectors against a copy of traffic or a sample of stored records to measure volumes and false positives before enforcing. Platform features such as CloudWatch's LogEventsWithFindings metric help size the problem.
4. Enforce at the earliest hop: fix emitters first (stop logging bodies and headers), then add collector and gateway rules with attribute allow lists rather than deny lists where possible.
5. Test with seeded data: inject canary values (fake card numbers, synthetic tokens, test names) at each source and confirm none reach storage, model providers or traces in clear.
6. Roll out in rings: one service or tenant first, then by environment. Keep the previous collector configuration versioned so you can roll back a rule that breaks debugging, but never roll back to a configuration that exports secrets.
7. Clean the past: view-time masking and new rules do not touch history. Purge or rewrite stored records that contain secrets or out-of-policy personal data, including backups and exports, and rotate exposed secrets.

Detection and measurement

• Residual findings rate: sensitive data detections per million stored records in sampled logs and traces, trending to zero for secrets.
• Canary escape count: seeded test values that reached storage or a model provider in clear per test cycle.
• Unmask and reidentify events: count and review every logs:Unmask, database unmask, content.reidentify or decrypt operation; alert on use by unexpected identities.
• Time from emission to redaction: how many hops and seconds a raw value survives before removal.
• Prompt redaction coverage: share of LLM calls passing through a gateway with redaction enabled, against total calls seen in egress logs.
• Exposed secret rotation time: hours from detection of a secret in telemetry to its revocation.

Common failure patterns

• Calling view-time masking redaction: the log platform hides values on screen, the risk register says "redacted", and the raw data sits in storage and exports readable by any role with the unmask right.
• Plain hashes of identifiers: unsalted SHA-256 of emails or numeric IDs labelled as anonymised, then reversed by a dictionary.
• Redacting only the message, not the attributes: the log line is clean but the same value sits in a span attribute, HTTP header field, exception stack or URL query string.
• Prompt content in traces by default: an observability SDK captures full prompts and completions, so the trace store becomes the largest unprotected copy of customer data.
• Reversible map stored next to the data: placeholder mappings or masking keys kept in the same database, bucket or trace as the masked output.
• Policy applied going forward only: new rules switched on, history never cleaned, secrets never rotated.

Framework mapping

Practitioner checklist

• Define outcome classes and map every masking or redaction setting in use to one of them.
• Name an owner and storage location for every reversible key, token vault and placeholder map.
• Stop emitting secrets, request bodies and auth headers at the source before adding pipeline rules.
• Use attribute allow lists at the collector and gateway; treat deny lists as a backstop.
• Keep prompt and completion content out of traces unless there is a documented need.
• Use keyed hashes, not plain hashes, wherever correlation is required.
• Seed canary values and test that none escape on every release of the pipeline.
• Alert on unmask, reidentify and decrypt operations by unexpected identities.
• Clean historical stores and rotate any secret found in telemetry.

Standards and references

• OWASP: LLM02:2025 Sensitive Information Disclosure
• OWASP: Logging Cheat Sheet
• NIST: SP 800-122 Guide to Protecting the Confidentiality of Personally Identifiable Information
• ICO: Pseudonymisation guidance
• OpenTelemetry: Handling sensitive data
• AWS: Help protect sensitive log data with masking
• Google Cloud: Sensitive Data Protection transformations reference
• Presidio: Anonymizer documentation

Related NHI Mgmt Group resources: Tokenisation and Data Masking Guide · Static vs Dynamic Data Masking · Encryption vs Hashing vs Masking · GenAI DLP vs Traditional DLP · Data Minimisation