Tokenisation and Data Masking Guide: Vaulted and Vaultless

Card numbers, national identifiers and other sensitive values spread into analytics, logs, support tools and test copies far faster than any encryption programme can follow, and every copy widens audit scope and breach impact. Tokenisation and data masking shrink that footprint by replacing real values with surrogates, so most systems and people never handle the original. This guide covers how data tokenisation works (vaulted, vaultless, deterministic and format-preserving), payment and network tokenisation for PCI DSS scope reduction, static and dynamic masking, redaction and test data management. For the wider picture see the Data Security Guide. It is for data security architects, payments and PCI teams, platform engineers and privacy leads choosing or running these controls.

Key takeaways

• Tokenisation moves risk, it does not remove it. PCI SSC states that tokenisation solutions "do not eliminate the need to maintain and validate PCI DSS compliance"; the token vault, keys and detokenisation API become the crown jewels.
• Vaulted and vaultless trade storage for cryptography. A vault keeps a mapping table that must be secured and replicated; vaultless derives tokens with keys or algorithms, so key management replaces database protection.
• Payment network tokens are a different control. Network tokens issued by a card network's token service are restricted to a merchant, device or channel, which limits what a stolen token is worth; merchant security tokens reduce your scope but are not payment instruments.
• Dynamic masking is a presentation control. It hides values in query results while the stored data is unchanged, and administrators typically see everything; use it for over-viewing, not as protection against a privileged or compromised account.
• Test data is where masking pays off fastest. Production copies in development and test environments are a common source of exposure; static masking or synthetic data, enforced in the refresh pipeline, removes that risk at the source.

Tokenisation, masking, redaction and encryption compared

These techniques are often used interchangeably but behave differently. Google's Sensitive Data Protection documentation is a useful neutral taxonomy: redaction "Deletes all or part of a detected sensitive value", masking replaces characters "with a specified fixed character", crypto-based tokenisation (pseudonymisation) "Encrypts the original sensitive data value using a cryptographic key" and is reversible with that key, and cryptographic hashing "isn't reversible".

Tokenisation vs encryption: encrypted data is mathematically derived from the original, so anyone with the key and ciphertext can recover it, and it usually changes length and format. A random vault token has no mathematical relationship to the original value; the ICO notes that knowing a token does not let an attacker reverse it. In practice tokenisation and encryption are combined: the vault encrypts what it stores, and vaultless schemes are encryption by design. Choose tokenisation when many systems need a stand-in and few need the original; choose encryption when the service that stores data also reads it. FPE details sit in the Encryption Guide.

Legal status: under UK and EU GDPR, tokenised data is pseudonymised, not anonymous. The ICO states that "Pseudonymised data is personal data in the hands of someone who holds the additional information", and recommends keeping the additional information (vault or keys) separate, restricted and logged. See the Privacy-Enhancing Technologies Guide.

How data tokenisation works: vaulted and vaultless

PCI SSC's Tokenization Guidelines define tokenisation as "a process by which the primary account number (PAN) is replaced with a surrogate value" and detokenisation as "the reverse process of redeeming a token for its associated PAN value". The same model applies to PII tokenisation and wider sensitive data tokenisation: national identifiers, email addresses, account numbers and health identifiers. The guidelines list three generation methods: "A mathematically reversible cryptographic function", "A one-way non-reversible cryptographic function (e.g., a hash function with strong, secret salt)", and "Assignment through an index function, sequence number or a randomly generated number".

• Deterministic tokenisation: the same input always yields the same token, so joins, counts and look-ups work on tokenised data. The ICO notes that consistent tokens can link people across databases. That linkage is also the risk, enabling frequency analysis and re-identification, so scope each deterministic key or vault namespace to one purpose.
• Format preserving tokenisation: the token keeps the length and character set of the original (16 digits, a valid-looking email), so legacy schemas and validation rules keep working. NIST SP 800-38G Rev. 1 (second public draft, February 2025) specifies only FF1 and states "The encryption method FF3 is no longer specified"; the 2016 version required a domain of at least one hundred values and recommended at least one million. Short fields such as dates of birth have too few possible values to tokenise safely this way, and Google advises against its FPE method "except when preserving the input alphabet space and size is a requirement".
• Preserving parts of the value: many schemes keep the BIN and last four digits of a PAN so routing and customer service still work. That is acceptable for display under PCI DSS 3.4.1, but the more digits a token preserves, the fewer it hides.
• Real time tokenisation: values are tokenised at the point of capture (browser, API gateway, message broker) before any application stores them, instead of in nightly batches. This gives the biggest scope reduction and the hardest availability requirement.
• Detokenisation and tokenisation with rehydration: rehydration is detokenising at the last possible moment, for example when a payment is sent to the processor or a letter is printed. Every detokenisation path is an access decision: restrict it to named services, return partial values where possible, and log each call.
• Unified tokenisation system: one token service, policy model and key hierarchy for all channels and data types, instead of separate tokenisation in each product. It gives consistent tokens and audit but is a single point of failure and a prime target, so it needs multi-region resilience and separated administration.

Payment tokenisation and PCI scope reduction

PCI tokenisation reduces scope only when systems handle tokens that cannot be used to retrieve PAN. PCI SSC states that components which can generate or redeem tokens remain in scope, and token-only components may fall out of scope only if they are segmented from the tokenisation system and the cardholder data environment. The card data vault, "the central repository for PANs and tokens", is an attractive target and needs controls beyond the PCI DSS baseline. See the PCI DSS 4.0 Guide.

• Merchant and acquirer security tokens: issued by a tokenisation service provider (often the acquirer, gateway or an in-house service) and usable only within that relationship.
• High-value tokens: PCI SSC warns that "Tokens that can be used as payment instruments (sometimes called 'high-value tokens')" may be as valuable to an attacker as PAN and may need additional fraud controls. A card on file token that your platform can charge without the customer is one of these; protect the charge path as you would PAN.
• Payment network tokenisation: under the EMV Payment Tokenisation framework, an EMV payment token replaces PAN and "is constrained in how it can be used", for example to a specific merchant, device or payment scenario. Tokens are issued by a token service provider registered with EMVCo, which "manages and assigns TSP Codes worldwide", and issuers can "control and replace payment tokens for specific merchants, devices or transaction types".
• Card on file tokenisation: storing network tokens instead of PANs for recurring and one-click payments. Network tokens follow the transaction through to issuer authorisation and are compatible with acquirer security tokens, so merchants commonly hold both.
• Tokenisation mandate: regulators and card schemes increasingly require tokens for stored cards. The Reserve Bank of India's circular of 28 July 2022 (RBI/2022-2023/95) states that "no entity in the card transaction / payment chain, other than the card issuers and / or card networks, shall store CoF data", with data purged before 1 October 2022, which made card-on-file tokenisation the practical route for Indian merchants.

Where PAN must still be stored, PCI DSS 3.5.1 lists four ways to render it unreadable: truncation, full encryption, index tokens and one-way hashes; PCI SSC FAQ 1492 describes truncation as removing the middle digits permanently and notes that hashes generated after 31 March 2025 must be keyed (3.5.1.1). Display masking is a separate rule: under 3.4.1 "no more than the BIN and last four digits of the PAN are displayed" without a documented business need. For fraud on tokenised flows see the Payment Fraud Guide.

Data masking and redaction

Data masking replaces sensitive values with realistic but fictitious or obscured ones. Sensitive data masking comes in three operational forms, and redaction is a fourth, cruder tool.

• Static and bulk data masking: masking rules run over a full copy of a database or file set, producing a masked copy that never contains the originals. Bulk data masking must preserve referential integrity (the same customer masked the same way in every table) and check digits, or tests break. It is irreversible by design.
• Dynamic data masking: the database or a proxy masks values in query results according to the user's role. Microsoft describes it as hiding "the sensitive data in the result set of a query over designated database fields, while the data in the database isn't changed", with functions for credit cards (last four digits), email and custom strings, and an UNMASK permission grantable at database, schema, table or column level. The same page notes that server admins, Entra admins and db_owner "can view the original data without any mask". Use it for support and reporting screens; it does not stop a user with query access from inferring values or an administrator from reading them. See the Database Security Guide.
• ERP and SAP data masking: large ERP estates such as SAP hold HR, payroll, banking and customer data across thousands of tables, and system copies for project and support landscapes are frequent. SAP data masking needs knowledge of the application data model and must run inside the system copy process, or unmasked clients persist in non-production for years.
• Data redaction and PII redaction: removing or blacking out values in documents, logs, call transcripts, tickets and AI prompts and outputs. Redaction depends on detection (pattern, dictionary and context models), so it misses what detection misses. Measure its recall on samples and never rely on it as the only control for regulated data. See the Generative AI Data Loss Prevention Guide.

NIST SP 800-188 adds a caution that applies to all of these: "not all tools that merely mask personal information provide sufficient functionality for performing de-identification." Masking direct identifiers leaves quasi-identifiers such as postcode and date of birth, which together can re-identify people.

Test data management

Test data management supplies development, test and training environments with data that is fit for purpose and safe, and is where test data masking delivers most value.

1. Set the rule: no live sensitive data outside production without a documented exception. PCI DSS 6.5.5 already prohibits live PANs in pre-production unless that environment is in the CDE and fully protected, and ISO/IEC 27001:2022 control 8.33 requires test information to be appropriately selected, protected and managed.
2. Classify first: discovery and classification decide which columns get masked. A masking job is only as good as its column inventory. See the Data Classification Guide.
3. Choose per use: masked production subsets for integration and performance testing, synthetic data for new features and outsourced development, deterministic tokens where test cases must join across systems.
4. Automate the refresh: masking runs inside the copy pipeline, before data lands in the target, and the pipeline fails closed if a masking rule errors.
5. Verify: sample masked copies for residual real values (known test identities, real email domains, valid PAN ranges) after every refresh.

Choosing an approach

• Values that must be recovered by a few services: vaulted or vaultless tokenisation with tightly controlled detokenisation.
• Values that must be joined but never recovered: keyed hashing or deterministic tokens with a purpose-specific key.
• Values people must see partially: dynamic masking or partial detokenisation (last four digits).
• Copies outside production: static masking or synthetic data; never dynamic masking alone.
• Unstructured text: redaction, backed by limits on where the text can go.
• Small organisations: use a hosted payment page and provider tokens so PAN never reaches you, and the masking built into your database or cloud platform for PII.
• Large organisations: a central token service with per-data-type policy, masking in every refresh pipeline, and owner approval for each detokenisation consumer.

Where identity fits: detokenisation is an access decision

Once data is tokenised, the attack path moves to whoever can call detokenise, hold the FPE key, or carry UNMASK. These are mostly service accounts and workload identities, often over-granted. Inventory every identity that can detokenise or unmask, give each the narrowest data types and return format (full, partial or masked), require separate administrators for the token service and for the applications that use it, and keep keys in an HSM or cloud KMS with usage logging. Support tooling deserves the same scrutiny: the Coinbase insider breach shows how much damage follows when support staff can see full customer records, which is exactly what default-masked support screens and audited unmask requests are meant to limit. See the Service Account Security Guide and the Cryptographic Key Management Guide.

Implementing tokenisation and masking

1. Map the data flows. For each sensitive data type, list where it enters, where it is stored or displayed, and who needs the original.
2. Pick the capture point. Tokenise as early as possible (payment page, API gateway, ingestion pipeline) so downstream systems are built on tokens.
3. Pilot on one flow in shadow mode. Write tokens alongside originals for a period, compare reports and application behaviour, and confirm joins and validations still work.
4. Cut over and backfill. Switch consumers to tokens, then replace historical originals in batches with checkpoints. Keep the originals in an encrypted, access-restricted holding store until reconciliation passes, which is your rollback.
5. Purge. Delete originals from non-vault systems, logs and caches (backups on normal expiry) and keep evidence for auditors.
6. Extend masking to non-production. Add static masking to every refresh pipeline, then dynamic masking to support and reporting roles.
7. Test resilience. Fail over the token service and key store; real-time tokenisation puts them on the critical path.

Detection and measurement

Log every detokenisation and unmask event with caller identity, data type, volume and purpose, and alert on volume spikes, new callers, calls outside business patterns and bulk detokenisation from interactive accounts. Scan logs, data lakes and non-production stores for original values. Useful metrics:

• Clear-text footprint: systems holding original values of each sensitive type.
• Detokenisation consumers: count of identities with detokenise or UNMASK rights, each with an owner and review date.
• Non-production coverage: share of non-production refreshes that passed masking verification.
• Residual exposure findings: real values found by discovery scans in tokenised or masked stores per quarter.
• Redaction recall: share of seeded test values caught by redaction in logs and transcripts.
• PCI scope: number of system components in scope before and after tokenisation.

Framework mapping

Common failure patterns

• Tokenising the database, not the flow. Original values still arrive in logs, message queues, analytics exports and error reports. Tokenise at capture and scan for leaks.
• Detokenise for everyone. Every application gets full detokenisation, recreating the exposure through an API. Default to partial returns.
• Dynamic masking treated as security. Administrators and anyone with direct query access bypass it, and repeated filtered queries can infer values. Pair it with least privilege and query monitoring.
• Format-preserving schemes on tiny domains. Tokenising dates of birth or short codes produces tokens that can be enumerated. Use random vault tokens or generalisation instead.
• Estate-wide deterministic keys. One key across all datasets makes every tokenised copy linkable. Scope keys per purpose and rotate them.
• Unmasked test copies. A one-off production refresh for a project becomes a permanent clear-text copy. Enforce masking in the refresh pipeline and expire non-production data.

Practitioner checklist

• Map each sensitive data type from capture to deletion and record who truly needs the original value.
• Choose vaulted or vaultless per data type and document the reason, the key or vault owner and the recovery plan.
• Avoid format-preserving tokens for small domains; follow NIST FF1 guidance where FPE is required.
• Scope deterministic tokens and keyed hashes to a single purpose with their own key.
• Use payment network tokens for card on file and treat chargeable tokens as high-value.
• Restrict detokenisation and UNMASK to named identities with partial returns by default, and review them quarterly.
• Log and alert on detokenisation volume, new callers and bulk requests from interactive accounts.
• Enforce static masking or synthetic data in every non-production refresh pipeline and verify each copy.
• Measure redaction recall with seeded values before relying on it for logs, transcripts or AI prompts.
• Re-run PCI scoping after tokenisation and keep evidence that original values were purged.

Standards and references

• PCI SSC: PCI DSS Tokenization Guidelines Information Supplement
• PCI SSC: FAQ on PAN masking and truncation
• EMVCo: EMV Payment Tokenisation
• NIST: SP 800-38G Rev. 1 Methods for Format-Preserving Encryption (draft)
• NIST: SP 800-188 De-Identifying Government Datasets
• ICO: Pseudonymisation
• Google Cloud: Sensitive Data Protection transformation reference
• Microsoft: Dynamic data masking

Related NHI Mgmt Group resources: Data Security Guide · Encryption Guide · Privacy-Enhancing Technologies Guide · PCI DSS 4.0 Guide · Database Security Guide