Differential Privacy vs Anonymisation for AI Training Data: Which Survives Memorisation and Re-identification

Teams that anonymise a dataset before training a model often assume the model inherits that protection, but models memorise, and memorised records can be extracted or inferred long after the anonymisation step has been signed off. Differential privacy applied during training (DP-SGD) and anonymisation applied before training defend against different attacks, fail in different ways and carry different legal weight. This guide compares them only for machine learning training sets: how each holds up against memorisation, membership inference, reconstruction and linkage re-identification, what epsilon budgets and k-anonymity thresholds actually guarantee for a model, what each costs in accuracy, how European regulators treat the resulting dataset and model, and how to choose by data type and release model. It is for ML engineers, privacy engineers, data protection officers and AI security reviewers. The mechanics of each technique are covered in the Differential Privacy Guide and the Anonymisation and Re-identification Risk Guide; this page does not repeat them.

Key takeaways

• They protect different things. Pre-training anonymisation protects the dataset against linkage; DP-SGD protects the trained model against revealing whether, and what, any one record contributed. A model trained on "anonymised" data has no formal guarantee of its own.
• Scrubbing reduces leakage but does not remove it. Research on GPT-2 models fine-tuned on case law, health and email data found scrubbing imperfect, and sentence-level DP still leaked about 3% of PII sequences.
• Epsilon is only meaningful with its unit. NIST SP 800-226 calls user-level privacy an excellent default and warns that large epsilon values may not provide meaningful privacy; record-level DP-SGD on data where one person has many records protects much less than the number suggests.
• DP costs accuracy, and public pre-training recovers much of it. NIST notes DP training typically lowers accuracy, sometimes significantly, and that pre-training on public data before private fine-tuning reduces that cost.
• The model's legal status is assessed separately. The EDPB's Opinion 28/2024 treats a model as anonymous only if both direct extraction and extraction through queries are insignificant in likelihood, assessed case by case, with DP and pre-processing as supporting evidence, not proof.

The short answer

If the threat is someone linking a released training dataset back to people, anonymisation is the relevant control, but it rarely holds for the rich, high-dimensional data models learn best from. If the threat is someone querying or inspecting the trained model to learn about individuals in the training set, only differential privacy applied during training gives a bound, and only for the unit of privacy you chose. For most models trained on personal data, the defensible design is layered: minimise and scrub before training to remove direct identifiers and obvious secrets, train or fine-tune with DP-SGD at a stated user-level epsilon where accuracy allows, and test the model for extraction and membership inference before release. Neither technique alone survives all three attack families.

Side-by-side comparison

Attack models against each

MITRE ATLAS groups the model-side attacks under AML.T0137 Training Data Information from AI Models, with sub-techniques AML.T0137.000 Infer Membership, AML.T0137.001 Infer Properties, AML.T0137.002 Invert AI Model and AML.T0137.003 Extract Memorized Data. Linkage re-identification is the dataset-side attack. The table shows which technique addresses each.

See the Membership Inference Attacks page for the inference mechanism.

Pre-training anonymisation for training sets

Anonymisation for training usually means removing direct identifiers, generalising or suppressing quasi-identifiers in structured fields, and scrubbing named entities and secrets from free text. The Article 29 Working Party's Opinion 05/2014 frames the test as three risks: singling out, linkability and inference. Its own comparison finds that k-anonymity removes singling out but leaves linkability and inference, and that pseudonymisation leaves all three, so it is not anonymisation.

Threat model

• Linkage: Narayanan and Shmatikov showed that the Netflix Prize ratings of about 500,000 subscribers could be re-identified with a small amount of auxiliary knowledge, using IMDb reviews, even though the data had been released as anonymous. Training data for recommendation, fraud and health models has the same high-dimensional shape.
• Residual identifiers in text: Carlini and colleagues extracted hundreds of verbatim sequences from GPT-2, including names, phone numbers and email addresses, some present in only one training document. A scrubber that misses a single occurrence leaves it learnable.
• Secrets and credentials: scraped corpora contain more than personal data. The 12,000 Live Secrets in LLM Training Data analysis found 11,908 live credentials in a single Common Crawl archive, a reminder that pre-processing filters give no guarantee of catching everything.

Where it fails in training specifically

• Quasi-identifiers in features: engineered features such as visit sequences, location histories or purchase baskets act as quasi-identifiers even when no single field does.
• Scrubbing recall: Lukas and colleagues found scrubbing lowers PII leakage but is imperfect, and trades disclosure against dataset usefulness.
• Utility collapse: generalising enough to reach a meaningful k on high-dimensional data often removes the signal the model needed, so teams quietly lower k or drop fields from the quasi-identifier list.
• Pseudonymised mistaken for anonymous: consistent tokens let records about one person be linked; under GDPR Recital 26 pseudonymised data remains personal data. See the Pseudonymisation Techniques Guide.

Differential privacy in training (DP-SGD)

NIST SP 800-226 (March 2025) states that machine learning techniques do not automatically protect privacy, that deep networks often memorise training data, and that DP-SGD is the most commonly used technique for training differentially private models, building on Abadi and colleagues' 2016 work showing deep networks can be trained under a modest privacy budget. In each step the gradient of every example is clipped to a maximum norm, Gaussian noise is added to the summed gradients, and an accountant tracks cumulative privacy loss across steps.

In the open source Opacus library for PyTorch, for example, PrivacyEngine.make_private() takes noise_multiplier and max_grad_norm, while make_private_with_epsilon() takes target_epsilon, target_delta, epochs and max_grad_norm and derives the noise; the accountant can be prv (the default), rdp or gdp, and get_epsilon(delta) reports the budget spent.

Threat model and limits

• What it bounds: how much the trained model's behaviour can depend on any one unit of privacy, which bounds membership inference and extraction for that unit.
• Unit of privacy: DP-SGD clips per example. If one person contributes many examples (emails, sessions, documents), record-level DP protects each record, not the person. NIST calls user-level privacy an excellent default and notes that bounding per-user contributions needs more noise for the same epsilon.
• Large epsilon: NIST cites a study suggesting an epsilon of 0.1 generally gives strong protection, describes values below 1 as reasonable, notes many deployments use values between 1 and 20, and warns that large values may not provide meaningful privacy, citing leakage from crafted training data at epsilon 10.
• Steps outside the accountant: NIST Section 4.3 notes that choosing hyperparameters by measuring accuracy on the private data is not differentially private. Non-private pre-processing, evaluation on private data and earlier non-DP fine-tuning all sit outside the guarantee.
• What it does not cover: data placed in a prompt or retrieval index at inference time. See the GenAI Data Security Risks and Mitigations Guide.

Utility loss

NIST states that DP training with current techniques typically lowers accuracy, sometimes significantly, and works best for simple models and very large training datasets; small groups within the data get worse accuracy. It also notes that large models pre-trained on public data show strong privacy-utility trade-offs when fine-tuned with DP-SGD, and that this improves with model size, while cautioning that the "public" data must itself be checked for sensitive content.

Epsilon budgets vs k-anonymity limits

GDPR status of each output

Under GDPR Recital 26, information is anonymous only if a person is no longer identifiable by any means reasonably likely to be used, considering cost, time and available technology; pseudonymised data remains personal data. The Article 29 Working Party also notes that a controller running DP queries still holds the original data, so for that controller the outputs can remain personal data. For models, the EDPB's Opinion 28/2024 (adopted 18 December 2024) says AI models trained on personal data cannot always be considered anonymous and that a model is anonymous only if the likelihood of direct, including probabilistic, extraction of personal data and the likelihood of obtaining it through queries are both insignificant.

The EDPB adds that if a supervisory authority cannot confirm effective anonymisation from the documentation, the controller may have failed its accountability duty under Article 5(2) (para 57). For lawful basis and wider obligations see the AI and GDPR Guide. This section summarises regulatory guidance and is not legal advice.

Choosing by data type and release model

Small vs large organisations: smaller teams without DP expertise get most value from minimisation, scrubbing, keeping models internal and running extraction tests. Larger teams releasing models externally should standardise a DP fine-tuning pipeline with a fixed accountant and a documented epsilon policy per release class.

Worked scenario

A support platform wants to fine-tune a language model on three years of customer tickets so it can draft replies, and offer it to enterprise customers through an API. Tickets contain names, account numbers, addresses and occasionally pasted credentials.

1. Anonymisation only: the team scrubs named entities and account numbers with a recogniser and calls the set anonymised. Testing with prefix prompts recovers a handful of addresses written in unusual formats and one pasted API key. The anonymisation claim does not hold, and the deployed model would fail the EDPB query-likelihood test.
2. Add DP fine-tuning: the team keeps scrubbing, adds secret scanning, and fine-tunes a public pre-trained model with DP-SGD, grouping tickets so the unit is the customer account rather than the single ticket. They set a target epsilon before training and tune hyperparameters on a separate, non-sensitive ticket sample to keep tuning outside the private data.
3. Measure the cost: draft quality drops on rare product lines. The team accepts this for the external API and keeps a non-DP model for internal agents only, behind access controls.
4. Test and document: before release they run membership inference and extraction tests, record epsilon, delta, unit, accountant and test results, and repeat tests on every retrain.

How to implement

1. Define the release model first: internal only, API, customer-hosted or public weights. The exposure drives the required strength.
2. Minimise and scrub: remove fields the model does not need, scrub direct identifiers and run secret scanning on every source. Record scrubber recall on a labelled sample.
3. Choose the unit of privacy: person or account where one individual has many records; document why if you choose a weaker unit.
4. Set an epsilon policy per release class before training, with delta and accountant fixed, and never adjust epsilon after seeing accuracy.
5. Prefer public pre-training plus DP fine-tuning for language and vision models, and check the public base data for sensitive content.
6. Pilot: train a DP and a non-DP model on the same split, compare accuracy and leakage tests, and decide with both numbers in front of the data owner.
7. Rollback: keep the previous model version deployable; if post-release tests show regurgitation, withdraw the model and retrain rather than adding output filters alone.

Detection and measurement

The EDPB lists structured testing against attribute and membership inference, exfiltration, regurgitation, model inversion and reconstruction as evidence for anonymity claims (para 55). NIST notes that auditing can find bugs but cannot prove a DP guarantee, so tests complement the accountant rather than replace it.

For a testing programme, see the AML.T0137 Training Data Information from AI Models guide.

Common failure patterns

• Calling a scrubbed or pseudonymised training set anonymous and treating the model as out of GDPR scope without testing it.
• Record-level DP on person-level data: under the standard group privacy bound, a user with 500 emails can be exposed up to 500 times the per-record epsilon.
• Quoting epsilon without context: no unit, no delta, no accountant, or an epsilon chosen after looking at accuracy.
• Non-private steps around a DP core: hyperparameter search on private data or an earlier non-DP fine-tune on the same data.
• Trusting public data: pre-training corpora contain personal data and live secrets, as the Common Crawl analysis showed.
• Testing once: leakage changes with every retrain, model size and data refresh; Carlini and colleagues found larger models more vulnerable.

Framework mapping

Practitioner checklist

• Record the release model and attacker access (API, weights, internal) for every model trained on personal data.
• Scrub direct identifiers and scan for secrets before training, and measure scrubber recall.
• Do not describe pseudonymised or scrubbed data as anonymous without re-identification testing.
• Use DP-SGD at user or account level for models exposed outside the organisation where accuracy allows.
• Fix epsilon, delta and accountant before training and keep hyperparameter tuning off private data.
• Prefer a public pre-trained base with DP fine-tuning, and check the base corpus for sensitive content.
• Run membership inference and extraction tests on every retrain and keep the results with the model card.
• Document the evidence an anonymity claim rests on, following the EDPB Opinion 28/2024 criteria.

Standards and references

• NIST: SP 800-226 Guidelines for Evaluating Differential Privacy Guarantees
• EDPB: Opinion 28/2024 on data protection aspects related to AI models
• Article 29 Working Party: Opinion 05/2014 on Anonymisation Techniques
• GDPR: Recital 26, Not applicable to anonymous data
• MITRE ATLAS: Training Data Information from AI Models
• Abadi et al.: Deep Learning with Differential Privacy
• Lukas et al.: Analyzing Leakage of Personally Identifiable Information in Language Models
• Carlini et al.: Extracting Training Data from Large Language Models

Related NHI Mgmt Group resources: Privacy-Enhancing Technologies Guide · Differential Privacy Guide · OWASP LLM Top 10 LLM02 Sensitive Information Disclosure Guide · AI Training Data Provenance and Copyright Exposure Guide