Differential Privacy vs Synthetic Data: Formal Guarantees vs Realistic Substitutes

Teams reach for synthetic data to feed AI training and test pipelines without exposing real people, but a realistic synthetic dataset carries no privacy guarantee unless the generator was built to provide one. Differential privacy is a mathematical property of a computation; synthetic data is an output format. The two are often confused, sold as alternatives, or assumed to be the same thing, and the gap between them is where membership inference and re-identification succeed. This guide compares plain synthetic data with differentially private (DP) synthetic generators and with DP applied directly to model training, focusing on AI training and test data, membership inference risk, utility trade-offs and how to decide. It assumes the basics of epsilon and DP-SGD from the Differential Privacy Guide and the wider synthetic data risks in the Synthetic Data Risk Guide, and is written for privacy engineers, ML engineers, data protection officers and security architects.

Key takeaways

• They answer different questions. Differential privacy bounds how much any one person's record can change what is released; synthetic data replaces real records with generated ones. Only the first is a guarantee.
• Plain synthetic data must be tested, not trusted. The ICO says synthetic data is not automatically anonymous and that some generation methods are vulnerable to membership inference, model inversion and attribute disclosure.
• A DP generator passes its guarantee to every record it emits. Because of the post-processing property, synthetic data sampled from a generator trained with differential privacy inherits that generator's epsilon, however many records are produced.
• The cost lands on the rare records. Formal protection blurs outliers first, which is exactly where fraud, rare disease and edge-case test data live. Some uses have no acceptable balance.
• Choose by release model and use. Functional test data needs neither; internal analytics may accept tested plain synthesis in a secure setting; external sharing and training data that leaves your control should use a DP generator or DP training with a documented budget.

The short answer

Differential privacy is a property of an algorithm: its output distribution changes by at most a bounded amount (set by epsilon, and delta where used) when any one individual's data is added or removed. It says nothing about the format of the output, which can be a count, a trained model, or a synthetic table. Synthetic data is artificial data produced by a model fitted to real data. The ICO describes it as generated by data synthesis algorithms that reproduce the characteristics of the real data, and warns that the more closely synthetic data mimics real data, the more useful it is but the more likely it is to reveal personal information.

So the real comparison is three-way: plain synthetic data (no formal guarantee), DP synthetic data (a generator built under a privacy budget), and DP applied to the downstream task directly, such as DP-SGD training on real data. Each has a different threat model and a different place in an AI data pipeline.

Plain synthetic data in depth

Threat model: the generator is trained on real records and can memorise them, especially rare ones. An attacker who receives the synthetic dataset (or can query the generator) tries to tell whether a target was in the source data, to reconstruct attributes of a known person, or to link a synthetic record to a real one through a rare combination of quasi-identifiers. The ICO lists membership inference, model inversion and attribute disclosure as the risks some synthesis methods are vulnerable to.

Architecture in words: real data is extracted to a controlled environment; a generator (a GAN, variational autoencoder, Gaussian copula, Bayesian network or, for text, a prompted language model) is fitted; samples are drawn; fidelity and privacy are tested; the dataset is released to its consumers. The generator itself is a sensitive asset: it encodes the source data and can be more revealing than any sample.

Failure modes:

• Near copies: overfitted generators emit records almost identical to real ones, often the outliers that are easiest to re-identify.
• Weak attacks used as evidence: a privacy score built from average-case attack accuracy can look reassuring while a stronger attack identifies specific members. Carlini and colleagues showed that most earlier membership inference attacks perform poorly when measured by true positive rate at very low false positive rates (below 0.1%), the measure that matters for confident identification.
• LLM-generated text that quotes training data: prompting a model fine-tuned on support tickets or clinical notes to "generate realistic examples" can reproduce fragments of the originals.

Controls: remove direct identifiers and cap or suppress outliers before fitting where the use does not need them (the ICO suggests suppressing outliers); store the generator with the same controls as the source data; run singling out, linkability and inference tests on every release; keep a hold-out control set so tests separate genuine leakage from general utility. Anonymeter, an open source framework for tabular synthetic data, runs exactly these three attacks, each with a main attack, a control attack on records not used for generation and a naive baseline, and its repository reports that technical experts at the CNIL reviewed the method positively while leaving the acceptability decision with the controller.

Differentially private synthetic data in depth

Threat model: the same attacker, but now the generator is trained by a mechanism with a proven bound on any individual's influence. Because differential privacy is closed under post-processing, anything computed from the generator, including every synthetic record and every model trained on them, carries the same bound without spending more budget. The residual risk moves from "what did the generator memorise?" to "is epsilon small enough, is the unit of privacy right, and is the implementation correct?"

Architecture in words, marginal-based: the most mature tabular approach selects a set of low-dimensional marginals (counts over small groups of columns), measures them with calibrated noise, then generates records consistent with the noisy marginals. McKenna, Miklau and Sheldon describe this three-step pattern for MST, using Private-PGM to estimate the full distribution; their NIST-MST mechanism won the 2018 NIST differential privacy synthetic data competition. AIM and MWEM follow related designs.

Architecture in words, deep generators: DP-CTGAN trains a tabular GAN with differentially private gradient updates; PATE-GAN and PATE-CTGAN train a set of teacher discriminators on disjoint partitions and release only noisy aggregated votes to the generator. They are aimed at mixed-type tables where choosing marginals by hand is impractical; compare them against a marginal method on your own data before choosing. The OpenDP SmartNoise synthesizer library implements MWEM, MST, QUAIL, DP-CTGAN, PATE-CTGAN, PATE-GAN and AIM, and notes that MWEM and MST need categorical columns, with continuous columns discretised in a way that does not reveal the data distribution.

Failure modes:

• Budget spent outside the mechanism: choosing bin edges, clipping bounds or category lists by looking at the real data leaks information that the epsilon does not count.
• Wrong unit of privacy: protecting each row when one person has many rows (transactions, visits, messages) gives each person far weaker protection than the stated epsilon.
• Epsilon chosen for accuracy: the budget is tuned upwards until the data "looks right", which turns a guarantee into a label.
• Tails vanish: the ICO notes that DP with synthetic data can reduce utility and make the data's characteristics less predictable, and that for uses such as fraud detection that depend on outliers an acceptable balance may not be achievable.

Controls: fix epsilon, delta and the unit of privacy in a documented policy before fitting; use public or domain knowledge, not the data, for schemas and bins; review the implementation against NIST SP 800-226, which sets out a differential privacy pyramid of factors and the common privacy hazards in real deployments; and get an external expert to set the budget if you lack the expertise, as the ICO advises.

DP on the task instead of synthetic data

When the goal is one model rather than a reusable dataset, training directly on real data with DP-SGD often gives better accuracy under the same guarantee, because the budget is spent only on what that model needs. Abadi and colleagues showed in 2016 that deep networks with non-convex objectives can be trained under a modest privacy budget at manageable cost to model quality. The trade-off is reuse: every new model or retraining run on the same individuals spends more budget, and teams without the data cannot experiment freely. A DP synthetic dataset spends the budget once and can then be shared with any number of teams. The mechanics of DP-SGD, clipping and accountants are in the Differential Privacy Guide; the comparison with pre-training anonymisation is in Differential Privacy vs Anonymisation for AI Training Data.

Membership inference: what each approach actually stops

Membership inference, catalogued in MITRE ATLAS as AML.T0137.000 Infer Membership, asks whether a specific record was in the training data. Against synthetic data it takes two forms: inferring membership from the released dataset alone, and querying a generator or a model trained on the data. Attribute inference and reconstruction sit alongside it under AML.T0137.002 Invert AI Model. The attack side is covered in the Membership Inference Attacks guide; the table below covers what each approach does about it.

DP does not prevent population-level inference: if a group shares a trait, the synthetic data will show it too. And a passed test on plain synthetic data is evidence about the attacks you ran, not a bound, so record each attack and its configuration with the release.

Utility trade-offs for AI training and test data

Measure utility the way the data will be used. For training data, compare a model trained on synthetic data and tested on held-back real data with the same model trained on real data, and break results down by subgroup so lost tails are visible. NIST's Collaborative Research Cycle publishes benchmark excerpts from the American Community Survey and the SDNist report generator, which produces more than ten privacy, fidelity and utility metrics for deidentified data, a useful baseline for comparing generators before you commit to one.

Worked scenario

A bank wants to give an external model vendor data to build a transaction categorisation model, and give its own QA team test data for a new payments service. The source holds many transactions per customer.

1. QA test data: the payments team needs valid account formats and edge-case amounts, not real behaviour. A rule-based generator built from the schema covers this with no real records, so no privacy assessment of the data is needed beyond confirming no real values were copied.
2. Vendor training data: the data leaves the bank's control, so plain synthetic data would rest on empirical tests alone. The privacy team fixes the unit of privacy as the customer, not the transaction, because one customer has many rows; caps contributions per customer; and sets epsilon in the release policy before fitting.
3. Generator choice: the relevant columns are mostly categorical (merchant category, channel, amount band, region), so a marginal-based DP generator fits. Amount bands and category lists come from public scheme definitions, not from the data.
4. Testing: the team runs train-synthetic-test-real on a hold-out and an attack evaluation with a control set, recording the results. Categorisation accuracy holds for common merchants; rare merchant categories degrade, which the vendor is told.
5. Fraud features: a request to include fraud labels is refused for the synthetic release because outliers would be lost; that work happens on real data inside the bank's secure environment.

Is synthetic data anonymous under GDPR?

Not by default. GDPR Recital 26 treats data as anonymous only where a person is no longer identifiable, taking account of all means reasonably likely to be used. The ICO applies this to synthetic data: whether it is anonymous depends on whether the personal information used to model it can be inferred from the synthetic data. Fitting a generator to real records is itself processing of personal data, so a lawful basis and usually a DPIA are needed. A DP generator with a small epsilon and the right unit of privacy is strong evidence for an anonymity assessment, but the assessment still has to consider singling out, linkability and inference, the three criteria Anonymeter tests. Plain synthetic data needs that assessment with attack evidence for every release. See the Anonymisation and Re-identification Risk Guide and the AI and GDPR Guide.

How to implement

1. Classify the use: functional test, internal analysis, external sharing or model training. Record it on the request.
2. Pick the approach from the utility table: start with schema-only data where it suffices; escalate to plain or DP synthesis only when realism is needed.
3. Write the privacy policy first: unit of privacy, epsilon, delta, contribution caps and who approves exceptions. Do not tune after seeing results.
4. Prepare inputs without leaking: derive schemas, bins and clipping bounds from public knowledge or a separate budget.
5. Pilot: fit two candidate generators on a benchmark such as the NIST CRC excerpts and on your data, compare utility and attack results.
6. Test every release: utility by intended use and subgroup; attack evaluation with a control set at low false positive rates; distance to closest real record for plain synthesis.
7. Release with a record: generator, version, configuration, epsilon, test results, recipient and expiry. Label every record as synthetic so it cannot be mistaken for real data later.
8. Roll back: if a later test or new attack shows leakage, withdraw the release, notify recipients under the sharing agreement, and refit with a lower epsilon or stronger preprocessing.

Detection and measurement

• Attack advantage at low false positive rate: true positive rate of the strongest membership inference attack at 0.1% false positives, per release; should be close to the control.
• Cumulative epsilon per dataset: total budget spent on each source dataset across all releases and training runs, against the policy limit.
• Utility gap: accuracy of train-synthetic-test-real against train-real-test-real, overall and for the smallest subgroup.
• Release coverage: share of synthetic releases with a recorded epsilon or attack test; target 100%.
• Generator exposure: number of identities with access to trained generators, and any access outside the approved group.

Common failure patterns

• "Synthetic means anonymous": plain synthetic data shared externally with no attack testing or DPIA.
• DP label without DP inputs: a DP generator fed bins and category lists computed from the real data.
• Per-row protection for per-person data: stated epsilon is per transaction, so heavy users are barely protected.
• Raising epsilon until it looks right: the budget becomes a tuning knob rather than a policy.
• Publishing the generator: sharing the trained model with the dataset, exposing whatever it memorised.

Framework mapping

Decision checklist

• Does the use need realistic behaviour, or only valid formats? If formats, use schema-only data.
• Will the data leave your control? If yes, use a DP generator or do not release.
• Is one person represented by many rows? Set the unit of privacy to the person.
• Do outliers matter to the use? If yes, consider real data in a secure environment instead.
• Is only one model needed? Compare DP-SGD on real data with DP synthesis.
• Can you show attack results at low false positive rates for this release?

Practitioner checklist

• Treat plain synthetic data as personal data until an identifiability assessment says otherwise.
• Fix unit of privacy, epsilon and delta in policy before fitting any DP generator.
• Derive schemas, bins and bounds from public knowledge, not the source data.
• Test every release for singling out, linkability and inference with a control set.
• Report membership inference results as true positive rate at low false positive rates.
• Protect trained generators as sensitive data and never publish them with the dataset.
• Track cumulative epsilon per source dataset across releases and training runs.

Standards and references

• NIST: SP 800-226 Guidelines for Evaluating Differential Privacy Guarantees
• ICO: Privacy-enhancing technologies, Synthetic data
• NIST: Privacy Engineering Collaborative Research Cycle
• NIST: SP 800-53 Rev. 5
• OpenDP: SmartNoise Synthesizers
• McKenna, Miklau and Sheldon: Winning the NIST Contest, a scalable and general approach to differentially private synthetic data
• Carlini et al.: Membership Inference Attacks From First Principles
• Anonymeter: unified framework for quantifying privacy risk in synthetic data

Related NHI Mgmt Group resources: Differential Privacy Guide · Synthetic Data Risk Guide · Differential Privacy vs Anonymisation for AI Training Data · Privacy-Enhancing Technologies Guide