Synthetic Data Risk Guide: Privacy, Quality, Collapse

Synthetic data is often adopted as a privacy fix and a cheap source of training data, but it can leak the records it was generated from, distort the patterns that matter, and degrade models that are trained on it repeatedly. Teams use it to share test data, to train fraud and health models, to augment fine-tuning sets and, increasingly, to have one model generate training data for another. Each use carries a different risk, and the evidence from regulators and peer-reviewed research is more cautious than vendor claims. This guide covers the privacy, quality, contamination and security risks across the synthetic data lifecycle, how to validate a synthetic dataset before release, and how to govern synthetic data in AI pipelines. It is written for data scientists, ML engineers, privacy engineers and security architects. It does not repeat the general treatment of anonymisation and differential privacy in the Privacy-Enhancing Technologies Guide; it focuses on what goes wrong with synthetic data specifically.

Key takeaways

Synthetic is not automatically anonymous. The ICO says anonymity depends on whether the personal information the generator was trained on can be inferred from the synthetic output, and generation itself is processing of personal data.
The privacy and utility trade-off is real and unpredictable. Peer-reviewed work at USENIX Security 2022 found synthetic data either did not prevent inference attacks or did not retain utility, and offered no better trade-off than traditional anonymisation.
Outliers carry most of the risk and most of the value. Rare records are the easiest to re-identify and the first to vanish under privacy protection or recursive training.
Recursive training causes model collapse. Research published in Nature in 2024 showed models trained on generations of model output lose the tails of the distribution first, and that keeping original data limits the damage.
Label and track synthetic data. Provenance that marks every record as real or synthetic, and which generator made it, is the control behind privacy review, quality testing and collapse prevention.

What synthetic data is and how it is used

The ICO describes synthetic data as artificial data generated by data synthesis algorithms, built from real data with a model trained to reproduce its characteristics. It distinguishes partially synthetic data, where only some variables are synthesised, from fully synthetic data, where all are. NIST SP 800-188, De-Identifying Government Datasets, lists synthetic data generation as one de-identification technique and publishing synthetic data based on identified data as one data-sharing model.

Use case	Main risk	What to check first
Test and development data replacing production copies	Records close to real customers leak into lower-security environments	Distance to the closest real record; whether rare combinations survive
Data sharing with partners or researchers	Re-identification and attribute inference by the recipient	Motivated intruder test; contractual controls; secure setting
Training tabular models (fraud, credit, health)	Loss of rare but important patterns; bias carried from source data	Train on synthetic, test on real; subgroup performance
LLM-generated fine-tuning or evaluation data	Errors, bias and style of the generator copied into the student model; evaluation contamination	Human review sample; deduplication against test sets
Recursive training on web data that includes model output	Model collapse over generations	Share of original human data retained; provenance labels
Bias testing and correction	Synthetic groups do not reflect real ones	Comparison with real subgroup statistics where lawful

Privacy risks: why synthetic data can still be personal data

The generator is trained on real records, so the question is what it memorised. The ICO's guidance on privacy-enhancing technologies states that some synthesis methods are vulnerable to model inversion attacks, membership inference attacks and attribute disclosure risk, and that re-identification risk assessment for synthetic data is still an area of development.

Membership inference: an attacker determines whether a particular person's record was in the training data. MITRE ATLAS catalogues this as Infer Training Data Membership (AML.T0024.000).
Model inversion and attribute inference: an attacker reconstructs features of training records, or infers a sensitive attribute of a known person from the synthetic data. ATLAS lists Invert AI Model (AML.T0024.001).
Singling out and linkage: the ICO's anonymisation guidance names singling out, linkability and inference as the identifiability risks to assess. A synthetic record that reproduces a rare combination of age, postcode and diagnosis can be linked to a real person even if no field was copied.
Outliers: the ICO suggests suppressing outliers or combining synthesis with differential privacy to protect records with uniquely identifying features.

Stadler, Oprisanu and Troncoso ("Synthetic Data: Anonymisation Groundhog Day", USENIX Security 2022) evaluated a range of generative models and found the privacy gain varied widely and the utility loss could not be predicted, because it was unclear which signals a synthetic dataset would keep. The practical consequence: treat synthetic data as personal data until a documented identifiability assessment says otherwise, and repeat that assessment when the generator, source data or recipients change. The ICO's anonymisation guidance requires the motivated intruder test as part of that assessment and says assessments should be reviewed when new attacks appear or new datasets are released.

Differential privacy: adding a formal privacy guarantee to the generator reduces leakage, but the ICO notes it can reduce utility and introduce unpredictability, and that in some cases, such as fraud detection where outliers matter, no acceptable balance is achievable. Where data stays in a trusted research environment, the ICO suggests less stringent noise may be acceptable.

Quality and bias risks

A synthetic dataset is a model's view of the source data, not the data itself. Common quality problems:

Lost tails: rare events (fraud, rare diseases, edge-case inputs) are under-represented or removed, so models trained on synthetic data look good on average and fail where it matters.
Bias carried through: the ICO notes that biases in the source data carry through to the synthetic data, and recommends diverse, representative source data and bias monitoring, especially where the data supports decisions about people.
Spurious correlations: the generator can invent relationships that do not exist, or break real ones between columns.
Generator errors in text: LLM-generated examples copy the generator's factual errors and style. Fine-tuning on them amplifies the generator's mistakes.
Evaluation contamination: synthetic test items generated from the same model family or from leaked benchmark content give inflated scores.

For high-risk AI systems, Article 10(3) of the EU AI Act requires training, validation and testing data sets to be relevant, sufficiently representative and, to the best extent possible, free of errors and complete in view of the intended purpose. Synthetic data used for such systems has to meet the same criteria, and Article 10(2) requires documenting data origin and preparation operations. Article 4a, inserted by the Digital Omnibus amendment, allows exceptional processing of special category data for bias detection only where that cannot be done effectively with other data, including synthetic or anonymised data.

Model collapse and synthetic data contamination

Shumailov and colleagues ("AI models collapse when trained on recursively generated data", Nature, 2024) describe model collapse as a degenerative process across generations of generative models, where generated data pollutes the training set of the next generation. In early collapse the model loses information about the tails of the distribution; in late collapse it converges to a distribution with little resemblance to the original and much lower variance. In their language model experiment, fine-tuning OPT-125m on wikitext2 over successive generations with no original data retained raised perplexity from about 20 to about 28, while keeping 10% of the original data in each generation limited the loss to a minor degradation.

The authors conclude that access to genuine human-generated data matters for tasks where the tails of the distribution are important, and that because model-generated content is hard to track at scale, the community needs coordination on data provenance. For an organisation this means:

Mark synthetic records at creation with a flag, the generator name and version, and the prompt or configuration, so they can be filtered out or weighted later.
Keep a protected store of original data and always mix a defined share of it into retraining; never replace real data wholesale with synthetic data.
Watch your own feedback loops: chatbot transcripts, generated documentation and agent outputs that flow back into data lakes become training data without anyone deciding so.
Treat third-party corpora as possibly contaminated with model output, and ask suppliers how they detect it.

Security risks in the synthetic data pipeline

Poisoning through the generator: if an attacker can influence the seed data, prompts or generator, every synthetic record inherits the change. OWASP LLM04 Data and Model Poisoning recommends tracking data origins and transformations and using data version control. See the AI Data Poisoning and Backdoors Guide.
Model extraction through synthetic data: OWASP LLM10 Unbounded Consumption describes attackers using a target model's API to generate synthetic training data and fine-tune a functionally equivalent model. Rate limits and terms on your own APIs are part of the defence.
The generator as a sensitive asset: a generator trained on production data can be more revealing than any single synthetic sample. Store it with the same controls as the source data, and do not publish it with the dataset.
Generator supply chain: third-party synthesis tools and hosted generators receive your source data. Assess them as processors and record them in your AI-BOM. See the AI Supply Chain and AI-BOM Guide.
Access credentials: generation pipelines usually run as service accounts with read access to production data. Scope them to the source tables needed and log their reads. See the AI Security Posture Management (AI-SPM) Guide.

Choosing an approach

Approach	Privacy	Utility	Choose when
Rule-based or schema-only fake data	Strong, no source records used	Low for analytics; fine for functional testing	Software testing that needs valid formats, not real patterns
Partially synthetic data	Weaker; real values remain in unsynthesised columns	High	Controlled research access with strong contracts and a secure setting
Fully synthetic data without formal guarantee	Unpredictable; must be tested	Medium to high	Internal use after identifiability testing, not open publication
Fully synthetic data with differential privacy	Formal guarantee bounded by the privacy budget	Lower, especially for rare events	External sharing or publication of aggregate patterns
Real data in a trusted research environment	Depends on environment controls	Full	Where outliers matter and no acceptable synthetic trade-off exists

Small and large organisations: a small team without privacy engineering expertise should prefer schema-only data for testing and a secure environment for analytics, and take the ICO's advice to consult an external expert before setting a differential privacy budget. Large organisations with repeated releases should set up a review body, in the way NIST SP 800-188 describes a Disclosure Review Board, to approve each synthetic release.

How to implement

Define the purpose and the recipient. Testing, internal analytics, external sharing and model training need different assurance levels. Record which one applies.
Complete a data protection impact assessment for generation, since the generator processes personal data, and record the lawful basis.
Prepare the source data. Remove direct identifiers, cap or suppress outliers where they are not needed, and document the version used.
Generate in a controlled environment with the generator, configuration and random seed recorded, and the generator stored as sensitive.
Test privacy. Run membership inference and attribute inference tests, distance to closest record checks and a motivated intruder review. Open tools exist, such as Anonymeter, which the ICO points to for tabular synthetic data.
Test utility. Compare marginal and joint distributions, train on synthetic and test on real data, and check subgroup and rare-event performance. NIST's Collaborative Research Cycle publishes benchmark data and an evaluation suite (SDNist) with univariate, multivariate, propensity and privacy metrics that can serve as a model.
Approve and label the release. Attach a data card stating synthetic status, generator, privacy parameters, test results and permitted uses.
Roll back when needed. If later testing or a new attack shows leakage, withdraw the dataset, notify recipients, and retrain any models built on it from the provenance record.

Detection and measurement

Log: generator runs with source data version, service account reads of source tables, release approvals, and every downstream training job that consumes synthetic data.
Alert on: synthetic datasets in training buckets without a provenance flag, and generators or source extracts copied outside the controlled environment.

Metrics:

Share of synthetic releases with a documented privacy and utility test before release (target 100%).
Membership inference advantage over random guessing on each release.
Gap in key model metrics between training on real and training on synthetic data, including rare-event recall.
Share of records in each training set labelled real or synthetic, and the minimum share of original data kept in retraining.
Number of synthetic datasets in use whose identifiability assessment is older than the review interval.

Common failure patterns

Calling it anonymous because it is synthetic. Releases go out without an identifiability assessment, against the ICO's view that anonymity depends on what can be inferred.
Publishing the generator with the data. The generator is a model of the source data and can be attacked directly.
Testing only average fidelity. Aggregate statistics look right while rare events and minority subgroups have gone.
Replacing real data in retraining. Each generation trained on the previous one's output loses more of the distribution.
Unlabelled synthetic records. Once mixed into a data lake they cannot be removed, weighted or excluded from evaluation.
Synthetic test data copied from production with light changes. Test environments end up holding near-copies of customer records with weaker controls.

Framework mapping

Framework	Reference	Relevance
NIST SP 800-53 Rev. 5	SI-19 De-identification, SI-19(5) Statistical Disclosure Control, SI-19(6) Differential Privacy, SI-19(7) Validated Algorithms and Software, SI-19(8) Motivated Intruder	De-identification controls that apply to synthetic releases
NIST SP 800-53 Rev. 5	PT-2, PT-3, PM-5(1), RA-3	Authority and purpose for processing personal data, PII inventory, risk assessment
NIST CSF 2.0	ID.AM-07, PR.DS-01, GV.OC-03	Data inventory and metadata, data protection, legal requirements
NIST SP 800-188	Synthetic data as technique and sharing model; Disclosure Review Board	Governance of de-identified and synthetic releases
MITRE ATLAS	AML.T0024.000, AML.T0024.001, AML.T0020; AML.M0025	Privacy attacks on models, poisoning, dataset provenance
OWASP LLM Top 10 2025	LLM04 Data and Model Poisoning, LLM10 Unbounded Consumption	Poisoned generation data; extraction via synthetic data
EU AI Act	Article 10(2) and 10(3), Article 4a	Data governance and quality criteria; bias detection with synthetic data first

Practitioner checklist

Record the purpose, recipient and assurance level for every synthetic dataset.
Run a DPIA for generation and store the generator under source data controls.
Test each release for membership inference, attribute inference and closeness to real records.
Test utility on real hold-out data, including rare events and subgroups.
Use differential privacy for external sharing, with an expert-set privacy budget.
Label every synthetic record with generator, version and configuration.
Keep a protected store of original data and a minimum real share in retraining.
Review identifiability assessments on a schedule and when attacks or recipients change.

Synthetic Data Risk Guide: Privacy, Quality and Model Collapse