Membership Inference Attacks: How Models Leak Training Data and How to Test

A membership inference attack tries to tell whether a specific record was part of a machine learning model's training data, using only the model's outputs. If the answer can be inferred reliably, the model itself discloses something about the people in its training set, even when the data was never published. This matters wherever membership is sensitive: a clinical model, a fraud model trained on known victims, or a fine-tuned assistant built on customer tickets. This guide explains, for ML engineers, AI security teams and privacy officers, how the attack works at a conceptual level, how to measure a model's exposure before release, which defences actually reduce it, and which common beliefs about it are wrong. The broader legal questions about AI and personal data are covered in the AI and Privacy Guide; this page goes deeper on the attack and on testing.

Key takeaways

• Memorisation is the root cause. Models behave measurably differently on records they were trained on, and that gap is what membership inference exploits; overfitting is sufficient for it but not required.
• Query access is enough. NIST AI 100-2 E2025 notes that membership inference can be mounted by an attacker with only query access to a deployed model, and MITRE ATLAS catalogues it as AML.T0024.000 Infer Training Data Membership.
• Measure at low false-positive rates. Average accuracy hides risk; current research evaluates attacks by true-positive rate at very low false-positive rates, because confidently identifying even a few members is a privacy failure.
• Differential privacy is the only defence with a guarantee. Output rounding, label-only APIs and rate limits raise attacker cost; DP training bounds what any attack can learn, at a cost in accuracy.
• Test before release and keep evidence. Run an audit with an open tool, record the results against an agreed threshold, and attach them to the model card and the DPIA.

What it is

Membership inference is a privacy attack on a trained model. The adversary holds a candidate record (a patient profile, a sentence, an image) and asks one yes or no question: was this record in the training set? NIST's taxonomy of adversarial machine learning (NIST AI 100-2 E2025, section 2.4.2) classes it as a data privacy attack, alongside data reconstruction, property inference and, for generative models, training data extraction. MITRE ATLAS lists it as a sub-technique of Exfiltration via AI Inference API, noting that some strategies use a shadow model while others use statistics of the model's prediction scores.

The yes or no answer can be sensitive on its own. The Privacy Meter project's documentation gives the standard example: if a record is known to be in an HIV patient dataset, the person is known to be an HIV patient. The original 2016 paper by Shokri and colleagues showed the attack against classification models trained on commercial machine learning as a service platforms from Google and Amazon, and against a model trained on hospital discharge data.

Where it sits among model privacy attacks

Model extraction often comes first: a stolen or replicated proxy model makes later membership inference easier. See the Model Extraction and Distillation Defence Guide.

How it works

Training pushes a model to fit the examples it sees. The Privacy Meter documentation describes memorisation as the model's outputs on training data ("IN" data) having a different distribution from its outputs on unseen ("OUT") data; the less those distributions overlap, the more the model leaks. Yeom and colleagues (2017) showed formally that overfitting is sufficient for an attacker to perform membership inference, and also that it is not necessary, so a well-generalised model can still leak about individual records, particularly outliers.

Attacks differ in what signal they use and how they calibrate it:

• Score thresholding: the attacker compares the model's confidence or loss on the candidate record with a threshold. Training members tend to get higher confidence and lower loss.
• Shadow models: the attacker trains models on similar data where membership is known, learns what member and non-member outputs look like, and applies that to the target. ATLAS notes a shadow model can come from Train Proxy via Replication (AML.T0005.001).
• Per-example calibration: some records are easy for any model, so a raw high score is not proof of membership. The Likelihood Ratio Attack (LiRA) by Carlini and colleagues compares the target's behaviour on a record with the distribution expected when the record is in versus out of training. The authors report it is about ten times more powerful at low false-positive rates than earlier attacks. Robust Membership Inference (RMIA), implemented in Privacy Meter, follows the same calibrated approach with fewer reference models.
• Label-only and generative variants: attacks can work from the predicted label alone, and for language models from per-token likelihoods or how readily the model completes a passage.

How to test a model for membership leakage

1. Define the threat model. Record what an attacker can see (label only, top-k scores, full probabilities, token log-probabilities, model weights) and how many queries they can make. Test the access you actually expose, and the access an insider or a leaked model file would give.
2. Build member and non-member sets. Hold out records from the same distribution and time period as the training data. Non-members from a different period or source create false "success" through distribution shift.
3. Insert canaries where you control training. Unique synthetic records inserted at known rates show how strongly the model memorises rare items and support differential privacy auditing.
4. Run calibrated attacks with an audit tool. Open-source Privacy Meter (MIT licence) supports membership inference, range membership inference, dataset usage cardinality inference and auditing of differential privacy lower bounds from configuration files. Use reference models where budget allows.
5. Report the right metrics. Report true-positive rate at fixed low false-positive rates (for example 0.1% and 1%), alongside ROC AUC, broken down by subgroup and for outliers.
6. Gate and record. Compare results with the release threshold agreed with the privacy officer, document them in the model card and DPIA, and repeat on every retrain or fine-tune.

Why it matters

A model that leaks membership turns its own API into a disclosure channel. The consequences are practical:

• Personal data inside the model: if individuals can be singled out from a model's behaviour, privacy regulators may treat the model, not only its training set, as involving personal data. Membership inference test results are evidence in that assessment. See the DPIA Guide.
• Sensitive inferences: membership of a disease, fraud-victim, criminal justice or minority dataset is special category or high-risk information in itself.
• Proof of unauthorised use: the same techniques let data owners test whether their content was used to train a model, which matters for consent and licensing disputes.
• Stepping stone: NIST lists membership inference among privacy compromises for both predictive and generative AI; leaks often combine with extraction and inversion.
• Weak links in the access path: unrestricted API keys or service accounts with unlimited query rights make high-volume probing cheap. Treat model endpoints as resources with scoped, monitored machine identities.

Defences and their trade-offs

NIST SP 800-226 (March 2025) gives guidelines for evaluating differential privacy claims, including the "privacy hazards" that arise when the theory is implemented. See the Differential Privacy Guide.

Common misconceptions

• "Our model generalises well, so it cannot leak." Average generalisation says little about individual outliers, which are the records most at risk.
• "The attack is about 60% accurate, so it is weak." Carlini and colleagues argue that average-case accuracy does not show whether an attack can confidently identify any member; evaluate by true-positive rate at false-positive rates as low as 0.1%.
• "Large language models are immune." Duan and colleagues (2024) found membership inference barely beat random guessing for most settings on models up to 12 billion parameters trained on the Pile, and that some apparent successes came from distribution shift. That is a finding about pre-training scale, not about fine-tuning on small private datasets, and training data extraction against GPT-2 (Carlini et al., 2020) recovered verbatim sequences that appeared in only one document.
• "Rounding confidence scores fixes it." Label-only attacks remain; obfuscation raises cost but provides no guarantee.
• "Anonymised training data means no risk." If the attacker already holds the record, the question is membership, not identity; anonymisation of the training copy does not answer it.
• "One test at launch is enough." Every retrain, fine-tune or new data source changes memorisation and needs a fresh test.

Related concepts

• Differential privacy: a definition that bounds membership inference by design. See the Differential Privacy Guide.
• Federated learning: keeps raw data distributed, but model updates and the final model can still leak membership. See Federated Learning Privacy.
• Synthetic data: generators trained on real records can leak membership too. See the Synthetic Data Risk Guide.
• Sensitive information disclosure in LLM apps: see the OWASP LLM Top 10 LLM02 Sensitive Information Disclosure Guide.
• AI red teaming: include privacy attacks in scope. See the AI Red Teaming Guide.

Metrics to track: true-positive rate at 0.1% false-positive rate per release; share of models trained on personal data with a recorded membership audit (target 100%); audited epsilon lower bound versus claimed epsilon for DP models; accuracy cost of the chosen defence; and the number of API clients exceeding query thresholds each month.

Practitioner checklist

• Inventory models trained or fine-tuned on personal or confidential data and record what outputs each exposes.
• Define the attacker's access for each model and test against that access, not a weaker one.
• Build matched member and non-member sets from the same distribution and period, and insert canaries where you control training.
• Run calibrated membership inference audits and report true-positive rate at 0.1% and 1% false-positive rates.
• Agree a release threshold with the privacy officer and record results in the model card and DPIA.
• Use DP training for high-sensitivity data, document epsilon and delta, and audit the claim empirically.
• Expose only the output detail consumers need, and bind API access to scoped identities with rate limits.
• Log per-client query volume and patterns and alert on systematic probing.
• Re-test after every retrain, fine-tune or data source change.

Standards and references

• NIST: AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
• MITRE ATLAS: Infer Training Data Membership
• OWASP: Top 10 for LLM Applications 2025
• NIST: SP 800-226, Guidelines for Evaluating Differential Privacy Guarantees
• Shokri et al.: Membership Inference Attacks against Machine Learning Models
• Carlini et al.: Membership Inference Attacks From First Principles
• Duan et al.: Do Membership Inference Attacks Work on Large Language Models?
• Privacy Meter: open-source privacy auditing library

Related NHI Mgmt Group resources: AI and Privacy Guide · Differential Privacy Guide · Model Extraction and Distillation Defence Guide · AI Threats and Model Security Guide