Model Extraction vs Inversion vs Membership Inference: What Each Attack Steals

Model extraction, model inversion and membership inference all work through the same prediction API, but they steal different things, and a control that stops one can leave the others wide open. Extraction copies the model's behaviour, inversion reconstructs sensitive attributes or representative examples of the people in the training data, and membership inference reveals whether a specific record was used at all. The last two are privacy attacks: under data protection law they decide whether a trained model still contains personal data. This guide compares the three, then goes deep on inversion and membership inference: what each needs, how each fails, and how differential privacy, output rounding, rate limits and monitoring change the odds. Extraction defence has its own guide and is summarised here only for contrast. It is written for ML engineers, privacy engineers and security architects who publish or host models trained on sensitive data.

Key takeaways

• Each attack steals a different asset. Extraction takes the model's function (intellectual property), inversion takes attributes or likenesses of training subjects, and membership inference takes the fact that a person's record was in the dataset.
• Membership itself can be the harm. The ICO notes that membership inference may not reveal further personal data, yet being in a dataset of patients with a particular condition is sensitive on its own.
• Output fidelity fuels all three. Confidence scores, full probability vectors, token log probabilities and embeddings give attackers signal; MITRE ATLAS AML.M0002 recommends returning only what the application needs.
• Only differential privacy gives a bound. Rounding, label-only outputs, quotas and monitoring raise attacker cost; DP training limits how much any single record can influence the model, at a measurable cost in accuracy.
• Regulators expect test evidence. EDPB Opinion 28/2024 lists membership inference, model inversion, attribute inference, regurgitation and reconstruction among the attacks to consider when claiming a model is anonymous.

Short answer: what each attack steals

MITRE ATLAS (release 2026.10) groups the two privacy attacks under AML.T0137 Training Data Information from AI Models, with sub-techniques AML.T0137.000 Infer Membership, AML.T0137.001 Infer Properties, AML.T0137.002 Invert AI Model and AML.T0137.003 Extract Memorized Data. Extraction sits elsewhere, as AML.T0005.001 Replicate Behavior from Model Outputs, with AML.T0048.004 AI Intellectual Property Theft as the usual impact. All of them can use AML.T0024 Exfiltration via AI Inference API: the response channel is the theft channel.

For extraction and distillation controls in depth, see the Model Extraction and Distillation Defence Guide. For membership inference mechanics and audit tools, see Membership Inference Attacks. This page focuses on how the privacy attacks differ from each other and from extraction, and how to choose defences.

Threat model: model inversion

ATLAS describes inversion as inferring sensitive aggregate attributes or reconstructing representative examples of training data by analysing a model's behaviour, outputs or exposed representations. It adds an important limit: the results do not by themselves establish that a reconstructed example was in the training set. Inversion produces something that looks like the training subject, not proof of who was used.

• Asset stolen: a sensitive attribute the attacker did not know (for example a medical value or a demographic trait), or an approximate likeness associated with a class or identity. The ICO frames it as an attacker who already holds some personal data using the model's inputs and outputs to infer further personal information about the same individuals.
• Access needed: query access with scores is often enough; white-box access to weights, gradients or embeddings makes it easier. OWASP ML03:2023 describes access either through an implementation flaw or by querying the model's API.
• Threat actor: someone targeting specific individuals whose partial records they already hold, or a researcher demonstrating a privacy failure that becomes a regulatory problem.
• Most exposed models: classifiers whose classes map to individuals, such as face or voice recognition; models trained on small cohorts with sensitive features; and embedding services, where OWASP LLM08:2025 lists embedding inversion attacks against vector stores.

The OWASP Machine Learning Security Top 10 rates inversion detectability as difficult: queries look like normal use, and the output is analysed offline.

Threat model: membership inference

ATLAS describes membership inference as inferring whether a particular sample was included in a model's training data, using prediction scores, token probabilities, loss values or generated responses, often with shadow models or auxiliary datasets to separate members from non-members.

• Asset stolen: a single bit per record, which is harmful when membership is sensitive. The ICO gives a model trained on patients with dementia or HIV as an example of serious risk.
• Access needed: the candidate record and query access. The ICO explains the signal: for someone in the training data, the model will be disproportionately confident in a prediction about that person.
• Threat actor: anyone who already holds a record about a person and wants to learn a sensitive association; also auditors and regulators, who use the same technique to test anonymity claims.
• Most exposed models: overfitted models, models trained on small or rare cohorts, and fine-tuned language models where records seen once can still be memorised. Carlini and colleagues recovered hundreds of verbatim sequences from GPT-2's training data, some from documents that appeared only once, and found larger models more vulnerable. That memorisation is what ATLAS separates out as AML.T0137.003 Extract Memorized Data.

Why extraction is different

Extraction targets the model owner's investment, not the data subjects. Its controls are commercial and identity controls: verified accounts, quotas, correlation of related accounts, contract terms and watermarking of outputs. Those controls help privacy only indirectly. A volume quota that stops a distillation campaign may not stop membership inference at all, because confirming one person's membership can take a handful of queries. Conversely, DP training does little against extraction: a DP model can still be copied, it just leaks less about individuals. Treat extraction and the privacy attacks as separate risk register entries with separate owners: product and legal for extraction, the DPO and ML engineering for inversion and membership. The AI Model Theft Response Playbook covers response to extraction.

Failure modes that enable the privacy attacks

Controls compared: what each one actually buys

ATLAS maps the same core mitigations to both privacy sub-techniques: output obfuscation (AML.M0002), query volume limits (AML.M0004), model validation (AML.M0008), telemetry (AML.M0024), red teaming (AML.M0035), adaptive response degradation (AML.M0040) and privacy-preserving training and inference (AML.M0041). Their value differs sharply by attack.

Differential privacy in practice

DP is a property of the training mechanism, not of the published model, so the claim must come with the parameters that produced it. Abadi and colleagues showed in 2016 that deep networks can be trained under a modest privacy budget at a manageable cost in training efficiency and model quality. ATLAS AML.M0041 describes the same approach: calibrated noise added to clipped gradients during training. Libraries such as Opacus for PyTorch expose the two knobs directly as max_grad_norm (the clipping bound) and noise_multiplier. NIST SP 800-226, Guidelines for Evaluating Differential Privacy Guarantees (March 2025), sets out the factors to weigh and the common implementation hazards. For the parameter choices and the unit of privacy, see the Differential Privacy Guide.

• Unit of privacy: decide whether the protected unit is a row, a document or a person. A person with many records needs person-level contribution bounding, or the guarantee covers far less than it appears to.
• Fine-tuning: applying DP only to fine-tuning protects the fine-tuning data, not whatever the base model memorised in pre-training.
• Utility check: evaluate accuracy per subgroup, because DP noise falls hardest on rare classes, which are often the sensitive ones.

Output rounding and minimisation in practice

• Classification APIs: return the label, or the top few labels, with scores rounded to one or two decimals or to coarse bands such as high, medium and low. ATLAS AML.M0002 suggests restricting the number of results, limiting the specificity of the output class ontology and withholding or reducing the precision of confidence scores, logits and embeddings.
• Generative APIs: disable log probabilities for external callers unless a contract and use case require them, and never expose them on models fine-tuned on personal data.
• Embedding APIs: treat embeddings derived from personal data as personal data; keep them server-side and return similarity results rather than vectors where possible.
• Limit of the control: rounding removes signal but does not remove the confidence gap that membership inference relies on, so it complements DP and never replaces it.

Decision criteria: which defences for which model

1. Is membership itself sensitive? If being in the dataset reveals health, finance, criminal justice or similar status, plan for DP training and a membership audit before release, whatever the API exposure.
2. Who gets the model? Published weights or on-device models give white-box access; only training-time defences such as DP and data minimisation still apply. A private API keeps output and volume controls available.
3. Do classes map to individuals? Face, voice and per-user models invite inversion; restrict outputs to decisions rather than scores and consider whether the model needs per-person classes at all.
4. Do you claim the model is anonymous? If so, EDPB Opinion 28/2024 expects the likelihood of extracting personal data, directly or through queries, to be insignificant, and documentation including test reports (paras 43, 55 and 58). Build the test evidence first. See the Anonymisation and Re-identification Risk Guide.
5. Is extraction the bigger business risk? If the model is commercially valuable but trained on non-personal data, prioritise extraction controls and accept lighter privacy controls.

Small and large organisations: a small team publishing one sensitive model should prefer a narrow API with label-only outputs and an off-the-shelf DP library over building custom monitoring. A large provider with many tenants needs per-caller telemetry, adaptive degradation (ATLAS AML.M0040) and a standing privacy red team.

Worked scenario: a hospital readmission model

A hospital group trains a gradient-boosted readmission risk model on discharge records from a specialist clinic and plans to offer it to partner GP practices through an API that returns a probability to four decimal places. A partner could submit a known patient's details and observe an unusually confident score, which would suggest the patient was in the clinic's training data and therefore attended a specialist service. With partial demographics, varying one unknown field and watching the score could also point to the most likely value of a sensitive attribute.

The privacy review changes four things. The API returns a three-band risk category instead of a probability. The model is retrained with DP and the accuracy loss per age band is accepted by the clinical lead after review. A membership inference audit, run against held-out members and non-members, is attached to the DPIA with the agreed threshold. Each practice gets its own credential with a daily quota and alerts on repeated near-identical queries. Extraction is noted but ranked lower: the model has little commercial value outside the group.

Implementation and testing

1. Classify models: record for each model the training data sensitivity, whether membership is sensitive, who receives outputs or weights, and which output fields are exposed.
2. Minimise outputs first: it is the cheapest change. Pilot label-only or banded outputs with one consumer group, confirm the use case still works, then roll out. Roll back by version, not by toggling fields in production.
3. Add identity and quotas: one credential per consumer, per-caller quotas and logging of inputs and outputs in line with privacy requirements, which the NCSC secure AI guidelines also call for.
4. Run privacy evaluation before release: a membership inference audit at low false-positive rates and, for models whose classes map to people, an inversion test of whether sensitive attributes can be recovered. Record results against a threshold (ATLAS AML.M0008 and AML.M0035).
5. Introduce DP where the audit fails or membership is sensitive: train a DP variant, compare utility per subgroup, document epsilon, delta and the unit of privacy.
6. Monitor and respond: alert on query patterns that sweep one attribute while holding others fixed, repeated near-duplicate queries, and callers requesting maximum output detail.

Detection and measurement

• Signals: per-caller query volume, input similarity clusters, systematic variation of a single feature, high share of queries near decision boundaries, and requests for the most detailed output options.
• Metric 1: membership inference true-positive rate at a fixed low false-positive rate (for example 0.1%) for each released model, against the agreed threshold.
• Metric 2: share of external endpoints returning only labels, top-k or banded scores.
• Metric 3: share of models trained on sensitive data with a documented DP budget or a recorded decision not to use DP.
• Metric 4: share of external callers with individual credentials and quotas.
• Metric 5: time from a suspicious query alert to credential suspension or response degradation.

Framework mapping

Common failure patterns

• Treating privacy attacks as extraction: investing only in quotas and watermarking, which protect the model owner, while the data subjects stay exposed.
• Calling a model anonymous without tests: EDPB Opinion 28/2024 makes clear that models trained on personal data cannot in all cases be considered anonymous, and that testing only against widely known attacks shows resistance to those attacks only.
• Quoting epsilon without the unit of privacy: a record-level guarantee on person-level data protects far less than the number suggests.
• Exposing debug detail in production: full probability vectors and log probabilities left on after development.
• Measuring audit accuracy instead of low false-positive performance: an average-case score hides the confident identification of a few individuals, which is the real privacy failure.

Practitioner checklist

• Record for every model which of extraction, inversion and membership inference matter, and who owns each risk.
• Return labels, top-k or banded scores externally; disable log probabilities on models trained on personal data.
• Keep embeddings and gradients server-side; treat embeddings of personal data as personal data.
• Issue one credential and quota per consumer and log inputs and outputs within privacy limits.
• Run a membership inference audit, and an inversion test where classes map to people, before every release.
• Use DP training where membership is sensitive or the audit fails; document epsilon, delta and unit of privacy.
• Attach privacy test results to the model card and the DPIA, and repeat them on retraining.
• Link extraction controls to the extraction defence programme rather than counting them as privacy controls.

Standards and references

• MITRE ATLAS: Training Data Information from AI Models
• NIST: AI 100-2 E2025 Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
• NIST: SP 800-226 Guidelines for Evaluating Differential Privacy Guarantees
• EDPB: Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models
• ICO: How should we assess security and data minimisation in AI?
• OWASP: ML03:2023 Model Inversion Attack
• OWASP: ML04:2023 Membership Inference Attack
• Abadi et al.: Deep Learning with Differential Privacy

Related NHI Mgmt Group resources: Membership Inference Attacks · Model Extraction and Distillation Defence Guide · Differential Privacy Guide · AI Model Security Guide · AI and Privacy Guide