A model trained on private records can reveal those records, or the fact that someone was in them, to anyone who can query it. MITRE ATLAS records this as AML.T0137 Training Data Information from AI Models, a Collection (AML.TA0009) technique added in ATLAS 2026.10 (released 28 October 2026) and rated Demonstrated. The adversary analyses a model's outputs or behaviour to learn about the data used to train or fine-tune it, exposing private associations, personal data or proprietary content. ATLAS splits it into four sub-techniques: Infer Membership, Infer Properties, Invert AI Model and Extract Memorized Data. This guide is for ML engineers, privacy teams and AI platform owners who train or fine-tune models on data that matters. It explains each sub-technique at a defender's level, the published research ATLAS cites, how to test your own models, and the mitigations ATLAS maps. Membership inference has its own deep page: see Membership Inference Attacks.
Key takeaways
- Four sub-techniques, four questions. Was this record used (.000), what does the dataset look like (.001), what do members of a class look like (.002), and what exact content was memorised (.003).
- Memorisation is real in production. Research ATLAS cites recovered gigabytes of training data from open, semi-open and closed language models including ChatGPT, and over 1,000 training images from diffusion models.
- Alignment does not remove it. The production-model study found a divergence attack made ChatGPT emit training data at 150 times its normal rate.
- Privacy-preserving training is the root control. ATLAS maps AML.M0041, including differential privacy, to the technique and all four sub-techniques.
- Test before you ship. ATLAS maps AML.M0008 Validate AI Model and AML.M0035 AI Red Team: measure membership disclosure and extraction with controlled samples, not assumptions.
What the technique is
ATLAS defines the technique as adversaries obtaining sensitive information about the data used to train or fine-tune an AI model by analysing its outputs or behaviour. The input is usually material gathered through AML.T0135 Outputs from AI Models or direct inference API access. It is a privacy attack rather than a theft of the model itself; the related step of copying the model's behaviour is covered in the Model Extraction and Distillation Defence Guide.
| Sub-technique | Maturity | Platforms | What the adversary learns | Signals ATLAS lists |
|---|---|---|---|---|
| AML.T0137.000 Infer Membership | Feasible | Predictive, Generative, Agentic | Whether a given record, passage, image or audio sample was in the training set | Prediction scores, token probabilities, loss values, generated responses; statistical tests, shadow models, auxiliary data |
| AML.T0137.001 Infer Properties | Feasible | Predictive, Generative, Agentic | Global dataset properties: provenance, population make-up, class proportions, domain coverage, collection period | Output distributions compared or modelled, auxiliary datasets, contrast with models of known properties |
| AML.T0137.002 Invert AI Model | Feasible | Predictive | Sensitive attributes or representative examples linked to an identity or class | Prediction scores, probabilities, embeddings or gradients where exposed |
| AML.T0137.003 Extract Memorized Data | Demonstrated | Predictive, Generative, Agentic | Verbatim or near-duplicate training content: text, code, images, personal data | Partial passages, contextual cues, repeated varied inputs, generate-and-filter |
ATLAS draws two useful lines. Infer Properties targets a property of the dataset, not any individual record. Invert AI Model results do not by themselves prove a reconstructed example was in the training set; establishing that is membership inference.
The research behind it
ATLAS lists no incident case study for T0137 as of release 2026.10. Its sub-technique pages cite published research, which shows each method working against real systems.
- Membership inference against ML-as-a-service (Shokri et al., 2016): the authors trained an inference model to tell apart how a target responds to training inputs versus other inputs, and evaluated it against classification models built on commercial services from Google and Amazon, including a hospital discharge dataset where membership itself is sensitive.
- Extracting training data from GPT-2 (Carlini et al., 2020): querying the model recovered hundreds of verbatim sequences, including names, phone numbers and email addresses, IRC conversations, code and 128-bit UUIDs, each present in only one training document. Larger models were more vulnerable.
- Scalable extraction from production models (Nasr et al., 2023): gigabytes of training data were extracted from Pythia, GPT-Neo, LLaMA, Falcon and ChatGPT. A divergence attack moved ChatGPT out of its chat behaviour and raised the rate of training data output 150 times; the authors conclude alignment does not eliminate memorisation.
- Extracting images from diffusion models (Carlini et al., 2023): a generate-and-filter pipeline recovered over 1,000 training examples from models including Stable Diffusion, from photos of private individuals to trademarked logos, and the authors found diffusion models considerably less private than GANs.
Who is exposed: any organisation that fine-tunes a model on customer records, support tickets, source code, medical or financial data and then exposes it to people who should not see that data. A model fine-tuned on one tenant's data and served to another is a direct path to this technique.
Mitigations ATLAS maps
| Mitigation | Sub-techniques mapped | What ATLAS says to do |
|---|---|---|
| AML.M0041 Privacy-Preserving AI Training and Inference | T0137 and .000 to .003 | Limit how much any individual record influences outputs, for example differential privacy applied to clipped gradients in training or to outputs; avoid overfitting |
| AML.M0040 Adaptive AI Response Degradation | T0137 and .000 to .003 | Reduce precision or detail of responses during suspected privacy attacks |
| AML.M0008 Validate AI Model | .000, .002, .003 | Measure membership disclosure with controlled member and non-member samples; test reconstruction and extraction; repeat after remediation |
| AML.M0035 AI Red Team | .000, .002 | Try to reveal membership of known samples and reconstruct sensitive records |
| AML.M0002 Predictive AI Output Obfuscation | .000, .002 | Fewer results, less specific class ontology, randomised smoothing, lower numeric precision |
| AML.M0004 Limit AI Service Query Volume and Rate | .000, .002 | Cap queries per period to limit what can be learned |
| AML.M0024 AI Telemetry Logging and Monitoring | .000, .002 | Log queries to spot privacy attacks and scope exposure |
| AML.M0020 Generative AI Guardrails | .003 | Inspect outputs for sensitive training content, match against protected references, block or redact |
NIST SP 800-226, Guidelines for Evaluating Differential Privacy Guarantees (final, March 2025), is the reference for judging whether a differential privacy claim is meaningful, including the privacy hazards that appear in implementations. The Differential Privacy Guide covers the mechanics.
How to implement: a privacy testing programme
- Classify training data: for every model, record what personal, confidential or proprietary data went into pre-training, fine-tuning and retrieval stores. A model inherits the classification of its most sensitive training data.
- Minimise before training: remove or mask direct identifiers, secrets and credentials, and deduplicate. The GPT-2 study shows that even content appearing in a single training document can be recovered.
- Choose the training approach by risk: differential privacy or other AML.M0041 methods for models trained on regulated personal data; regularisation and early stopping at minimum for everything else. Record the privacy budget where differential privacy is used.
- Hold out canaries: insert unique synthetic records before training and keep a matching non-member set, so membership and extraction tests have ground truth.
- Test before release (AML.M0008): run membership inference against members and non-members, and extraction attempts using partial prefixes of known records. Set a pass threshold agreed with the privacy team and block release if it fails.
- Minimise outputs (AML.M0002): return labels instead of score vectors, round probabilities, drop logprobs, and keep embeddings internal unless a use case needs them.
- Guard generative outputs (AML.M0020): scan responses for personal data and matches against protected reference text before display, in report-only mode first, then blocking.
- Monitor and re-test: log queries (AML.M0024), apply rate limits (AML.M0004), and repeat validation after every fine-tune. If a disclosure is confirmed, roll back to the previous model version while you retrain.
Decision guidance: where the training data could not lawfully be shown to the model's users, treat the model as able to leak it and either restrict access to people entitled to the data or train with formal privacy guarantees. Where the data is public or synthetic, output minimisation and rate limits are usually proportionate. Small teams fine-tuning on internal data often get most of the benefit from steps 1, 2, 5 and 6.
Detection and measurement
| Signal | Sub-technique | Where to see it |
|---|---|---|
| Many queries with near-identical inputs and small variations | .000, .002 | Inference gateway logs |
| Requests for maximum-precision scores or full probability vectors | .000, .002 | API parameters |
| Prompts made of partial records, repeated tokens or long verbatim prefixes | .003 | Prompt classifiers |
| Outputs matching protected reference text or containing personal data | .003 | Output guardrail logs |
| Systematic probing across demographic or class categories | .001, .002 | Prompt and input analysis |
- Membership advantage on canaries: how much better than chance a test attacker separates members from non-members, per release.
- Canary extraction rate: share of inserted canaries recovered by extraction tests.
- Guardrail hits: outputs blocked or redacted for training-data matches, per million responses.
- Models with documented training-data classification: coverage across the model inventory.
- Re-test compliance: share of fine-tunes validated before release.
Common failure patterns
- Fine-tuning on raw tickets or chat logs: personal data and credentials pasted by customers become training content, then output.
- Shared models across tenants: a model fine-tuned on one customer's data serves others, turning membership and extraction into cross-tenant disclosure.
- Exposing full score vectors: the exact signal membership inference and inversion rely on, returned by default.
- Trusting alignment as a privacy control: the production extraction study shows an aligned chat model still emitted training data under a divergence attack.
- Testing once: each fine-tune changes memorisation, so a pass at first release says nothing about the next version.
- Overfitting left unchecked: OWASP ML04:2023 and ATLAS AML.M0041 both point to regularisation, because models that overfit separate members from non-members more clearly.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| MITRE ATLAS | AML.T0137, .000 to .003 (AML.TA0009 Collection) | The technique and sub-techniques |
| MITRE ATLAS mitigations | AML.M0041, AML.M0040, AML.M0008, AML.M0035, AML.M0002, AML.M0004, AML.M0024, AML.M0020 | All mapped by ATLAS 2026.10 |
| OWASP Machine Learning Security Top 10 2023 | ML03:2023 Model Inversion Attack; ML04:2023 Membership Inference Attack | The same attack classes with prevention measures |
| OWASP Top 10 for LLM Applications 2025 | LLM02:2025 Sensitive Information Disclosure | Training data revealed through output; data sanitisation and differential privacy |
| NIST SP 800-53 Rev. 5 | PT-3, SI-12, SI-15, RA-3 | Processing purposes for personal data, information retention, output filtering, risk assessment |
| NIST CSF 2.0 | PR.DS-01, ID.RA-01, DE.CM-09 | Data protection, vulnerability identification, monitoring |
Practitioner checklist
- Record the sensitivity of every model's training and fine-tuning data.
- Strip identifiers, secrets and duplicates before training.
- Use differential privacy or equivalent for models trained on regulated personal data.
- Insert canaries and run membership and extraction tests before each release.
- Return labels, not score vectors, and drop logprobs unless required.
- Scan generative outputs for personal data and protected reference matches.
- Never serve a model fine-tuned on one tenant's data to another tenant.
- Re-validate after every fine-tune and keep the previous version ready for rollback.
Standards and references
- MITRE ATLAS: Training Data Information from AI Models
- MITRE ATLAS: Privacy-Preserving AI Training and Inference
- NIST: SP 800-226 Guidelines for Evaluating Differential Privacy Guarantees
- OWASP: ML04:2023 Membership Inference Attack
- OWASP: ML03:2023 Model Inversion Attack
- Shokri et al.: Membership Inference Attacks against Machine Learning Models
- Nasr et al.: Scalable Extraction of Training Data from (Production) Language Models
- Carlini et al.: Extracting Training Data from Diffusion Models
Related NHI Mgmt Group resources: AML.T0144 Recover AI Model Internals · MITRE ATLAS Guide · Membership Inference Attacks · Differential Privacy Guide · OWASP LLM Top 10 LLM02 Sensitive Information Disclosure Guide · MITRE ATLAS Mitigations Guide