A model behind an inference API can leak its training data, or its own behaviour, to anyone allowed to query it often enough, without a single file ever leaving your network. MITRE ATLAS tracks this as AML.T0024 Exfiltration via AI Inference API, in the Exfiltration tactic (AML.TA0010), with three sub-techniques: membership inference, model inversion and model extraction. This guide treats the technique as a detection and response problem: what the query traffic looks like, which ATLAS mitigations apply and how to tune them, how to run an investigation when you suspect extraction, and how API keys and accounts become the real control point. It is written for AI platform owners, API security teams and SOC analysts who run models that others can call. For the wider framework, see the MITRE ATLAS Guide.
Key takeaways
- The API is the exfiltration channel. AML.T0024 needs only legitimate query access (AML.T0040 AI Model Inference API Access), so the attack looks like usage, not like a breach.
- Three goals, different signals. Membership inference and inversion probe for private training data; extraction (AML.T0024.002) harvests outputs at scale to train a copy. Volume, repetition and output fidelity tell them apart.
- Accounts and keys are the control point. The best documented case, the 2026 distillation campaigns reported by Anthropic, used about 24,000 fraudulent accounts. Per-account limits fail if account creation is cheap.
- ATLAS links four mitigations. AML.M0002 Passive AI Output Obfuscation, AML.M0004 Restrict Number of AI Model Queries, AML.M0019 Control Access to AI Models and Data in Production and AML.M0024 AI Telemetry Logging. We add AML.M0012 Encrypt Sensitive Information for the model weights themselves.
- Response is about accounts, not hosts. Containment means suspending clusters of linked accounts and keys, preserving query logs, and deciding whether the training data exposure is a personal data breach.
How the technique works
ATLAS describes AML.T0024 as adversaries exfiltrating private information through inference API access. Models have been shown to leak information about their training data, and the model itself can be extracted for AI intellectual property theft (AML.T0048.004). ATLAS rates the parent technique as realised, meaning it has been seen in real incidents. The three sub-techniques differ in what leaves and how.
| Sub-technique | What the adversary obtains | Typical query pattern | ATLAS maturity |
|---|---|---|---|
| AML.T0024.000 Infer Training Data Membership | Whether a specific record (for example a patient or customer) was in the training set, or global properties of the data | Many queries around a small set of candidate records, often compared against a shadow model (AML.T0005.001 Train Proxy via Replication); heavy reliance on confidence scores | Feasible |
| AML.T0024.001 Invert AI Model | Reconstructed sensitive features of training data | Iterative, optimisation-style queries that adjust inputs and watch confidence scores move | Feasible |
| AML.T0024.002 Extract AI Model | A functional copy of the model, trained offline on harvested outputs | High volume, broad or narrowly targeted capability coverage, highly repetitive prompt templates, requests for step-by-step reasoning | Realised |
For generative models the extraction case is usually called distillation: the target's outputs become labelled training data for a student model. The OWASP Top 10 for LLM Applications 2025 lists model extraction via API and functional model replication under LLM10 Unbounded Consumption. Privacy attacks on training data are covered in the Membership Inference Attacks guide, and the business side of extraction in the Model Extraction and Distillation Defence Guide. This page stays with the ATLAS technique as an operational problem.
Where it shows up
- Frontier and commercial LLM APIs: ATLAS case study AML.CS0056 records the distillation campaigns Anthropic reported in February 2026. Anthropic attributed about 24,000 fraudulent accounts and over 16 million exchanges to three labs, targeting capabilities such as agentic reasoning, tool use and coding. Access came through commercial proxy services and networks of accounts, one of which managed over 20,000 accounts at once and mixed distillation traffic with ordinary requests.
- Classifier and scoring APIs: fraud scores, credit decisions, malware verdicts and content moderation endpoints often return a probability. Confidence scores are exactly what membership inference and inversion need.
- Models fine-tuned on private data: a support assistant tuned on tickets, or a clinical model tuned on records, carries training data risk that the base model did not. The inference API is the only path an outsider needs.
- Internal model endpoints: models published inside the company for many teams are often reachable with a shared key and no per-caller limits. An insider or a stolen workload key can run the same harvest.
- Agent and tool endpoints: an agent that wraps a model and exposes it through a chat or MCP interface is still an inference API for this technique, and often has looser limits than the provider API behind it.
Impact
| Impact | Who carries it | Why it matters |
|---|---|---|
| Intellectual property loss | Model owner | A distilled copy captures capability built with expensive training and data; ATLAS links extraction to AML.T0048.004 AI Intellectual Property Theft |
| Privacy breach of training subjects | Data controller | Confirmed membership or reconstructed features can be personal data; ATLAS notes private training data may include personally identifiable information |
| Loss of safeguards | Society, model owner | Anthropic notes that distilled models lack the safeguards that stop misuse of the original |
| Attack staging | Model owner and users | An offline copy lets adversaries craft evasion or jailbreak inputs without being seen, then replay them against the real model |
| Cost | Model owner | ATLAS notes extraction can be used simply to avoid paying per query in an AI-as-a-service setting |
Detection
ATLAS mitigation AML.M0024 AI Telemetry Logging asks for logging of model inputs and outputs, and states that telemetry can help identify whether sensitive data has been exfiltrated. In practice detection is behavioural analytics over API logs, joined to account and billing data.
What to log
- Caller identity: account ID, organisation, API key ID, OAuth client, source IP and autonomous system, user agent, and payment instrument fingerprint where you bill.
- Request shape: prompt or input hash, template similarity cluster, token counts in and out, parameters such as temperature and whether log probabilities or top-k scores were requested.
- Output metadata: output length, refusal flags, and a content classifier label for the capability area (code, reasoning, tool use, a given domain).
- Account lifecycle: creation date, verification level, plan, and links to other accounts through shared payment, device or network attributes.
Signals that point to AML.T0024
| Signal | Points to | Notes |
|---|---|---|
| Synchronised traffic across many accounts with identical patterns, shared payment methods and coordinated timing | Extraction through account farms | All three signals were among those Anthropic reported using |
| Massive volume concentrated in a narrow capability area with highly repetitive prompt structures | Targeted distillation | Cluster prompts by template, then look at volume per template |
| Prompts that ask for step-by-step reasoning or grading rubrics at scale | Collecting reasoning traces as training data | Named by Anthropic as a distillation pattern |
| Traffic shifting to a new model within hours of release | Organised extraction programme | Anthropic observed one campaign redirect nearly half its traffic within 24 hours |
| Repeated near-identical inputs with small perturbations and requests for full probability vectors | Membership inference or inversion | Most visible on classifier APIs that return scores |
| Systematic coverage of the input space (grid-like or random sampling) rather than task-shaped queries | Extraction of a smaller model | Compare input distribution to known customer traffic |
Map these analytics in MITRE D3FEND terms to D3-UBA User Behavior Analysis and D3-NTA Network Traffic Analysis. Single-account thresholds rarely fire on their own: the strongest detections join accounts into clusters first, then score the cluster.
Measuring it
- Attribution coverage: share of inference traffic made with keys bound to a verified owner.
- Cluster detection time: days from the first query of a confirmed extraction cluster to its suspension.
- Score exposure: number of endpoints returning full confidence vectors or log probabilities to unverified callers.
- Recurrence: share of suspended clusters that reappear through new accounts within 30 days.
Prevention and hardening
No control stops a determined adversary with enough accounts. The goal is to raise the cost per useful output and make campaigns visible early.
| ATLAS mitigation | What it means for an inference API | Trade-off |
|---|---|---|
| AML.M0019 Control Access to AI Models and Data in Production | Require authentication for every endpoint, bind keys to verified organisations, monitor queries against usage policy. Anthropic reported strengthening verification for educational, security research and startup accounts, which it said are commonly exploited. | Friction at sign-up; verification tiers need a fast path for real customers |
| AML.M0004 Restrict Number of AI Model Queries | Rate limits and quotas per key, per account, per organisation and per payment instrument; ATLAS states the aim is to regulate the amount and fidelity of information an attacker can learn | Per-key limits alone are defeated by many keys, so aggregate across linked accounts |
| AML.M0002 Passive AI Output Obfuscation | ATLAS suggests restricting the number of results shown, limiting output class specificity, randomised smoothing and reducing numeric precision. For LLMs: do not return log probabilities by default. | Some customers need scores; offer full precision only on verified tiers |
| AML.M0024 AI Telemetry Logging | Log inputs, outputs and caller identity with enough retention to reconstruct a campaign | Logging prompts and outputs is itself personal data processing; set retention and access deliberately |
| AML.M0012 Encrypt Sensitive Information | Protects weights at rest; relevant when T0024 is paired with direct theft. See the Securing AI Model Weights guide. | Does not affect API-based extraction |
- Training-side defences for privacy attacks: differential privacy during training and deduplication of training data reduce what membership inference can find. NIST AI 100-2 E2025 classifies these privacy attacks and mitigations; see the Adversarial ML Taxonomy Guide.
- Output watermarking: OWASP LLM10 lists watermarking to detect unauthorised use of outputs. Treat it as evidence for later, not prevention.
- Abuse-aware limits: general API rate limiting practice applies; see the Rate Limiting and API Abuse Guide.
- Usage terms: prohibit training competing models on outputs in your terms of service. Anthropic described the campaigns as violating its terms and regional access restrictions, which gives a basis for account action.
Where identity fits
Every query in AML.T0024 is made with a credential: an API key, an OAuth token or a session. That makes the inference API an identity problem as much as a model problem. Our recommendations (NHI Mgmt Group):
- One key per workload: issue a separate key to each customer application and each internal consumer, so volume can be attributed and one key can be revoked without collateral damage.
- Bind keys to verified owners: tie each key to an organisation and a human owner, with the verification tier recorded on the key.
- Short-lived credentials for internal callers: internal services should reach model endpoints with workload identity tokens rather than shared static keys, so a stolen key from a notebook or CI log does not become a harvesting tool.
- Watch for resale: proxy services that resell access spread one organisation's quota across many unknown users. Treat sudden diversity in prompts and source networks behind one key as a signal.
See the Anthropic API Keys Security Guide for the customer side of key handling with one provider.
Response
- Confirm and scope: build the account cluster (shared payment, device, network, timing, template), estimate total queries and the capability areas covered, and record the date range.
- Preserve evidence: snapshot query logs, account records and billing data for the cluster before retention jobs delete them, with hashes and access logged.
- Contain: suspend the account cluster and revoke its keys together, so the adversary cannot shift load to the survivors. Tighten verification on the sign-up path that was used.
- Assess data exposure: if the evidence shows membership inference or inversion against a model trained on personal data, involve the privacy team to decide whether this is a personal data breach under the applicable law.
- Adjust controls: lower precision or remove scores on the affected endpoint, add the campaign fingerprints to detection, and retest.
- Share indicators: Anthropic reported sharing technical indicators with other AI labs, cloud providers and relevant authorities. Sector sharing groups are the equivalent for most organisations.
Record the incident against AML.T0024 in your AI incident log; see the AI Incident Reporting Guide.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| MITRE ATLAS | AML.T0024, AML.T0024.000, AML.T0024.001, AML.T0024.002; enabling AML.T0040; outcome AML.T0048.004 | Technique, sub-techniques and related steps |
| MITRE ATLAS mitigations | AML.M0002, AML.M0004, AML.M0019, AML.M0024 | Mitigations ATLAS links to the technique |
| OWASP Top 10 for LLM Applications 2025 | LLM10 Unbounded Consumption; LLM02 Sensitive Information Disclosure | Model extraction via API and training data leakage |
| NIST SP 800-53 Rev. 5 | AC-6, SC-5, SI-4, AU-6, AU-12 | Least privilege for keys, resource limits, monitoring and log review |
| NIST CSF 2.0 | PR.DS-10, DE.CM-09, DE.AE-02, RS.MI-01 | Data in use, runtime monitoring, analysis, containment |
| MITRE D3FEND | D3-UBA, D3-NTA | Behavioural and traffic analytics |
See the OWASP LLM Top 10 LLM10 Unbounded Consumption Guide for the OWASP view.
Common failure patterns
- Limits per key only: quotas that never aggregate across linked accounts let a farm of thousands of small accounts stay under every threshold.
- Full confidence vectors by default: classifier APIs that return every class probability to every caller hand membership and inversion attacks their main input.
- No prompt retention: teams that drop prompts for privacy reasons and keep only token counts cannot cluster templates or reconstruct a campaign later. Keep hashed or classified prompts with controlled access instead.
- Treating it as fraud only: billing teams spot the volume, but nobody asks what capability was harvested or whether training data could leak.
- Shared internal keys: an internal endpoint reached with one static key per environment gives no attribution when it is copied into a notebook or a CI log.
Practitioner checklist
- Inventory every model inference endpoint, internal and external, with its authentication method and whether it returns scores or log probabilities.
- Require authentication on every endpoint and issue one key per workload, bound to a verified owner.
- Apply rate limits and quotas per key, account, organisation and payment instrument, with aggregation across linked accounts.
- Return the minimum output fidelity by default and offer full scores only on verified tiers.
- Log caller identity, prompt template cluster, token counts and output class with a retention period that covers a slow campaign.
- Build cluster analytics for synchronised timing, shared payment and repetitive templates, and review them weekly.
- Write an extraction response runbook: cluster, preserve, suspend together, assess privacy, adjust controls, share indicators.
- For models trained on personal data, record the privacy risk assessment and training-side mitigations such as differential privacy.
Standards and references
- MITRE ATLAS: Exfiltration via AI Inference API
- MITRE ATLAS: Extract AI Model
- MITRE ATLAS: Restrict Number of AI Model Queries
- MITRE ATLAS: Model Distillation Campaigns Targeting Anthropic Claude
- Anthropic: Detecting and preventing distillation attacks
- OWASP: LLM10:2025 Unbounded Consumption
- NIST: AI 100-2 E2025 Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
Related NHI Mgmt Group resources: Model Extraction and Distillation Defence Guide · Membership Inference Attacks · MITRE ATLAS Mitigations Guide · Rate Limiting and API Abuse Guide