Join our Newsletter — 33% off our NHI Course
Home› Guides› Model Extraction and Distillation Defence Guide: Detect and…
Guide Agent Threats & Attacks

Model Extraction and Distillation Defence Guide: Detect and limit query-based theft of your models

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 13 November 2026 13 min read
On this page

Anyone who can query a model can, with enough queries, build a copy of what it does without ever touching its weights. MITRE ATLAS calls this Replicate Behavior from Model Outputs (AML.T0005.001), and in February 2026 Anthropic reported distillation campaigns that used about 24,000 fraudulent accounts and more than 16 million exchanges to train competing models on Claude's outputs. The same economics apply to any organisation that exposes a fine-tuned classifier, a domain LLM or an agent through an API: a cheap copy undercuts the product, reveals what the model knows and gives an attacker an offline proxy to test evasion attacks against. This guide covers how extraction and distillation work, how to tell abusive querying from heavy legitimate use, which controls cost the attacker most, and how to respond. It is for AI platform owners, API product teams, fraud and trust and safety teams, and security engineers. Model attacks in general are covered in the AI Model Security Guide; this page goes deeper on query-based theft.

Key takeaways

  • Extraction is an abuse of legitimate access. The attacker uses AI Model Inference API Access (AML.T0040) as designed, so perimeter controls do not see it; detection lives in account, usage and content telemetry.
  • Distillation is account fraud at scale. The 2026 campaigns relied on proxy resellers and "hydra" networks of accounts that replace each banned account, so identity verification and multi-account correlation matter as much as rate limits.
  • Output fidelity is the attacker's fuel. Full probability vectors, token log probabilities and verbose reasoning traces make copies cheaper; return the minimum each customer needs (ATLAS AML.M0002 Predictive AI Output Obfuscation).
  • Quotas slow extraction but do not stop it. Restricting query volume (AML.M0004) raises cost; distributed campaigns stay under per-account limits, so pair quotas with cross-account analytics.
  • Contracts and evidence are part of the defence. Terms that prohibit training competing models, plus retained telemetry and output watermarks, give you grounds to suspend, litigate and share indicators.

How extraction and distillation work

ATLAS describes extraction as repeatedly querying the victim's inference API, collecting the outputs into a dataset and using them as labels to train a separate model that mimics the target. The same technique covers building a stand-in for staging further attacks, and a related technique, Infer Membership (AML.T0137.000), asks what the model was trained on. The research record shows a progression from simple models to frontier LLMs.

VariantWhat is copiedWhat the attacker needsPublic example
Classic model stealingParameters or decision boundary of a classifier or regression modelBlack-box queries, ideally with confidence scoresTramèr et al. (USENIX Security 2016) extracted logistic regression, neural network and decision tree models from BigML and Amazon Machine Learning, and showed that omitting confidence values does not prevent extraction
Partial architecture extractionSpecific layers or hidden dimension of a production LLMAPI features exposing token-level probabilitiesCarlini et al. (2024) recovered the embedding projection layer of OpenAI's Ada and Babbage models for under $20 and the hidden dimension of gpt-3.5-turbo
Capability distillationBehaviour on targeted tasks (coding, tool use, reasoning), learnt by a student modelLarge prompt sets (AML.T0065 LLM Prompt Crafting) and millions of completionsAnthropic's February 2026 report on three AI labs (ATLAS case study AML.CS0056)
Reward model abuseThe target's judgement, used to grade a student's outputs during reinforcement learningPrompts asking the model to score or rank answersAnthropic reported rubric-based grading among one lab's targets
Replication from public informationA functionally similar model rebuilt from papers and open artifactsPublished architecture, data descriptions and computeATLAS case study AML.CS0007, GPT-2 replication (2019)

OWASP's LLM10:2025 Unbounded Consumption entry lists both "model extraction via API" and functional model replication, where the target model generates synthetic training data for fine-tuning another model. The second is the one most organisations face: it needs no special API features, only volume and well-chosen prompts.

What is at stake and who is exposed

  • Frontier and foundation model providers: loss of capability lead, erosion of export control and safety policy (Anthropic noted distilled models lack the original safeguards), and cost harvesting through fraudulent accounts (AML.T0034).
  • Enterprises exposing fine-tuned models: a fraud score, underwriting model, code assistant or customer service model trained on proprietary data. The copy reveals the business logic and lets attackers probe evasion offline before attacking the real system.
  • Agent and application vendors: the prompts, retrieval corpus and tool orchestration around a model are also copyable through outputs, even if the underlying model is a third party's.
  • Customers of model providers: your own contract usually prohibits using outputs to train competing models. OpenAI's terms of use (effective 1 January 2026) bar using output "to develop models that compete with OpenAI" and automatically or programmatically extracting output. An internal team distilling a vendor model into an in-house one can put your organisation in breach.

ATLAS maps the end result to AI Intellectual Property Theft (AML.T0048.004). Treat a confirmed extraction like a data exfiltration incident against a trade secret: scope, contain, preserve evidence and decide on legal action.

Controls that raise the cost of extraction

ControlATLAS mitigationEffect on attackerCost to legitimate users
Strong account verification for high-volume tiers (business verification, payment instrument checks, enterprise contracts)AML.M0019 Control Access to AI Models and Data in ProductionMakes account farms expensive; Anthropic tightened verification for educational, research and startup pathwaysOnboarding friction for small customers
Per-account and per-organisation quotas, with burst limitsAML.M0004 Limit AI Service Query Volume and RateForces distribution across more accountsLow if tiers match real usage
Return labels instead of full probability vectors; restrict or round log probabilities; cap top-k alternativesAML.M0002 Predictive AI Output ObfuscationRemoves the cheapest extraction signal for classifiers and partial LLM extractionSome calibration use cases need scores: grant per contract
Summarise or withhold raw reasoning tracesAML.M0002Reduces chain-of-thought training data, a target in the 2026 campaignsDevelopers lose debugging detail
Serve models in the cloud rather than on devices; protect weightsAML.M0017 AI Model Distribution Methods, AML.M0005, AML.M0012Prevents the easier path of copying weights from an app or edge device; ATLAS case study AML.CS0058 describes researchers recovering TensorFlow Lite models from the Google Photos Android appLatency or offline use
Limit public technical detail about architecture and training dataAML.M0001 Limit Model Artifact ReleaseSlows replication from public informationTransparency obligations may require some disclosure
Output watermarking or fingerprintingNo dedicated ATLAS mitigation; listed by OWASP LLM10Does not prevent copying; helps prove a suspect model learnt from your outputsText watermarks may be weakened by paraphrasing
Contractual prohibition plus enforcementPolicyCreates grounds to terminate and pursue claimsNone for honest users

ATLAS itself notes that query limits may not protect against attacks that need few requests or that run against an offline model, and advises monitoring for attempts to evade limits through distributed requests, account rotation or stolen credentials. Stolen customer keys are a real path: treat inference API keys as privileged credentials (see the LLMjacking Guide).

Watermarking in practice: Kirchenbauer et al. describe a scheme that biases generation toward a pseudo-random "green list" of tokens and detects it with a statistical test that needs neither API access nor model parameters. For extraction defence its value is evidential, and its robustness against paraphrasing and further training of a student model is limited, so do not rely on it alone. Image and media provenance is a separate topic covered in the AI Model Provenance and Watermarking Guide.

Decision guidance

  • Public self-service API: highest exposure. Invest first in sign-up fraud controls, payment and device correlation, and output minimisation by default, with richer outputs unlocked by contract.
  • B2B API with known customers: rely on contracts, per-customer quotas and usage reviews; extraction more likely comes from a compromised customer key or a reseller, so watch for key sharing.
  • Internal model behind SSO: the threat is an insider or a compromised account exporting outputs in bulk; data loss prevention and query logging matter more than fraud scoring. See the Insider Data Theft Guide.
  • Small teams: start with authenticated access, quotas and logging on an API gateway; leave classifier-based distillation detection to later. Large providers need dedicated trust and safety tooling and intelligence sharing with peers and cloud platforms.

Detecting extraction and distillation

Single requests look benign. Anthropic noted that a prompt that is unremarkable alone becomes a signal when repeated tens of thousands of times across hundreds of accounts. Detection therefore aggregates across time, accounts and content. Log prompts, response metadata and account context in line with AML.M0024 AI Telemetry Logging, subject to your privacy commitments.

SignalWhat to measureWhy it indicates extraction
Capability concentrationShare of an account's traffic in a narrow task category (coding, tool use, grading)Distillation targets differentiated capabilities rather than mixed business use
Template repetitionPrompt structure similarity across requests and accountsGenerated prompt datasets reuse scaffolds with varied fillers
Training-shaped requestsRequests for step-by-step reasoning, multiple alternative answers, rubric scoresOutputs formatted as training labels or reward signals
Coordinated accountsShared payment methods, IP ranges, request metadata, synchronised timingHydra clusters spread load to stay under per-account limits
Rapid pivot after releasesTraffic shifting to a new model version within hoursAnthropic reported one campaign redirecting nearly half its traffic within 24 hours of a release
Coverage sweepsSystematic traversal of input space or topics; low revisit rateClassic extraction samples the decision boundary methodically
Reseller patternsOne account serving many end users or mixing unrelated trafficProxy services (AML.T0008.005) blended distillation with ordinary requests

Metrics that show the programme works: median time from campaign start to detection; share of high-volume accounts with verified identity; percentage of API responses returning full scores or log probabilities (aim for contract-only); number of correlated account clusters disrupted per quarter; and repeat rate (how quickly a disrupted cluster reappears).

Implementing an extraction defence programme

  1. Classify the models you expose. For each endpoint record the value of the model, who can call it, what output detail it returns and which contract governs use.
  2. Centralise access through a gateway. Authenticate every caller, attach organisation and account identifiers, and enforce quotas. The AI Gateway and LLM Gateway Guide covers the pattern.
  3. Minimise outputs by default. Remove probability vectors and log probabilities from default responses, then grant them per customer. Test the change with your top integrations in a pilot before switching defaults.
  4. Build the telemetry pipeline. Store per-request metadata, prompt fingerprints (hashes or embeddings of structure rather than raw text if privacy requires) and account graph attributes.
  5. Write detections in monitor mode first. Score accounts against the signals above for four to eight weeks, review the top results manually and measure false positives against known heavy legitimate users.
  6. Define graduated responses. Step up verification, reduce quotas, degrade output detail, then suspend. Graduated responses let you roll back if a large customer is misclassified.
  7. Update terms and onboarding. Prohibit training competing models and automated extraction, require disclosure of resale, and make verification proportionate to tier.
  8. Share and learn. Exchange indicators with peers and cloud platforms where your legal team approves, as Anthropic reports doing.

Responding to a confirmed campaign

  • Preserve evidence: account records, payment data, request logs and samples of outputs before suspension, under legal hold.
  • Map the cluster: expand from confirmed accounts through shared attributes before acting, so you disrupt the whole network at once rather than one account that is replaced the next day.
  • Contain: suspend or degrade together, rotate any API keys shared with resellers, and watch for re-registration.
  • Assess the loss: which capabilities and model versions were queried, and how many outputs left. Treat this as input to legal and product decisions.

Common failure patterns

  • Rich outputs by default: returning full probability distributions or top log probabilities to every caller because one customer once asked for it.
  • Per-account limits only: quotas that every individual account respects while a cluster of thousands extracts at scale.
  • Free and trial tiers with no verification: the pathways Anthropic named as most often exploited were educational, research and startup programmes.
  • No retained telemetry: logs kept for days, so a slow campaign cannot be reconstructed or evidenced.
  • Trust and safety and security working apart: fraud teams see payment patterns, security sees API patterns, and nobody joins them.
  • Ignoring your own side: engineers distilling a supplier's model into an internal one in breach of its terms, discovered only during a contract dispute. Cover it in your AI acceptable use policy and the DeepSeek and High-Risk AI Vendor Assessment Guide process.

Lessons from incidents

Anthropic's report of 23 February 2026, recorded in ATLAS as case study AML.CS0056, is the most detailed public account so far. It attributed the activity to three AI labs: over 150,000 exchanges from one, over 3.4 million from another and over 13 million from the third. The lessons for defenders are practical. Access came through commercial proxy services that resell model access, so regional and policy restrictions were bypassed without touching the provider's own sign-up controls. Account networks absorbed bans by replacing accounts, so single-account enforcement failed. And the campaigns targeted the most differentiated capabilities, so capability-level analytics found them faster than volume alone. Older research tells the same story at smaller scale: the 2016 attacks on BigML and Amazon Machine Learning worked with ordinary prediction API access.

Framework mapping

AreaMITRE ATLASNIST SP 800-53 Rev. 5NIST CSF 2.0
Account creation and verificationAML.T0021 Establish Accounts; AML.M0019AC-2 Account ManagementPR.AA-01
Query limitsAML.M0004SC-5 Denial-of-service Protection (resource limits)PR.AA-05
Output minimisationAML.M0002AC-3 Access EnforcementPR.DS-01
Monitoring and analyticsAML.M0024SI-4 System Monitoring; AU-6 Audit Record ReviewDE.CM-09, DE.AE-02
Contracts with customers and resellersAML.T0008.005 AI Service ProxiesNot directly mappedGV.SC-05

NIST AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, provides the shared vocabulary for privacy and extraction attacks when you write policy and threat models.

Practitioner checklist

  • List every externally reachable model endpoint with its value, output detail and governing contract.
  • Require authentication and organisation-level identifiers on every inference call through a gateway.
  • Remove probability vectors and log probabilities from default responses; grant them per contract.
  • Set per-account and per-organisation quotas aligned to real tiers.
  • Verify identity and payment for high-volume, trial, research and startup tiers.
  • Retain request telemetry long enough to reconstruct a slow campaign.
  • Build multi-account correlation on payment, network, metadata and timing.
  • Add capability-concentration and template-repetition detections, tuned in monitor mode first.
  • Publish terms that prohibit training competing models and automated extraction, and check your own teams comply with suppliers' terms.
  • Write a response runbook covering evidence preservation, cluster mapping and coordinated suspension.

Standards and references

Related NHI Mgmt Group resources: AI Model Security Guide · MITRE ATLAS Guide · OWASP LLM Top 10 LLM10 Unbounded Consumption Guide · MITRE ATLAS Mitigations Guide

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 13 November 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.