Every answer your model gives is a training example someone else can keep, and at enough volume those answers become a copy of your model's capability. MITRE ATLAS records this collection step as AML.T0135 Outputs from AI Models, a Collection (AML.TA0009) technique added in ATLAS 2026.10 (released 28 October 2026), rated Realized and listed for the Predictive AI, Generative AI and Agentic AI platforms. The adversary gathers, retains and organises a model's outputs as material for later operations: replicating behaviour, inferring training data, recovering model internals or crafting adversarial inputs. This guide is for teams that expose a model through an API, chat product or embedded feature, and for the detection engineers who watch that traffic. It covers what is collected, the real campaigns ATLAS cites, how to detect coordinated collection, and the controls ATLAS maps. The wider extraction and distillation problem has its own page: see the Model Extraction and Distillation Defence Guide.
Key takeaways
- Collection is the hinge step. ATLAS places T0135 between discovering what a model outputs (AML.T0063) and using the material to replicate behaviour (AML.T0005.001), infer training data (AML.T0137) or recover internals (AML.T0144).
- It looks like normal use, one request at a time. The signal is in aggregate: volume, topic concentration, repeated prompt structures and requests for reasoning traces across many accounts.
- The cases are industrial in scale. Anthropic reported about 24,000 fraudulent accounts and more than 16 million exchanges in campaigns it disclosed on 23 February 2026.
- Account and access controls carry much of the defence. Fraudulent sign-ups, resold access through proxy services and shared account pools are how collection reaches scale.
- ATLAS maps one mitigation directly. AML.M0040 Adaptive AI Response Degradation reduces what suspected collection receives; rate limits, output minimisation and telemetry support it.
What the technique is
ATLAS defines Outputs from AI Models as adversaries gathering, retaining and organising outputs from an AI model or service as material for subsequent adversarial operations. Systematic selection and aggregation of responses can produce datasets or observations useful for analysing the target, reproducing its behaviour or developing further attacks. Outputs may be taken directly from inference responses, or indirectly from product artefacts and records in which they are embedded or retained.
The technique is new in 2026.10 and sits alongside several other new IDs that describe how collection is hidden: AML.T0140 Centralized AI Service Request Routing, AML.T0138 Indicator Removal and AML.T0141 AI Service Request Metadata Sanitization. ATLAS also notes that adversaries may use AML.T0065 LLM Prompt Crafting to elicit the outputs they need. The closest older technique for moving data out of an inference endpoint is MITRE ATLAS AML.T0024 Exfiltration via AI Inference API; T0135 is about building the collection, whatever channel it comes through.
What gets collected
| Output type (as listed by ATLAS) | Typical source | Why an adversary wants it | Exposure control |
|---|---|---|---|
| Predictions and scores | Classifier and scoring APIs | Labels a dataset for an imitation model; confidence scores help membership and inversion attacks | Return labels only where scores are not needed; reduce numeric precision (AML.M0002) |
| Embeddings or exposed feature representations | Embedding endpoints, vector search APIs | Dense signal about how the model represents inputs | Restrict embedding endpoints to authenticated internal callers |
| Generated content | Chat, completion, translation, code and image generation | Supervision data for distillation into a student model | Per-account quotas, abuse detection, terms enforcement |
| Reasoning traces | Reasoning model outputs, visible or summarised chains of thought | High-value supervision for reasoning capability | Summarise or withhold raw reasoning; watch for prompts that ask for it |
| Grades or preference judgements | Rubric scoring, model-as-judge use | Reward signal for reinforcement learning | Detect bulk rubric-style grading from consumer or trial accounts |
| Model-generated plans or tool calls | Agent frameworks, function calling | Training data for agentic and tool-use capability | Log and rate tool-use heavy sessions per account |
For predictive models, OWASP's LLM10:2025 Unbounded Consumption guidance adds a specific control: restrict or obfuscate exposure of logit_bias and logprobs in API responses, giving only the information the use case needs.
How it has been used: cases
ATLAS links T0135 to six case studies. The four below show the range, from academic demonstration to state-reported industrial collection.
- Machine translation imitation (AML.CS0005): researchers queried public translation services with monolingual sentences and trained imitation models on the outputs, coming within 0.6 BLEU of three production systems. They then crafted adversarial inputs on the imitation and transferred them to the real services. The defence they proposed, perturbing outputs to mislead imitation training, also cost the defender some translation quality and speed.
- Distillation of Claude (AML.CS0056): Anthropic reported on 23 February 2026 that three AI labs used about 24,000 fraudulent accounts and more than 16 million exchanges to collect Claude's responses, targeting reasoning, rubric grading, agentic tool use, coding and computer use. Access came through commercial proxy services; one proxy network managed more than 20,000 accounts and mixed distillation traffic with ordinary customer requests.
- Distillation of Gemini (AML.CS0075): Google Threat Intelligence Group reported on 12 February 2026 a campaign of over 100,000 prompts that tried to make Gemini output its full reasoning, which is normally summarised. Google says its systems recognised the activity in real time and protected the reasoning traces. ATLAS adds that later campaigns exceeded 100 million prompts.
- Cross-provider collection (AML.CS0076): ATLAS summarises a joint NSA, CISA and FBI advisory (AA26-251A) attributing industrial-scale distillation to six China-based AI companies since at least late 2024. ATLAS reports fraudulent accounts, bulk premium subscriptions, gray-market proxies known as "transfer stations", automatic removal of organisational identifiers from requests, and automated checks that told degraded outputs apart from ordinary service problems.
The other two cases, ProofPoint Evasion (AML.CS0008) and Confusing Antimalware Neural Networks (AML.CS0014), show the same step against security products: collect the product's verdicts, then train a model that predicts them.
Who is exposed and how much it matters
Model providers and teams that sell model access: the main risk is loss of capability and intellectual property. Google's February 2026 post says extraction does not usually threaten the confidentiality, availability or integrity of AI services for ordinary users; the risk falls on model developers and service providers.
Enterprises exposing a fine-tuned or specialised model: a small, valuable model (fraud scoring, document classification, a domain assistant) needs far fewer queries to imitate than a frontier model. Treat it like any other proprietary asset.
Security product vendors and their customers: any model whose verdict is visible (a spam score in an email header, a malware detection result) can be harvested to train an evasion model, as the ProofPoint and antimalware cases show.
Internal-only models: lower exposure, but insiders and compromised service accounts still have query access. The access control and logging guidance below still applies.
Detection: what to log and what to alert on
Single requests look legitimate, so detection works on accounts, clusters of accounts and prompt populations. Anthropic describes classifiers and behavioural fingerprinting for distillation patterns, including chain-of-thought elicitation, and tooling to spot coordinated activity across many accounts. Build the equivalent at your gateway (see the AI Gateway and LLM Gateway Guide).
| Signal | Where to collect it | Example alert logic |
|---|---|---|
| Volume and pace per account and per organisation | API gateway, billing records | Sustained near-quota use from new or trial accounts; machine-regular request timing |
| Topic concentration | Prompt classification on sampled traffic | One account or cluster focused on a single capability (reasoning, coding, tool use) far beyond its stated use case |
| Template reuse | Prompt structure hashing | The same prompt skeleton with systematic variations across many accounts |
| Reasoning elicitation | Prompt classifiers | Repeated requests to reveal, translate or reconstruct hidden reasoning |
| Account clustering | Sign-up data, payment, IP and device signals | Many accounts sharing payment instruments, proxy egress or sign-up patterns; banned accounts quickly replaced |
| Metadata stripping | Request headers and client fingerprints | Requests consistently missing identifiers that normal clients send (ATLAS T0141) |
Metrics that show the programme is working:
- Time to detect a collection cluster: from first request to analyst confirmation.
- Share of traffic attributable to verified organisations: rising share means less anonymous capacity for collection.
- Fraudulent account rate at sign-up: accounts blocked or later banned for abuse, per thousand sign-ups.
- Reasoning-elicitation requests blocked or degraded: count and trend.
- Replacement rate after bans: how quickly banned clusters reappear, a measure of how well account controls hold.
Defences that work
ATLAS maps AML.M0040 Adaptive AI Response Degradation to T0135: when collection is suspected, route requests to a more constrained model, reduce output precision or completeness, withhold reasoning or confidence details, restrict structured high-fidelity responses, or add controlled perturbations. ATLAS asks that degradation be proportional to detection confidence and evaluated so it does not produce unsafe or misleading output, disclose the detection decision, or break authorised use. The AA26-251A summary is a warning here: collectors run automated quality checks to spot degradation, so subtle, varied responses matter more than an obvious drop in quality.
| Control | ATLAS or OWASP reference | What it achieves | Trade-off |
|---|---|---|---|
| Adaptive response degradation | AML.M0040 | Collected material is worth less as training data | Risk of hurting legitimate users on false positives |
| Query volume and rate limits | AML.M0004; OWASP LLM10:2025 | Raises the cost and account count needed | Pushes collectors to account pools, so pair with identity controls |
| Output minimisation (scores, logprobs, reasoning) | AML.M0002; OWASP LLM10:2025 | Less signal per response | Some customers need scores; offer them under verified tiers |
| Telemetry and monitoring | AML.M0024 | Makes the detection table above possible | Prompt logging needs privacy and retention rules |
| Account verification for high-risk tiers | Anthropic's reported response | Fewer fraudulent accounts at scale | Friction for genuine startups and researchers |
| Intelligence sharing | Anthropic and Google reported practice | Indicators reach other providers before the same cluster moves | Needs agreements and a sharing channel |
Implementation sequence
- Classify your models: list each externally reachable model, its value if copied, and what outputs it returns (scores, logprobs, reasoning, embeddings).
- Minimise outputs first: remove scores, logprobs and raw reasoning from tiers that do not need them. This is cheap and needs no detection.
- Turn on telemetry: per-request account, organisation, client fingerprint, prompt hash and capability category at the gateway, with a defined retention period.
- Set quotas by trust tier: tight for anonymous and trial accounts, higher for verified organisations.
- Build detections in monitor-only mode: run the signals above for two to four weeks, review hits, and tune against known heavy legitimate users.
- Add graduated responses: step from monitoring to throttling, then degradation, then suspension, with a documented appeal path. Keep a switch to roll back degradation per account if a customer is affected.
- Exercise it: include a distillation scenario in AI red team plans (AML.M0035) and in the AI Model Theft Response Playbook.
Small and large organisations: a team exposing one internal model can stop at steps 1 to 4. A provider selling model access at scale needs the full sequence, account-clustering analytics and a sharing relationship with peers.
Where identity fits
Every case above relied on access at scale: fraudulent accounts, resold API keys, shared subscriptions and pools of credentials that replace banned ones. That makes customer API keys and service accounts the control surface. Bind keys to verified organisations, scope them to the models and output types the customer needs, rotate them, and watch for the same key used from many networks. See AML.T0012 Valid Accounts for the access side and AI Agent Rate Limits for quota design.
Common failure patterns
- Per-account limits only: collection spread over thousands of accounts stays under every individual limit. Aggregate by organisation, payment, network and prompt template.
- Returning everything by default: exposing full probability vectors, logprobs or raw reasoning to every tier hands collectors the richest signal for free.
- Obvious degradation: a sudden quality drop tells the collector it has been detected; ATLAS's AA26-251A summary shows collectors test for exactly this.
- No retained telemetry: without prompt hashes and account metadata, a campaign found later cannot be scoped or attributed.
- Treating it as a billing issue: heavy usage that is paid for is not reviewed, even when its pattern matches collection.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| MITRE ATLAS | AML.T0135 (AML.TA0009 Collection) | The technique; leads to AML.T0005.001, AML.T0137, AML.T0144 |
| MITRE ATLAS mitigations | AML.M0040 (mapped by ATLAS); AML.M0002, AML.M0004, AML.M0024, AML.M0035 (supporting) | Degradation mapped to T0135; the others are mapped by ATLAS to related privacy techniques and support collection defence |
| OWASP Top 10 for LLM Applications 2025 | LLM10:2025 Unbounded Consumption | Model extraction via API and functional model replication |
| NIST SP 800-53 Rev. 5 | AC-6, SI-4, AU-6, SI-15 | Least privilege for API keys, monitoring, log review, output filtering |
| NIST CSF 2.0 | PR.AA-05, DE.CM-09, DE.AE-02 | Access permissions, monitoring of runtime and data, analysis of adverse events |
Practitioner checklist
- Inventory every externally reachable model and the output types it returns.
- Remove scores, logprobs and raw reasoning from tiers that do not need them.
- Log account, organisation, client fingerprint and prompt hash for every request.
- Set quotas by trust tier and aggregate limits across related accounts.
- Detect template reuse, topic concentration and reasoning-elicitation prompts.
- Define graduated responses, including degradation, with a per-account rollback.
- Verify organisations before granting high-volume or high-fidelity access.
- Run a distillation scenario in AI red team exercises at least yearly.
Standards and references
- MITRE ATLAS: Outputs from AI Models
- MITRE ATLAS: Adaptive AI Response Degradation
- MITRE ATLAS: China-Based AI Companies Conducting Industrial-Scale Distillation Campaigns (case study)
- OWASP: LLM10:2025 Unbounded Consumption
- Anthropic: Detecting and preventing distillation attacks
- Google Threat Intelligence Group: Distillation, Experimentation, and Integration of AI for Adversarial Use
- Wallace et al.: Imitation Attacks and Defenses for Black-box Machine Translation Systems
Related NHI Mgmt Group resources: MITRE ATLAS Guide · Model Extraction and Distillation Defence Guide · AML.T0063 Discover AI Model Outputs · OWASP LLM Top 10 LLM10 Unbounded Consumption Guide · MITRE ATLAS Mitigations Guide