Every score, probability or debug field an AI system emits beyond what the user needs is a free lesson for an attacker on how to beat the model. MITRE ATLAS AML.T0063 Discover AI Model Outputs covers adversaries discovering what information an AI model or service exposes through its outputs, both the intended responses and additional model-generated information exposed through the surrounding application. ATLAS places it in the Discovery tactic (AML.TA0008), rates it Realized in ATLAS 2026.10, and lists it for predictive, generative and agentic AI; it has no ATT&CK equivalent. This page is for teams that build or expose models, AI APIs and agents. It covers where surplus outputs leak, two ATLAS case studies, how to inventory and reduce output exposure, and how to detect someone harvesting it. Hallucination probing has its own page, AML.T0062 Discover LLM Hallucinations, and what attackers do with the outputs afterwards is covered in the Model Extraction and Distillation Defence Guide.
Key takeaways
- Surplus output is the problem, not output. ATLAS targets outputs whose presence is not required for the system to work: raw scores, ensemble details, token probabilities, debug fields.
- Both ATLAS cases started with a score left in plain view. One security product wrote model scores to verbose logs; another left them in email headers.
- Scores make every later attack cheaper. Research going back to 2016 shows confidence values help attackers copy models, and a 2024 paper used token log probabilities with logit bias to recover part of production LLMs.
- Output exposure is a design decision you can inventory. List every field each model, API, log and agent tool returns, and ask who needs it.
- Four ATLAS mitigations apply. AML.M0002 Predictive AI Output Obfuscation, AML.M0012 Encrypt Sensitive Information, AML.M0017 AI Model Distribution Methods and AML.M0019 Control Access to AI Models and Data in Production.
What counts as a model output here
A user-facing AI system usually needs one thing from its model: a decision, a label or generated text. Around that, models and their serving stacks produce far more: per-class probabilities, raw scores before thresholds, the threshold itself, per-model votes in an ensemble, token-level log probabilities and alternatives, safety classifier category scores, retrieval similarity scores, and timing and debug data. AML.T0063 is about an adversary discovering that some of this is reachable. ATLAS 2026.10 lists predictions, scores, rankings, representations, generated content, reasoning, explanations and evaluation results as relevant outputs.
| Where surplus output appears | Typical examples | Who can usually see it |
|---|---|---|
| Inference API responses | Full probability vectors, raw scores, token log probabilities and top alternatives | Any API caller |
| Product artefacts | Scores in email headers, HTTP headers, file metadata, verdict pages | Recipients, users, anyone who receives the artefact |
| Local and verbose logs | Endpoint or appliance logs showing scores, which sub-model fired, thresholds | Anyone with the product installed or log access |
| Error and debug messages | Stack traces, model names and versions, threshold failures | Callers who trigger errors |
| Guardrail and moderation layers | Per-category safety scores returned to the client | Application users if passed through |
| RAG and agent tooling | Retrieval similarity scores, ranking, tool call traces returned to the user or another agent | Users, connected agents, MCP clients |
| Edge and on-device models | Every output of a model running on hardware the attacker holds | Device owner |
Cases in ATLAS
- Bypassing Cylance's AI Malware Detection (AML.CS0003, 2019): Skylight Cyber researchers turned on verbose logging in the product's advanced interface. The logs showed, for each analysed file, a numeric score, the detector that fired and the verdict; by testing known good and bad files the researchers worked out that the score ran from -1000 (most malicious) to +1000 (most benign). That score gave them a reliable harness for studying the model, which led to a universal bypass. ATLAS records the verbose logs as the AML.T0063 step: they exposed the model's reputation scoring and ensembling.
- ProofPoint Evasion (AML.CS0008, 2019): researchers found that the email protection system left model output scores in email headers. Using those scores they built a copy-cat model and used it to craft emails that scored as acceptable and went undetected. The issue was assigned CVE-2019-20634. Every recipient of a filtered email had access to the scores.
Neither case needed a breach of the vendor: the outputs were delivered to the attacker by the product itself. That is the pattern to look for in your own systems.
ATLAS 2026.10 also links AML.T0063 to a third case, AML.CS0076, on industrial-scale model distillation campaigns against US AI companies.
Why the outputs matter: what they enable
- Model copying: Tramèr and colleagues (2016) showed that confidence values returned with predictions let simple attacks extract logistic regression, neural network and decision tree models with near-perfect fidelity from commercial ML services. They also found that omitting confidence values does not stop extraction, which is why hiding scores is a cost increase, not a complete defence.
- Production LLM extraction: Carlini and colleagues (2024) recovered the embedding projection layer of two production models and the hidden dimension of another, using logit bias together with top-K log probabilities. The authors report that OpenAI and Google implemented defences after disclosure, and that OpenAI stopped logit bias from affecting the returned log probabilities. Their suggested defences include not allowing logit bias and log probabilities in the same query, adding noise to output logits, rate-limiting logit-bias queries and detecting malicious query patterns.
- Evasion tuning: a score tells the attacker how close an input is to the threshold, turning a blind search into a guided one; see the Adversarial Examples and Evasion Attacks Guide.
- Privacy attacks: confidence on specific records is the raw material for Membership Inference Attacks.
- Further discovery: labels and score structure help an attacker map what the model recognises, the subject of AML.T0013 Discover AI Model Ontology.
Reducing output exposure
ATLAS mitigations map to four kinds of control. The first is the one most teams can apply this quarter.
| ATLAS mitigation | What ATLAS says it does for AML.T0063 | Practical control |
|---|---|---|
| AML.M0002 Predictive AI Output Obfuscation | Obfuscating outputs prevents adversaries collecting sensitive information about them | Return labels or coarse bands instead of raw scores; round or bucket probabilities; strip scores from headers and artefacts |
| AML.M0012 Encrypt Sensitive Information | Encrypting outputs prevents discovery of sensitive information about the system or its operations | Where scores must travel with an artefact (for example between filtering stages), sign or encrypt them so recipients cannot read them |
| AML.M0017 AI Model Distribution Methods | Avoiding deployment to edge devices reduces the ability to collect outputs | Keep high-value models server-side; if on-device, expose only the decision to the calling app |
| AML.M0019 Control Access to AI Models and Data in Production | Controlling access helps prevent inferring information from outputs | Authenticate callers, scope keys, and gate rich outputs (log probabilities, scores) to named clients that need them |
- Separate the user response from telemetry: log full outputs server-side for monitoring and keep them out of anything the user receives. See Secure AI logging.
- Treat verbose and debug modes as privileged: require administrator rights to enable them on shipped products and appliances, and keep debug output off in production APIs.
- Control LLM output options: where your platform or gateway passes through token log probabilities or logit bias, disable them by default and allow them only for callers with a documented need. An AI Gateway and LLM Gateway Guide explains enforcing this centrally.
- Sanitise errors: return generic error messages; keep model names, versions and threshold detail in server logs.
- Review agent and RAG outputs: tool results, retrieval scores and traces returned to users or other agents are outputs too; return only what the consumer needs.
- Limit query volume: rate limits (AML.M0004 Limit AI Service Query Volume and Rate) do not hide outputs but raise the cost of harvesting them.
Detecting output harvesting
Discovery is quiet, but harvesting outputs at scale leaves patterns in the places that log model use.
| Signal | Where to look |
|---|---|
| Callers requesting log probabilities, top alternatives or logit bias with no matching use case | API gateway request logs, per-key parameter statistics |
| Logit bias and log probability options used together, repeatedly, on similar prompts | Gateway or provider request logs |
| Many near-duplicate inputs with small variations, especially around the decision threshold | Inference logs, input similarity analysis |
| Deliberately malformed requests that trigger errors | API error logs by caller |
| Verbose or debug mode enabled on product installations | Endpoint configuration management, product audit logs |
| High query volume from a single key or a cluster of new accounts | Rate limiter and billing data |
How to test
- Inventory outputs: for each model, API, product artefact and agent tool, list every field returned and every log it writes, and record who can read each.
- Diff against need: mark each field as required for the user's task, required internally, or surplus. Surplus fields reachable by users are findings.
- Test as a low-privilege caller: call the API with an ordinary key, request every optional output parameter, trigger errors, and inspect responses and headers.
- Inspect artefacts: check emails, files and pages produced by AI filters for scores or verdict detail in headers or metadata, the AML.CS0008 pattern.
- Check shipped products: confirm that verbose logging cannot be enabled, or read, without administrator rights, the AML.CS0003 pattern.
- Retest after changes: new model versions, gateways and SDK upgrades often add output fields by default.
Metrics
- Surplus fields: number of user-reachable output fields marked surplus in the inventory (target: zero).
- Rich output gating: percentage of API keys or clients allowed log probabilities or raw scores, with a documented reason.
- Debug exposure: number of production endpoints or installations with verbose or debug output enabled.
- Error hygiene: percentage of API error responses that reveal model names, versions or thresholds.
- Harvesting alerts: alerts for parameter or near-duplicate query patterns, and time to review.
Framework mapping
| Framework | Item | Relevance |
|---|---|---|
| MITRE ATLAS | AML.T0063 Discover AI Model Outputs (AML.TA0008 Discovery) | The technique; no ATT&CK equivalent is listed |
| MITRE ATLAS mitigations | AML.M0002, AML.M0012, AML.M0017, AML.M0019, AML.M0004 | Output reduction, encryption, distribution, access, rate limits |
| MITRE ATLAS related techniques | AML.T0024 Exfiltration via AI Inference API, AML.T0005 Replicate AI Model Behavior | What the discovered outputs are typically used for |
| OWASP Top 10 for LLM Applications 2025 | LLM02:2025 Sensitive Information Disclosure; LLM10:2025 Unbounded Consumption | Disclosure of system details; extraction through repeated queries |
| NIST SP 800-53 Rev. 5 | SI-11 Error Handling, AC-3 Access Enforcement | Error messages without internal detail; gating rich outputs to authorised callers |
Common failure patterns
- Returning the full probability vector because the framework does by default. The user needed a label.
- Scores in headers and metadata. Fields meant for the next processing stage end up in what customers and attackers receive, as in AML.CS0008.
- Verbose mode available to any user. AML.CS0003 began with a logging option on the product.
- Passing provider options straight through. A wrapper or gateway that forwards every parameter exposes log probabilities and logit bias to every caller.
- Assuming hidden scores stop extraction. Research shows label-only access still allows copying; output reduction needs rate limits and monitoring alongside it.
Practitioner checklist
- Inventory every output field and log for each model, API, artefact and agent tool.
- Return labels or coarse bands to users; keep raw scores server-side.
- Remove model scores from email headers, file metadata and other artefacts.
- Disable token log probabilities and logit bias by default at the gateway.
- Restrict verbose and debug modes to administrators and keep them off in production.
- Return generic errors without model names, versions or thresholds.
- Alert on rich-output parameter use and near-duplicate query bursts.
- Retest output exposure after every model, gateway or SDK change.
Standards and references
- MITRE ATLAS: Discover AI Model Outputs (AML.T0063)
- MITRE ATLAS: Bypassing Cylance's AI Malware Detection (AML.CS0003)
- MITRE ATLAS: ProofPoint Evasion (AML.CS0008)
- MITRE ATLAS: Predictive AI Output Obfuscation (AML.M0002)
- NIST: AI 100-2 E2025, Adversarial Machine Learning taxonomy
- Skylight Cyber: Cylance, I Kill You!
- arXiv: Stealing Machine Learning Models via Prediction APIs
- arXiv: Stealing Part of a Production Language Model
Related NHI Mgmt Group resources: MITRE ATLAS AML.T0024 Exfiltration via AI Inference API · Slopsquatting and Package Hallucination Guide · LLM Observability Guide · MITRE ATLAS Guide