An attacker who knows exactly which classes, identities or categories your model can output knows what to aim a targeted attack at, and many deployments hand that list over through their API, their configuration files or their documentation. MITRE ATLAS records this as AML.T0013 Discover AI Model Ontology, a Discovery (AML.TA0008) technique for Predictive AI rated Demonstrated. The ontology is the model's output space: the object types a detector can find, the people a face identification system is enrolled to recognise, the categories a fraud or content classifier assigns. This guide explains how that output space leaks, why it matters, how to spot an adversary enumerating it, and how to reduce what the model gives away. It is for ML engineers, AI platform owners and security teams that expose predictive models through APIs or products. General inference API hardening is covered in the Model Extraction and Distillation Defence Guide; this page focuses on the ontology.
Key takeaways
- Targeted attacks need a target class. ATLAS notes the ontology tells the adversary how the victim uses the model and helps create targeted attacks; in MITRE's own red team case it gave the list of identities to impersonate.
- There are three leak paths. Repeated queries that force the model to enumerate its outputs, configuration files that ship with the model, and documentation about it.
- Rich outputs make enumeration cheap. Full ranked label lists and precise confidence scores reveal far more of the output space per query than a single top label.
- ATLAS maps two mitigations. AML.M0002 Predictive AI Output Obfuscation and AML.M0004 Limit AI Service Query Volume and Rate.
- Enumeration has a shape you can detect. One client steadily covering more of the label space than any real user is a strong signal, provided you log the labels returned per client.
What the technique is
ATLAS describes AML.T0013 as discovering the ontology of an AI model's output space, for example the types of objects a model can detect. The adversary may discover it through repeated queries that force the model to enumerate its output space, or find it in a configuration file or in documentation about the model. The ontology helps the adversary understand how the victim uses the model and is useful in creating targeted attacks.
A closely related technique, AML.T0014 Discover AI Model Family, covers learning the general type of model rather than its output space, and has its own ATLAS entry. Discovering the outputs themselves is AML.T0063 Discover AI Model Outputs, also a separate entry. Both often run alongside ontology discovery, and the mitigations overlap.
| Model type | What the ontology is | Why an attacker wants it |
|---|---|---|
| Face identification | The enrolled identities the model can return | Choose an identity to impersonate in a targeted misclassification |
| Object detection and image classification | The object classes it detects | Know which class to hide from, or which harmless class to be mistaken for |
| Malware and spam detection | Family or category labels and verdict levels | Learn how verdicts are structured to aim evasion at the most lenient one |
| Fraud and risk scoring | Risk bands, reason codes and decision outcomes | Understand which outcome triggers review and steer just below it |
| Content moderation and intent routing | Policy categories and routing intents | Find categories that are not covered, or the intent that reaches a privileged path |
How the ontology leaks
Through queries
If an API returns a ranked list of labels with scores, each query discloses several classes at once, and a modest set of varied inputs reveals most of the label space. Even a top-1 API gives the ontology away over enough queries. Research on prediction APIs makes the general point: Tramèr and colleagues showed in 2016 that confidence values returned by public ML services gave attackers enough information to extract near-identical copies of models, and that removing confidence values reduced but did not remove the risk. The same logic applies to the label space.
Through configuration and artefacts
Model packages commonly carry their label map. In the Hugging Face Transformers library, for example, the model configuration includes id2label, a map from prediction index to label, and label2id, its inverse. Anyone who can download the model, or the configuration file shipped with an app or container, reads the ontology directly. Mobile and edge apps that embed a model often include a label file alongside it.
Through documentation and errors
API reference pages, schema enums, product help pages and error messages that list valid categories all describe the ontology. Some disclosure is necessary for legitimate clients; the question is whether the full list is needed by every caller or only by integrators.
Case: face identification evasion (AML.CS0012)
MITRE's AI Red Team demonstrated a physical-domain evasion attack on a commercial face identification service, aiming for a targeted misclassification. The steps ATLAS records show where ontology discovery sits in the chain:
- Reconnaissance on the target model (AML.T0000) and acquiring representative open source data (AML.T0002.000).
- Access to the service and its API through a valid account (AML.T0012) and inference API access (AML.T0040).
- Querying the inference API to identify the list of identities the model targeted (AML.T0013).
- Building a proxy model (AML.T0005) and optimising an adversarial patch against it (AML.T0043.000).
- Printing the patch and placing it in the physical environment (AML.T0008.003, AML.T0041), causing targeted misclassifications (AML.T0015).
Without the identity list, the team could only aim for untargeted errors. With it, they could choose who to be recognised as. The valid account step is covered in AML.T0012 Valid Accounts; the final step is AML.T0015 Evade AI Model.
Mitigations
| Control | ATLAS reference | Implementation detail |
|---|---|---|
| Restrict the number of results shown | AML.M0002 Predictive AI Output Obfuscation | Return the top label only, or only the decision the application needs, not a ranked list |
| Limit specificity of the output ontology | AML.M0002 | Map fine-grained internal classes to coarse external outcomes, for example "match / no match" rather than an identity name for callers that only need verification |
| Reduce the precision of numerical outputs | AML.M0002 | Round or bucket confidence scores; withhold logits, embeddings and class rankings unless a caller has a documented need |
| Randomised smoothing | AML.M0002 | Listed by ATLAS as a suggested approach; test its effect on accuracy before use |
| Limit query volume and rate | AML.M0004 Limit AI Service Query Volume and Rate | Per-identity rate, burst and quota limits; ATLAS says this reduces the ability to discover the full output ontology, and warns to watch for account rotation and distributed requests |
Practical controls beyond the two ATLAS mappings:
- Tiered API responses: give the detailed output only to authenticated integrators that need it, and the minimal output to everyone else.
- Keep label maps out of shipped artefacts: where a model runs on a client or device, map indices to labels on the server, or ship only the labels the client must display.
- Generic errors: do not list valid categories in error messages to unauthenticated callers.
- Documentation review: publish the categories customers need, and keep internal class lists, enrolment counts and identity lists out of public pages.
- Control access to the model itself: anyone who can download weights and configuration reads the ontology with no queries at all. Treat model files as sensitive artefacts.
What to detect and where
| Where | What to log | Signal of ontology discovery |
|---|---|---|
| Inference API gateway | Caller identity, request count, response labels and scores per call | Distinct labels returned to one caller rising steadily toward the full label set |
| Inference API gateway | Request parameters such as top-k or return-all-scores flags | Callers always requesting the maximum result count |
| Input monitoring | Input features or hashes | Synthetic, near-duplicate or systematically varied inputs rather than real-world data |
| Identity and billing | New accounts, trial sign-ups, API key creation | Several new accounts with similar query patterns, the rotation ATLAS warns about |
| Artefact stores | Downloads of model files and configuration | Unexpected access to label maps or model packages |
A useful single metric is label coverage per caller: the share of the model's classes returned to a given identity over a rolling window. Legitimate users of most applications see a skewed subset; an enumerating adversary trends toward full coverage.
Measuring it
- Share of predictive AI endpoints that return only the outputs the application needs.
- Share of endpoints returning raw confidence scores, logits or full rankings to unauthenticated or general callers (target zero).
- Maximum label coverage reached by any single caller per month, with alerts above an agreed threshold.
- Number of shipped apps, containers or public repositories containing a full label map.
- Time to detect a test enumeration run in red team exercises.
How to test
- Documentation and artefact review: list everything an outsider can read about the output space from public docs, SDKs, app packages and model hubs.
- Controlled enumeration: in a test environment, run a scripted set of varied benign inputs against the API under a test identity and record how many queries reveal how much of the label space.
- Compare output settings: repeat with top-1 only and rounded scores to show the effect of AML.M0002 controls on enumeration cost.
- Check detection: confirm the label coverage alert fires and that rate limits apply across rotated test accounts.
- Record and roll back: keep results in the AI red team report and remove test identities. Plan these tests with the AI Red Team Test Plan Template.
Common failure patterns
- Returning everything by default: APIs that return all class scores because the framework does, not because a caller needs it.
- Internal and external outputs are the same: the fine-grained ontology built for analysts is exposed unchanged to customers.
- Label files in client packages: on-device models shipped with a complete label map, readable without a single query.
- Rate limits per key only: limits that reset when an adversary creates another trial account.
- No per-caller output logging: request counts are logged but returned labels are not, so enumeration cannot be seen.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| MITRE ATLAS | AML.T0013 (AML.TA0008 Discovery) | The technique; precedes targeted AML.T0043 and AML.T0015 |
| MITRE ATLAS mitigations | AML.M0002, AML.M0004 | Output obfuscation; query volume and rate limits |
| MITRE ATLAS case study | AML.CS0012 | Identity list discovered through the inference API before a targeted physical evasion |
| NIST AI 100-2 E2025 | Adversarial ML taxonomy | Attacker knowledge and capabilities, evasion attacks and their mitigations |
| NIST SP 800-53 Rev. 5 | AC-3, SC-5, AU-12 | Access enforcement for detailed outputs, rate limiting against resource abuse, audit record generation |
For the extraction risk that rich outputs also create, see MITRE ATLAS AML.T0024 Exfiltration via AI Inference API.
Practitioner checklist
- Return only the labels and precision each caller needs; default to top-1 and coarse outcomes.
- Withhold logits, embeddings and full class rankings from general callers.
- Apply rate limits and quotas per authenticated identity, and watch for account rotation.
- Log returned labels per caller and alert on rising label coverage.
- Keep full label maps out of client apps, public repositories and public documentation.
- Treat model files and configuration as sensitive artefacts with access control.
- Test enumeration cost and detection in AI red team exercises.
Standards and references
- MITRE ATLAS: Discover AI Model Ontology
- MITRE ATLAS: Predictive AI Output Obfuscation
- MITRE ATLAS: Limit AI Service Query Volume and Rate
- MITRE ATLAS: Face Identification System Evasion via Physical Countermeasures
- NIST: AI 100-2 E2025 Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
- Tramèr et al.: Stealing Machine Learning Models via Prediction APIs
- Hugging Face: Transformers configuration (id2label, label2id)
Related NHI Mgmt Group resources: MITRE ATLAS Guide · Model Extraction and Distillation Defence Guide · MITRE ATLAS Mitigations Guide · AI Red Teaming Guide