Every product with AI inside it is also a door to the model, and that door often reveals more about the model than its owners intended. MITRE ATLAS tracks this as AML.T0047 AI-Enabled Product or Service, in the AI Model Access tactic (AML.TA0000): adversaries use an ordinary product or service that relies on AI under the hood to reach the underlying model, and the product may reveal details of the model or its inferences in logs or metadata. ATLAS rates the technique as realised. This guide is for product security teams, application owners and SOC analysts who ship AI features: how to find what your product discloses, how to reduce it, how to spot someone mapping your model through the product, and how to respond. For the framework as a whole, see the MITRE ATLAS Guide.
Key takeaways
- Indirect access is still access. AML.T0047 needs no API key or stolen weights: a customer account, a free trial or an installed app is enough to start probing the model inside.
- Metadata is the leak. Scores in headers, verdict fields in responses, model names in error messages and verbose client logs give adversaries the feedback loop they need to build evasions.
- ATLAS links one mitigation, we add four. ATLAS maps AML.M0024 AI Telemetry Logging to this technique. Output fidelity, query limits, production access control and where the model runs are our additions (NHI Mgmt Group), drawn from related ATLAS mitigations.
- It is a staging step. ATLAS case studies show product access used to build copy-cat models, craft evasions and inject prompts, so detection here buys time before impact.
- Treat it as a design review item. Most of the fix is deciding, feature by feature, what the user actually needs to see, and removing the rest before launch.
How the technique works
ATLAS separates direct model access, such as AML.T0040 AI Model Inference API Access, from indirect access through a product. With AML.T0047 the adversary uses the product as designed: they upload files to a malware scanner, send email through a filtering service, present a face to an identity check, or type into a chat feature. Each interaction returns something, and the adversary studies those responses to learn how the model behaves.
The product becomes a measuring instrument when it exposes any of the following:
- Scores and verdicts: a numeric score, confidence value or class label in a response body, a header, a webhook or an exported report. ATLAS describes this as AML.T0063 Discover AI Model Outputs: outputs such as class scores that the system does not need and that the end user was never meant to use.
- Model identity: model family, version or provider names in error messages, response fields or client logs, which support AML.T0014 Discover AI Model Family.
- Configuration: system prompts, tool lists and agent instructions disclosed through the chat interface, mapped by ATLAS as AML.T0069 Discover LLM System Information and AML.T0084 Discover AI Agent Configuration.
- Client-side logic: desktop or mobile clients that extract features locally or store the token the client uses to call the model backend.
Once the adversary has that feedback loop, the usual next steps are AML.T0005 Create Proxy AI Model, AML.T0043 Craft Adversarial Data and AML.T0015 Evade AI Model. Extraction through APIs is covered in the Model Extraction and Distillation Defence Guide; this page stays with the product surface.
Where it shows up
ATLAS links AML.T0047 to more case studies than most techniques. The pattern is the same across them: a normal user path gave enough access to study or misuse the model.
| Product type | What the product exposed | ATLAS case study |
|---|---|---|
| Email protection service | Per-message model scores in email headers; researchers sent many emails to collect them, built a copy-cat model and crafted emails that scored as safe | AML.CS0008 |
| Endpoint anti-malware with cloud classifier | Verdicts from a product that extracted features locally and sent them to a cloud ML detector, giving black-box access | AML.CS0014; also AML.CS0003 |
| Identity verification and KYC | Accept or reject decisions from facial recognition and liveness checks, probed with generated or injected video | AML.CS0004, AML.CS0017, AML.CS0033, AML.CS0034 |
| LLM application or assistant | The underlying model reached through an app feature, then steered with crafted prompts | AML.CS0016, AML.CS0035, AML.CS0026 |
| AI agent reachable by email or chat | Agent responses delivered to an address the adversary controlled | AML.CS0037 |
| LLM desktop application | Access to the model backend as if the adversary were the desktop client | AML.CS0036 |
Two lessons stand out. Security products are attractive targets because their verdicts are exactly what an attacker wants to defeat. And identity verification products face the same probing at scale, so repeated failed attempts are a signal in their own right. See the Biometric Liveness and Presentation Attack Detection Guide.
Finding what your product discloses
Start with an exposure review for every AI feature, before and after launch. Our recommended method (NHI Mgmt Group):
- List the AI features: every place a model makes or shapes a decision the user can observe, including background scoring such as fraud, spam and moderation.
- Capture the full response: for each feature, record the raw HTTP response, headers, webhooks, exported reports, notification emails and client logs, not just what the user interface shows.
- Classify each field: needed by the user, needed by a downstream system, or not needed. Mark anything numeric that changes smoothly with the input as high risk, because it lets an adversary optimise against it.
- Check error paths: malformed inputs, oversized files and timeouts often return stack traces, model names or upstream provider errors.
- Test the chat surface: ask the assistant for its instructions, tools and model name as part of red teaming, and record what it reveals. See the OWASP LLM Top 10 LLM07 System Prompt Leakage Guide.
- Inspect clients: for desktop and mobile apps, check what runs locally, what is logged on the device and how the client authenticates to the model backend.
| Disclosure | Why it helps an adversary | Default action (our recommendation) |
|---|---|---|
| Raw score or probability | Gradient-like feedback for evasion and proxy training | Return a coarse verdict; keep scores server side |
| Score in a header or report | Invisible to product designers, visible to anyone reading raw traffic | Strip at the edge; review every outbound header |
| Model name and version | Lets the adversary pick a matching public model as a proxy | Remove from responses and errors; keep in internal logs |
| Detailed rejection reason | Tells the adversary which feature triggered the block | Generic user message; detailed reason to the analyst queue |
| System prompt or tool list | Maps the agent's capabilities and guardrails | Assume it will leak; keep secrets and access rules out of prompts |
| Client-held backend token | Direct model access outside the product's controls | Short-lived, user-bound tokens; server-side policy enforcement |
Detection
ATLAS mitigation AML.M0024 AI Telemetry Logging calls for logging the inputs and outputs of deployed models, and for agents, the intermediate steps, data access, tool use and agent identity. ATLAS notes that telemetry logging can help identify whether sensitive model information has been sent to an attacker. For a product, join that model telemetry to the product's own account and session data.
| Signal | What it suggests | Where to look |
|---|---|---|
| One account submitting many near-identical inputs with small variations | Probing a decision boundary | Input hashes and similarity clusters per account |
| High rate of rejected or flagged items followed by a sudden pass | Iterative evasion that found a gap | Verdict sequences per account and per sender |
| Trial or newly created accounts with automation-like timing | Disposable access for study | Sign-up records, device and network attributes |
| Prompts asking for instructions, tools, model name or configuration | Discovery of system information | Prompt classifier labels in model telemetry |
| Client traffic to the model backend that lacks normal product calls | Backend access outside the client, as in AML.CS0036 | API gateway logs, client version and token audience |
| Large-scale submissions of crafted media to identity checks | Probing liveness and face matching | Verification attempt logs per device and document |
In MITRE D3FEND terms these analytics sit under D3-UBA User Behavior Analysis. Measure the programme with:
- Exposure review coverage: share of AI features with a completed disclosure review in the last 12 months.
- Score exposure count: number of features or endpoints that still return raw scores or model identifiers to end users.
- Probe detection time: time from the first query of a confirmed probing campaign to account action.
- Telemetry coverage: share of AI features whose inputs, outputs and caller identity reach the SOC.
Prevention and hardening
ATLAS maps only AML.M0024 to AML.T0047. The other rows below are our additions (NHI Mgmt Group): related ATLAS mitigations that reduce what product access can learn.
| Control | What to do in the product | Trade-off |
|---|---|---|
| AML.M0024 AI Telemetry Logging (ATLAS-mapped) | Log inputs, outputs, verdicts and caller identity for every AI feature, with retention long enough to reconstruct slow probing | Logs hold user content; set access and retention deliberately |
| AML.M0002 Predictive AI Output Obfuscation | Return the coarsest output that serves the user: a label rather than a probability, rounded values, fewer results | Some business customers need scores; offer them only to verified, contracted tiers |
| AML.M0004 Limit AI Service Query Volume and Rate | Per-account, per-device and per-tenant limits on AI-backed actions, separate from general API limits | Bulk legitimate users need higher tiers |
| AML.M0019 Control Access to AI Models and Data in Production | Require authentication for AI features, and do not expose the backend model endpoint to clients directly | Friction for anonymous or free use |
| AML.M0017 AI Model Distribution Methods | Prefer serving models and computing features in the cloud over shipping them in clients, where the use case allows | Latency, offline use and privacy may favour on-device; see the On-Device AI Security Guide |
Decision guidance
- Consumer and free-tier products: assume any account is an adversary account. Return verdicts only, rate limit AI actions per device, and keep scores out of every channel.
- Business products with integrations: customers may need scores for their own tuning. Gate them behind contracts and verified tenants, log who receives them, and never put them in headers that travel with the content.
- Security and fraud products: the verdict is the target. Vary the detail of feedback, delay or batch notifications where the workflow allows, and watch for accounts that only ever test.
- Small teams: start with the exposure review and stripping scores and model names from responses; both are cheap and remove the easiest feedback loops.
Where identity fits
The adversary in AML.T0047 uses a legitimate identity: a customer account, a trial tenant or an installed client. The product, in turn, calls the model with its own machine identity. Two controls follow (our recommendations):
- Bind AI actions to the end user: pass a user or tenant identifier with each model call so limits, logging and suspension work per user rather than per product service account.
- Keep backend credentials off the client: clients should hold short-lived tokens scoped to the product's own API, not a key that can call the model provider directly. The AIKatz case study (AML.CS0036) shows what happens when a client credential reaches the backend.
See the AI Gateway and LLM Gateway Guide for brokering model calls through one controlled point.
Response
- Confirm the pattern: group accounts, devices and tenants by shared attributes and input similarity, and establish what the adversary could see (scores, verdicts, configuration).
- Preserve evidence: export model telemetry, verdict history and account records for the cluster before retention removes them.
- Contain: suspend the accounts together, revoke client tokens and tighten limits on the affected feature.
- Close the leak: remove the disclosed field, header or error detail, and redeploy. Retest with the same inputs.
- Assume staging: if scores or configuration leaked, expect evasion or prompt injection to follow. Review recent passes by the same cluster, retrain or retune where needed, and add detections for the crafted inputs.
- Record and report: log the incident against AML.T0047 and any follow-on techniques; see the AI Incident Reporting Guide.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| MITRE ATLAS | AML.T0047 in AML.TA0000; related AML.T0063, AML.T0014, AML.T0069, AML.T0084; follow-on AML.T0005, AML.T0043, AML.T0015 | Technique, discovery steps and what comes next |
| MITRE ATLAS mitigations | AML.M0024 (mapped); AML.M0002, AML.M0004, AML.M0017, AML.M0019 (our additions) | Telemetry, output fidelity, limits, access, deployment |
| OWASP Top 10 for LLM Applications 2025 | LLM02 Sensitive Information Disclosure; LLM07 System Prompt Leakage | What generative features reveal |
| NIST SP 800-53 Rev. 5 | AC-3, SI-4, AU-12, SC-5 | Access enforcement, monitoring, logging, resource limits |
| NIST CSF 2.0 | DE.CM-09, DE.AE-02, RS.MI-01 | Runtime monitoring, analysis, containment |
| MITRE D3FEND | D3-UBA | Behavioural analytics on product accounts |
Common failure patterns
- Debug fields left in production: a score or model version added for the data science team ships in the public response and nobody removes it.
- Reviewing the UI, not the wire: designers check the screen while scores travel in headers, webhooks or exports.
- Limits on the API but not the product: the public API is rate limited, while the web feature that calls the same model is not.
- One service identity for all users: every model call carries the product's key, so probing cannot be tied to an account.
- Secrets in the system prompt: teams treat the prompt as hidden, then the assistant repeats it on request.
Practitioner checklist
- Inventory every AI feature, including background scoring, and name an owner for each.
- Run a disclosure review per feature covering responses, headers, webhooks, exports, errors and client logs.
- Remove raw scores, model names and detailed rejection reasons from user-facing channels by default.
- Apply per-account and per-device limits to AI-backed actions.
- Log inputs, outputs, verdicts and caller identity for every AI feature and send them to the SOC.
- Build analytics for near-duplicate probing, rejection-then-pass sequences and configuration-seeking prompts.
- Keep model backend credentials off clients and bind model calls to the end user or tenant.
- Add AML.T0047 probing to AI red team plans and repeat the review after major model or feature changes.
Standards and references
- MITRE ATLAS: AI-Enabled Product or Service
- MITRE ATLAS: Discover AI Model Outputs
- MITRE ATLAS: AI Telemetry Logging
- MITRE ATLAS: AML.CS0008 email protection evasion case study
- OWASP: LLM07:2025 System Prompt Leakage
- NIST: AI 100-2 E2025 Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
- NCSC: Guidelines for secure AI system development
Related NHI Mgmt Group resources: MITRE ATLAS Mitigations Guide · Model Extraction and Distillation Defence Guide · Adversarial Examples and Evasion Attacks Guide · AI Red Teaming Guide