AML.T0062 Discover LLM Hallucinations: Spotting Attackers Probing Your LLM for Hallucinated Names

Before an attacker can register a package or domain that a model invents, they have to find out which names the model invents, and they find out by asking it, at scale. MITRE ATLAS tracks this as AML.T0062 Discover LLM Hallucinations, in the Discovery tactic (AML.TA0008). ATLAS describes adversaries prompting models for software packages, commands, URLs, organisation names or email addresses and keeping the answers that have no real-world source, as targets for AML.T0060 Publish Hallucinated Entities. It explains what harvesting looks like on AI services you operate, which outputs to stop exposing, and how to run the same discovery yourself so you find the names before an attacker does. It is for teams running LLM applications, model APIs and AI gateways, and for security teams building detection for them.

Key takeaways

• Most of this discovery happens where you cannot see it. An attacker can harvest hallucinations from a public assistant or API; you only get telemetry when they harvest from a model or application you host.
• Repeatability is what the attacker is measuring. A name worth registering comes back for many users. Lasso Security found repetition rates of 13.6% to 24.2% across four models when the same question was asked 100 times, and ATLAS notes that different LLMs produce the same hallucinations.
• Harvesting has a shape. Templated questions, the same prompt repeated many times, sampling settings varied between runs and requests for token probabilities are all signals an AI gateway can log and alert on.
• Do not hand out more output than the use case needs. Token log probabilities, raw scores and debug fields help an attacker probe your model; excess output is covered on the AML.T0063 Discover AI Model Outputs page.
• Discover your own hallucinations first. Running the same questions your users ask, and checking every returned entity, gives you the blocklist an attacker would otherwise turn into a target list.

What the technique covers

ATLAS added AML.T0062 in March 2025 and rates its maturity as Demonstrated (ATLAS 2026.10 data).

How attackers harvest hallucinations

ATLAS case study AML.CS0022 ChatGPT Package Hallucination records the method in its first step. Researchers from Vulcan Cyber and Lasso Security used the public ChatGPT API (AML.T0040 AI Model Inference API Access) to ask coding questions, then checked the suggested packages against the public registry. One answer recommended installing huggingface-cli, a name that did not exist on PyPI; the real tool ships in the huggingface_hub package. The next step, publishing an empty package under that name, is AML.T0060.

Lasso's follow-up research shows the scale an attacker can reach with ordinary API access. It drew 47,803 "how to" questions across Python, Node.js, Go, .NET and Ruby, sent them to four models through their APIs, and measured both how often each model hallucinated and how often the same hallucination returned:

Lasso also found 215 hallucinated packages that every model produced. The larger academic study by Spracklen et al. (USENIX Security 2025) generated 576,000 code samples from 16 models and recorded 205,474 unique hallucinated package names, with average rates of at least 5.2% for commercial models and 21.7% for open-source models. For an attacker these numbers describe a cheap, repeatable harvest: ask many plausible questions, keep the names that do not exist and recur, and register those.

The same method works for web addresses. Netcraft asked a GPT-4.1 family model, in plain language, where to log in to 50 brands. Of the 97 domains returned, 28 (29%) were unregistered, parked or inactive and 5 (5%) belonged to unrelated businesses. Netcraft found smaller institutions, such as credit unions and regional banks, were more often wrong. Anyone can repeat that survey for a sector and register the free domains.

The Slopsquatting and Package Hallucination Guide covers the package defences in depth; the companion page on AML.T0060 Publish Hallucinated Entities covers what happens after a name is registered.

Related: excess model outputs

Scores, token log probabilities and debug fields that a system exposes without need are a separate ATLAS technique, AML.T0063 Discover AI Model Outputs, with its own page. See the Model Extraction and Distillation Defence Guide for the extraction side.

What to detect and where

You will not see harvesting of a third-party public model. You will see it on any assistant, API or agent you expose to customers, partners or the internet, provided the AI gateway or application logs prompts, parameters and responses.

Tie each request to an identity. Harvesting through a shared API key or an anonymous endpoint is indistinguishable from normal use; per-client keys or workload identities with their own quotas make the pattern visible and let you cut off one caller. The LLM Observability Guide covers logging design and the AI Gateway and LLM Gateway Guide covers where to enforce.

Metrics: share of AI endpoints with per-caller identity and prompt logging; number of callers exceeding repetition thresholds per week and time to review; share of API keys allowed to request log probabilities; rate of non-existent entities in your model's answers per use case, tracked across model versions; count of debug or score fields found in production responses (target zero).

Mitigations

ATLAS links four mitigations to AML.T0062. The table lists them with the practical control each translates to.

Beyond the ATLAS mitigations

• Ground the model: OWASP LLM09:2025 Misinformation recommends retrieval from verified sources to reduce hallucination. Supplying an approved package list, an internal API catalogue and a verified directory of your own domains and support addresses leaves less for an attacker to harvest.
• Strip debug output: review API responses, HTTP headers and client-side logs for scores, model identifiers and routing data, and remove anything the client does not use. Verbose logging should never be switchable by end users.
• Check existence in the agent loop: TrendAI research from Trend Micro (June 2025) found coding agents that checked package existence through search, documentation lookups or test failures produced fewer invented names, while noting that validation reduces rather than eliminates the risk.

Run the discovery yourself

The most effective response to AML.T0062 is to do it first, inside an authorised programme, and act on the results.

1. Collect real questions: sample the prompts your developers, employees and customers actually send (with privacy review), plus the questions about your own products and brand that outsiders are likely to ask.
2. Run them repeatedly: send each question to the models you deploy and the main public assistants, several times at production settings, and extract every package name, command, URL, organisation name and email address.
3. Check each entity: look up packages in the relevant registry, domains in DNS and WHOIS, and addresses against your verified directories. Record names that do not exist and how often they recur.
4. Act on recurring misses: block the names in your package proxy and protective DNS, add correct answers to grounding data, and, where it is your namespace or brand, reserve the name or prepare a takedown. The Domain Portfolio and Lookalike Domain Management Guide covers the domain side.
5. Repeat on change: rerun after every model, system prompt or retrieval change, and track the hallucination rate as a release metric.

Never register a hallucinated name on a public registry to "test" it outside an authorised research programme. The AI Red Teaming Guide covers scoping.

How to test your detection

1. Replay a harvesting pattern: in a test tenant, script a few hundred templated package and login-page questions from one test identity, repeat a subset many times and vary temperature. Confirm the gateway logs it, the repetition alert fires and the rate limit applies.
2. Request what you should not return: call your endpoints asking for log probabilities, top alternatives and debug fields with a standard client key. Each should be refused or ignored for keys not scoped for it.
3. Inspect responses end to end: capture full HTTP responses, headers and client logs from production-like builds and confirm no scores or model internals leak.
4. Roll back safely: deploy new rate limits and output checks in monitor-only mode first, measure impact on legitimate heavy users, then enforce with per-client exceptions.

Common failure patterns

• Shared API keys: every caller looks the same, so one harvester cannot be distinguished or cut off.
• Logging parameters but not prompts, or the reverse: both are needed to see the same question repeated with varied sampling settings.
• Exposing provider features by default: a proxy that passes every upstream parameter through hands callers log probabilities and logit bias they were never meant to use.
• Debug fields in production: scores, model names and routing data left in headers or client logs.
• Measuring accuracy without measuring invented entities: a release can pass quality tests while inventing more package names and domains than the last one.

Framework mapping

Practitioner checklist

• Give every caller of your AI endpoints its own identity or key, with its own quota.
• Log prompts, sampling parameters and responses at the AI gateway, with retention long enough to see repetition.
• Alert on high-volume templated questions and repeated identical prompts with varied sampling settings.
• Disable log probabilities, top alternatives and logit bias for callers that do not need them.
• Remove scores, model identifiers and debug fields from responses, headers and client logs.
• Check package names, domains and addresses in model output before returning them.
• Run your own hallucination discovery on every model and prompt release and block or reserve recurring invented names.
• Ground assistants on approved package lists and verified domain and contact directories.

Standards and references

• MITRE ATLAS: Discover LLM Hallucinations (AML.T0062)
• MITRE ATLAS: Discover AI Model Outputs (AML.T0063)
• MITRE ATLAS: ChatGPT Package Hallucination (AML.CS0022)
• OWASP: LLM09:2025 Misinformation
• Lasso Security: Diving into AI Package Hallucinations
• arXiv: Spracklen et al., A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs
• Netcraft: Large Language Models are Falling for Phishing Scams

Related NHI Mgmt Group resources: MITRE ATLAS Guide, MITRE ATLAS Mitigations Guide, Slopsquatting and Package Hallucination Guide, AI Hallucination Business Risk Guide, MITRE ATLAS AML.T0024 Exfiltration via AI Inference API.