Package hallucination occurs when an LLM suggests a package name that does not exist, creating a gap attackers can exploit. If the suggested name is later registered by an attacker, developers may unknowingly install malicious code from a seemingly plausible source.
Expanded Definition
Package hallucination is a supply chain and software trust problem that appears when an AI system produces a plausible but non-existent package name, dependency, or module reference. In practice, the risk is not the hallucination itself but the human workflow that treats the suggestion as authoritative and installs or approves it without verification. This makes package hallucination especially relevant in developer tooling, code assistants, autonomous build pipelines, and other AI-assisted software delivery paths.
The concept sits close to dependency confusion, typosquatting, and poisoned package registries, but it is distinct because the initial failure originates in model output rather than an attacker’s naming choice. Industry usage is still evolving, and no single standard governs the term yet, so guidance should be read as a security pattern rather than a formal category. For governance context, controls around software integrity, configuration management, and change control in NIST SP 800-53 Rev 5 Security and Privacy Controls are the closest fit.
The most common misapplication is assuming the LLM merely made a harmless suggestion, which occurs when teams skip package verification and trust the output because it looks syntactically credible.
Examples and Use Cases
Implementing controls against package hallucination rigorously often introduces friction in developer velocity, requiring organisations to weigh automation speed against dependency verification overhead.
- An AI coding assistant recommends a library name that resembles a real open-source package, and a developer copies it into the build file without checking the registry.
- A CI pipeline uses generated dependency instructions from an LLM, but no allowlist or package provenance check is in place, so a later malicious registration could be pulled into the software supply chain.
- A security engineer uses an AI tool to draft remediation steps and receives a fake package suggestion that appears credible because it matches the application’s technology stack.
- An internal developer portal accepts generated code snippets, but the organisation has not required review against trusted package sources such as official registries or signed metadata.
- A prompt-enabled build agent proposes an install command for a niche utility, and the team only discovers the package never existed after troubleshooting a failed deployment.
For teams building AI-enabled software workflows, this is why dependency checks, trusted-source verification, and human review remain essential. Security baselines such as NIST SP 800-53 Rev 5 Security and Privacy Controls are useful when mapping review gates, provenance checks, and change approval steps to operational controls.
Why It Matters for Security Teams
Package hallucination matters because it turns a productivity feature into a software supply chain exposure. If a team trusts generated package names, it can introduce malicious code, unstable builds, or unreviewed dependencies into production. That creates security, integrity, and availability risk at the same time, particularly when code generation is integrated into automated delivery pipelines.
For security teams, the issue is less about whether an LLM is “wrong” and more about whether downstream processes can detect and reject unverified artifacts. This is especially relevant where AI agents have execution authority, because a package hallucination can become a direct action rather than a draft suggestion. Teams should pair registry validation, provenance checks, dependency allowlists, and review gates with the identity of the tool or service making the request. In NHI and agentic AI environments, the same discipline used to govern privileged automation should be applied to model-generated software actions.
Organisations typically encounter the real impact only after a failed build, unexpected dependency pull, or post-deployment incident, at which point package hallucination becomes operationally unavoidable to address.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST CSF 2.0, NIST SP 800-53 Rev 5 and NIST AI RMF set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | PR.IP-1 | Protects the term through secure configuration and change management in software delivery. |
| NIST SP 800-53 Rev 5 | CM-3 | Configuration change control governs unauthorized dependency introduction and package drift. |
| NIST AI RMF | GOVERN | Defines governance expectations for AI outputs that influence operational decisions. |
| OWASP Agentic AI Top 10 | Addresses agentic tool use and unsafe model output in software workflows. | |
| OWASP Non-Human Identity Top 10 | Covers trust in non-human actors and their actions in software supply chains. |
Treat model-generated package names as untrusted until validated against authoritative sources.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on August 2, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
