Treat invented dependencies as supply chain risk, not just a code-quality defect. Validate that every referenced package exists, that its provenance is legitimate, and that build pipelines reject unapproved additions. If attackers can later register the same name, the assistant’s output can become an entry point into the software supply chain.
Why hallucinated package references are a supply-chain issue, not just a coding mistake
When an AI coding tool invents a package name, the risk is not limited to a bad suggestion in the editor. The real issue is that the tool has created a dependency decision that may later be trusted by a build, a reviewer, or an automation pipeline. If the name is later registered by an attacker, the same hallucinated reference can become a delivery path for malicious code.
That is why the first response should be to treat the reference like any other untrusted dependency claim. The right question is not “did the model make an error?” but “would this package be allowed into our software supply chain if someone copied it into source control or a lockfile?”
Teams should also assume that a hallucinated package can be copied across environments faster than it can be reviewed. A developer may paste it into a manifest, another tool may resolve it automatically, and a CI job may install it before anyone notices that the package was never real.
What teams should validate before they trust any generated dependency
Every package reference should be checked for existence, publisher legitimacy, and provenance before it is accepted. That means verifying the name against the canonical registry, confirming the source repository or signing metadata where available, and rejecting additions that do not come from approved channels. The control is simple: if the dependency cannot be proven, it does not enter the build.
For AI-assisted coding workflows, this check should happen at the point where code is introduced, not after release. AI Coding Agents Security Guide covers the supply-chain and sandboxing patterns that matter here, including hallucinated packages and the need to constrain assistant output before it reaches tooling.
This validation step is especially important because package ecosystems reward speed and name reuse. A plausible-looking package string can be enough to mislead a hurried reviewer, and an attacker who later claims that name can turn a simple typo or hallucination into a malicious installation event.
How to stop hallucinated dependencies from becoming production risk
The most effective defense is to make unapproved dependency introduction mechanically hard. Build pipelines should reject unknown packages by default, require allowlisted registries or namespaces, and fail closed when dependency metadata is missing or inconsistent. Where teams use lockfiles, those lockfiles should be generated from trusted sources and reviewed like code.
Dependency policy also needs to account for provenance, not just existence. An approved package from an untrusted source can be as dangerous as a fake one. For broader software and open-source supply-chain controls, OpenSSF is a useful reference point for registry hygiene, package integrity, and secure consumption practices.
At scale, the main mistake is letting AI tools bypass the same controls used for human-authored changes. If developers can add dependencies through natural language and the pipeline only checks syntax, the organisation has outsourced trust to the model. The safer pattern is to force generated suggestions through the same dependency approval, provenance, and review gates that apply to every other external component.
Risk and Threat Considerations
Hallucinated package references create a direct software supply-chain exposure because they can be mistaken for legitimate dependencies, then later resolved into real attacker-controlled packages. The risk increases when assistant output is copied into manifests, lockfiles, or build scripts without registry verification.
Failure mechanism: A nonexistent package name is treated as trustworthy input, then an attacker registers the name or a lookalike, allowing malicious code to enter the build path through normal dependency resolution.
Impact: The result can be code execution, credential theft, poisoned builds, or downstream compromise of deployed software and developer environments.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Non-Human Identity Top 10 addresses the attack and risk surface, while SLSA, CIS Controls v8, NIST SP 800-53 Rev 5 and OWASP ASVS set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Non-Human Identity Top 10 | NHI-02 — Secret Leakage | Generated package hallucinations can expose unsafe dependency handling in AI-assisted builds |
| NHI-03 — Vulnerable Third-Party NHI | Package hallucinations can introduce risky third-party components into the supply chain | |
| NHI-06 — Insecure Cloud Deployment Configurations | Build and deployment pipelines must fail closed on unapproved dependency changes | |
| Recommendation — Reject unverified dependency additions before they reach source control or build steps. Verify third-party package provenance and block unapproved registries. Enforce pipeline checks that reject unknown or unauthorised dependencies. | ||
| SLSA | Supply chain integrity | The issue is software supply-chain integrity, especially dependency provenance and build trust |
| Recommendation — Require provenance checks and trusted sources for all new dependencies. | ||
| CIS Controls v8 | CIS-16 — Application Software Security | AI-generated dependencies should be governed as software supply-chain inputs |
| Recommendation — Apply software security review gates to dependency changes from AI tools. | ||
| NIST SP 800-53 Rev 5 | SA-12 — Supply Chain Protection | Untrusted dependencies are a supply-chain risk that needs supplier and component controls |
| SI-7 — Software, Firmware, and Information Integrity | Rejecting hallucinated packages depends on integrity checks for supplied artifacts | |
| Recommendation — Screen and approve external components before they are accepted into builds. Verify dependency integrity and block tampered or unverified components. | ||
| OWASP ASVS | V15 — Secure Coding and Architecture | AI-generated dependencies are part of secure design and dependency control decisions |
| V13 — Configuration | Build pipelines should enforce approved dependency configuration and source restrictions | |
| Recommendation — Review dependency introductions as part of secure architecture and coding practice. Lock dependency sources and fail builds on unapproved additions. | ||
Practitioner Guidance
What to verify: Confirm that every assistant-suggested dependency exists in the expected registry, belongs to the expected publisher or namespace, and is approved for the target environment before it is merged.
Decision rule: If a package is suggested by an AI tool but cannot be provenance-checked quickly, treat it as untrusted input and block it until a human reviews the dependency choice.
What good looks like: Build and review workflows make it easier to reject invented dependencies than to accept them, and every external package has a traceable source, version, and approval path.
Practitioner takeaway: The goal is not to prevent hallucinations entirely, but to ensure they cannot cross the trust boundary from generated text into executable software.
Related resources from NHI Mgmt Group
- Who should be accountable for protecting developer workstations when teams use multiple package ecosystems and AI coding tools?
- How should teams respond when AI coding tools can ingest untrusted registry content?
- How should security teams respond when developer tools and AI coding assistants share the same trust model?
- How should security teams govern AI coding tools that create non-human identities?