AI package hallucinations are creating a new supply-chain risk

At a glance

What this is: This research shows that LLMs can hallucinate nonexistent software packages at scale, creating a supply-chain risk when developers trust model-generated dependency recommendations.

Why it matters: IAM and security teams need to treat model output as an untrusted software discovery channel because AI-assisted development can introduce dependency risk without any identity, secret, or approval control in the path.

By the numbers:

• In total we received 24.2% of hallucinations, with 19.6% of repetitiveness.
• In total we received 64.5% of hallucinations, with 14% percentages of repetitiveness.
• In three months the fake and empty package got more than 30k authentic downloads!

👉 Read Lasso Security's research on AI package hallucinations and supply-chain risk

Context

AI package hallucinations are a software supply-chain problem, not just a model-quality annoyance. When developers ask an LLM what package to use, the model can invent a plausible dependency name, and that invented package may later be searched for, installed, or mirrored into internal workflows. That creates a new trust boundary for AI-assisted development and for NHI governance around automation, tokens, and build-time access.

The practical identity issue is that the recommendation path is now part of the attack surface. A developer, CI pipeline, or internal assistant may consume model output as if it were authoritative, while the package ecosystem itself cannot distinguish a real dependency from a fabricated one until someone tries to resolve it. For security teams, this turns provenance, validation, and package allowlisting into control points that sit alongside IAM and secrets governance.

Key questions

Q: What is the biggest risk when developers rely on LLMs for package recommendations?

A: The biggest risk is that the model invents a package that sounds legitimate, and the developer treats it as real. That can lead to dependency installation, documentation drift, or even malicious package registration by an attacker who notices the false name. Teams should verify package existence, ownership, and provenance before any AI-suggested dependency is trusted.

Q: Why do hallucinated packages create supply-chain risk even when the model is not directly compromised?

A: Because the attacker does not need to break the model to exploit the output. If developers or pipelines trust an invented package name, the error becomes an intake problem and can lead to malicious registration, copy-paste installation, or internal propagation. The control point is external verification, not faith in the model’s confidence.

Q: How can security teams tell whether AI-generated package suggestions are being trusted too much?

A: Look for invented dependency names in code, tickets, build scripts, and chat threads, then trace whether they were ever validated against a registry or approved source. If teams cannot show a verification step before installation, the organisation is treating generated text as a trusted software supply signal.

Q: What should organisations do before allowing AI-generated dependencies into production?

A: They should require a controlled approval workflow that checks package existence, maintainer identity, signature or checksum, and vulnerability history. That workflow should sit between model output and installation so an invented package cannot move directly into production tooling or release pipelines.

Technical breakdown

How LLM package hallucinations become a supply-chain entry point

LLMs do not retrieve only verified package names. They generate likely completions based on training patterns, so a prompt about implementation can produce a package that sounds real but has no legitimate maintainer, release history, or repository. That matters because developers often treat the model output as a shortcut to dependency selection. Once a hallucinated package enters a build, the ecosystem search itself becomes an attack surface. An attacker can register the name, publish a malicious package, or wait for internal users to copy the false recommendation into code. The weakness is the trust given to generation, not the package registry alone.

Practical implication: require independent package verification before any AI-suggested dependency reaches a build or deployment path.

Why cross-model repetition makes hallucinations easier to exploit

A repeated hallucination across multiple LLMs is more dangerous than a one-off error because it creates the appearance of consensus. In this research, the same fictitious packages appeared across models, which means an attacker does not need to bias one model only. They can benefit from shared training patterns and common prompting behaviour. For practitioners, that undermines the assumption that using a different model automatically reduces supply-chain exposure. The real issue is whether the organisation has a verification step that does not depend on the model being queried. Without that step, hallucination becomes reusable guidance rather than isolated noise.

Practical implication: add a human or automated verification gate that checks package existence, ownership, and reputation outside the model.

Why fake packages can still attract real downloads

The research shows that even an empty package name can attract authentic downloads when it looks credible in developer workflows. That is a classic supply-chain trust failure: a name alone can trigger scanning, copy-paste installation, or internal documentation reuse. In identity terms, the build and developer process is acting on unauthenticated information. The package registry cannot protect teams if the decision to install happens before provenance is checked. This is why hallucination is not just a model problem. It becomes a governance problem once model output influences software acquisition and execution.

Practical implication: tie dependency admission to provenance checks, signed packages, and approved-source policies before installation.

Breaches seen in the wild

• LiteLLM PyPI package breach — LiteLLM PyPI supply chain attack, credentials stolen from users.
• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI package hallucination is a supply-chain trust failure, not a content-quality bug. The article shows that model output can create false dependencies that developers may search for, install, or document. That shifts the attack surface from language generation into software acquisition and build-time trust. The practitioner implication is simple: any AI-suggested package must be treated as untrusted until provenance is independently confirmed.

Cross-model repetition turns hallucinated packages into repeatable attack material. When the same invented package appears across multiple models, attackers do not need a single-model compromise to weaponise the output. They can rely on shared training bias and repeated recommendation patterns. That weakens the assumption that model diversity is a compensating control. Practitioners should assume that hallucinated package names can recur and can be operationally useful to attackers.

Identity governance now extends into developer tooling and software intake decisions. This research shows that the person or pipeline consuming model output becomes part of the control path. If a build system, assistant, or engineer accepts a hallucinated dependency, the organisation has effectively granted trust to unauthenticated software advice. The implication is that IAM, PAM, and SDLC governance need a shared policy boundary for AI-assisted dependency selection.

Package provenance is the named control gap this research exposes. The most important failure mode is not that the model was wrong, but that the surrounding workflow did not force package verification before trust was extended. That is the governance gap attackers can exploit. Practitioners should recognise package provenance as a first-class control, not a downstream review task.

Dependency hallucination creates identity risk through tool-chaining. Once a fabricated package is copied into documentation, ticketing, or CI configuration, the error can propagate through multiple systems without a human re-check. That is how a model suggestion becomes a persistent operational artefact. The practitioner implication is to bind package approval to controlled registries and review workflows before automation amplifies the mistake.

From our research:

• 52 real-world breaches show that identity failures become exploitable when trust is granted before provenance is verified, according to The 52 NHI breaches Report.
• In our research, supply-chain and secret exposure patterns recur across many incidents, which is why package validation must be treated as a governance control, not a developer preference.
• For a broader control map, read Top 10 NHI Issues for the identity gaps that most often turn automation into exposure.

What this signals

Package hallucination is now part of the software governance boundary. If AI systems can invent dependencies with enough plausibility to be downloaded and reused, then build security must assume that recommendation quality is not the same as provenance. Security leaders should treat AI-assisted dependency intake as a controlled workflow, not an informal developer shortcut, and align it with Top 10 NHI Issues where trust and lifecycle control intersect.

The broader signal is that identity programmes will need to govern the systems that make software choices, not just the identities that run software. As AI assistants become embedded in development, the line between recommendation and execution narrows, which means approval workflows, source allowlists, and artifact validation need to sit closer to the point of use.

Provenance debt: this is the control gap that grows when teams trust model-generated names before checking ownership, signatures, or repository history. The problem is not limited to package hallucinations. Any AI-mediated intake path can create the same debt if the organisation does not force a verification step before action.

For practitioners

• Verify every AI-suggested dependency against a trusted registry Check package existence, maintainer identity, release history, and repository ownership before accepting any model-recommended dependency into code, documentation, or a ticket.
• Add a provenance gate before installation Block builds unless the package comes from an approved source, has a verifiable signature or checksum, and matches an internal allowlist for the language ecosystem in use.
• Train developers to cross-check model output Make AI-generated package names part of secure coding review. Ask teams to validate unfamiliar dependencies through repository documentation, community activity, and vulnerability history before use.
• Monitor for hallucinated dependency names in internal tooling Search code repos, chat logs, and build pipelines for invented package names so you can remove false dependencies before they become repeatable internal guidance.

Key takeaways

• LLM package hallucinations create a real supply-chain exposure because invented dependencies can be searched for, copied, and installed before anyone checks whether they exist.
• The research’s scale matters because hallucinations appeared across multiple models and, in one case, a fake package still attracted more than 30,000 authentic downloads.
• Teams should enforce provenance checks, approved-source controls, and dependency verification before AI-generated package names can influence code or production workflows.

Key terms

• AI Package Hallucination: An AI package hallucination occurs when a language model invents a software dependency that sounds plausible but has no legitimate existence or trustworthy provenance. In practice, this can mislead developers into searching for, installing, or documenting a false package, creating a supply-chain risk around unverified software intake.
• Dependency Provenance: Dependency provenance is the evidence trail that shows where a package came from, who maintains it, and whether it can be trusted. Security teams rely on provenance to distinguish legitimate software from fabricated or tampered dependencies before they are allowed into build or production workflows.
• Package Intake Control: Package intake control is the set of checks that decide whether a dependency may enter an organisation’s software pipeline. It typically includes registry validation, maintainer verification, checksum or signature checks, and approved-source rules so that untrusted or hallucinated packages do not move into production.

Deepen your knowledge

AI package hallucinations and dependency provenance are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If your organisation is building controls for AI-assisted development or software intake, it is worth exploring.

This post draws on content published by Lasso Security: Diving Deeper into AI Package Hallucinations. Read the original.