OWASP LLM09 Misinformation Guide: Hallucination Risk

A language model that states something false with full confidence is not malfunctioning, it is doing what it was built to do, and the security risk comes from the systems and people that act on that output without checking it. OWASP lists this as LLM09:2025 Misinformation in the Top 10 for LLM Applications, covering hallucination, bias and incomplete information, and the overreliance that turns a wrong answer into a wrong decision. This guide works through the OWASP entry, turns its eight mitigations into engineering and governance requirements, and covers how to evidence them for an assessor. It is written for AppSec teams, AI product owners and risk functions that ship chatbots, copilots, coding assistants or agents. Business loss, liability cases and the verification tier model are covered in the AI Hallucination Business Risk Guide; this page stays close to the OWASP control set.

Key takeaways

Misinformation is a security risk, not only a quality issue. OWASP's first attack scenario is package hallucination: attackers register the library names that coding assistants invent, and developers install them.
No attacker is required for real harm. OWASP's second scenario is a medical chatbot that gives poor advice and leaves its operator liable, with no malicious actor involved.
Overreliance is the multiplier. OWASP treats excessive user trust as part of the risk, so interface design, labelling and reviewer training are controls, not extras.
Grounding reduces errors but does not remove them. Retrieval from verified sources is OWASP's first mitigation, yet a grounded answer can still misread its source, which is why citation checks and automatic validation sit alongside it.
Machine consumers need the strictest controls. When output feeds code, configuration or an agent's tool call, there is no human to notice the error, so validation must be automatic and fail closed.

What it is and who it applies to

OWASP defines the risk as LLMs producing "false or misleading information that appears credible", leading to security breaches, reputational damage and legal liability. The entry names three causes. Hallucination is the main one: the model fills gaps using statistical patterns and produces content that sounds right but is fabricated. Bias and incomplete information in the training data also contribute. Overreliance, where users trust output without verifying it, is what makes the error matter. NIST AI 600-1 describes the same core problem as confabulation, "the production of confidently stated but erroneous or false content".

The official risk name, confirmed on genai.owasp.org, is LLM09:2025 Misinformation. The 2023/24 edition had a separate Overreliance entry; the 2025 entry folds it into Misinformation, which matches this page's title. For how LLM09 relates to the other nine risks, see the OWASP Top 10 for LLM Applications Guide.

Who consumes the output	How misinformation causes harm	Exposure level
Customers (support chatbot, public assistant)	Wrong policy, price or eligibility statements the organisation may be held to	High: external reliance, legal and reputational exposure
Professionals (legal, clinical, financial drafting)	Fabricated citations, unsupported claims, misrepresented expertise	High: professional duty sits with the human signer
Developers (coding assistants)	Non-existent or insecure packages and APIs, insecure code patterns	High: direct path to supply chain compromise
Agents and automation	Hallucinated tool names, parameters, identifiers or findings acted on automatically	Highest: no human checkpoint by default
Internal staff (search, summarisation)	Wrong summaries of policy or data reused in decisions	Medium: depends on what the summary feeds

LLM09 has two neighbours. When false output is the result of planted data, the cause is LLM04 Data and Model Poisoning; see the OWASP LLM Top 10 LLM04 Data and Model Poisoning Guide. When output is passed unvalidated into a browser, shell or query, the failure is LLM05; see the OWASP LLM Top 10 LLM05 Improper Output Handling Guide.

Structure and key requirements

The OWASP entry has a description, four common examples of risk, eight prevention and mitigation strategies and two example attack scenarios.

The four common examples

Factual inaccuracies: incorrect statements that lead users to bad decisions. OWASP cites Air Canada's chatbot, which gave a traveller wrong information; the airline was successfully sued.
Unsupported claims: baseless assertions, especially harmful in healthcare or legal settings. OWASP cites fabricated legal cases submitted to a court.
Misrepresentation of expertise: the model appears to understand a complex subject and, in OWASP's health example, suggested uncertainty where there was none, implying unsupported treatments were still under debate.
Unsafe code generation: suggestions of insecure or non-existent libraries that introduce vulnerabilities when trusted without checking.

The eight mitigations, translated into requirements

#	OWASP mitigation	What to build	Limit to plan for
1	Retrieval-Augmented Generation from verified sources	Answers grounded in an approved, owned corpus, with citations returned to the user	The model can still misquote or over-generalise its source; corpus quality becomes critical (see LLM04 and LLM08)
2	Model fine-tuning (OWASP names parameter-efficient tuning and chain-of-thought prompting)	Domain tuning and prompting patterns that improve accuracy for the specific task	Improves the rate, does not remove the failure mode; re-evaluate after every change
3	Cross-verification and human oversight	Fact-checking steps for critical content, with reviewers trained not to over-trust AI output	Reviewers fatigue and rubber-stamp; sample their work
4	Automatic validation mechanisms	Schema checks, system-of-record lookups, citation resolvers, package existence and allowlist checks	Only checks what can be checked mechanically
5	Risk communication	Documented harms and limitations, communicated to users	Disclaimers alone did not protect deployers in the cases OWASP cites
6	Secure coding practices	Code review, SAST and dependency controls applied to AI-generated code exactly as to human code	Volume of generated code can outpace review capacity
7	User interface design	Labelling of AI-generated content, content filters, stated intended-use limits	Labels fade into the background with repeated exposure
8	Training and education	User training on limitations and verification, domain-specific for expert users	Needs refreshing as tools and models change

The two attack scenarios

The first scenario is the one security teams should own. Attackers query popular coding assistants to find package names they commonly invent, publish malicious packages under those names to public registries, and wait for developers to install them. The scale is not small: a USENIX Security 2025 paper by Spracklen and colleagues generated 576,000 code samples across 16 models and found hallucinated packages averaged at least 5.2% for commercial models and 21.7% for open-source models, with 205,474 unique invented names. See the Slopsquatting and Package Hallucination Guide. The second scenario, the medical chatbot, is the reminder that LLM09 can produce liability without any attacker at all.

Identity and NHI requirements

Misinformation becomes an identity problem when the output drives an action taken with a credential. An agent that hallucinates a customer ID, a resource name or a tool parameter will act on it with whatever permissions its service account or token holds. The controls are the familiar ones, applied to a new failure source.

Least privilege for agent credentials: scope each agent's identity to the specific resources and actions its task needs, so a hallucinated target outside that scope fails at authorisation. See the OWASP LLM Top 10 LLM06 Excessive Agency Guide.
Validate identifiers against the system of record: before an agent writes, deletes or sends, resolve every account, ticket or resource identifier it produced against the authoritative source rather than trusting the generated value.
Human approval for consequential actions: payments, access grants, deletions and external messages need a human checkpoint where an error would be costly. See the Human-in-the-Loop for AI Agents Guide.
Package installation identities: build systems and developer machines should resolve dependencies through an internal registry proxy whose service identity can only fetch approved or vetted packages, so a hallucinated name does not reach the public registry unchecked.
Attribution in logs: record which model, prompt version and agent identity produced each automated action, so an incident can be traced back to the generated output that caused it.

Implementation steps

Inventory LLM use cases by consumer. For each application record who or what consumes the output (customer, professional, developer, agent, internal user) and what decisions or actions it feeds. Consumer type drives everything below.
Set a verification requirement per use case. Low-consequence internal drafting may need only labelling; customer-facing statements of policy need approved content or grounded answers with citations; automated actions need automatic validation and, above a threshold, human approval.
Build an evaluation set before go-live. Write a golden set of questions with known correct answers, including questions the system should refuse or say it cannot answer. Measure accuracy, groundedness and abstention, not accuracy alone.
Ground and cite. Connect answers to an approved corpus with clear ownership, return citations, and resolve each citation automatically to confirm it exists and supports the claim (NIST AI 600-1 action MS-2.5-003 calls for reviewing and verifying sources and citations in outputs).
Return approved text where precision matters. For refund rules, eligibility, legal terms and safety instructions, retrieve and display the approved wording rather than letting the model paraphrase it.
Add automatic validators. Schema validation for structured output, system-of-record lookups for identifiers and prices, package existence and allowlist checks for code, and fail-closed behaviour when a validator cannot run.
Pilot in shadow mode. Run the system alongside the existing process, compare its answers with what staff actually said or did, and review disagreements before exposing it to users.
Release in rings with labelling. Start with internal users, then a small external group, with AI-generated content labelled and a visible way to report wrong answers.
Re-evaluate on every change. A new model version, prompt template or corpus update reruns the golden set before release; regressions block it.
Define rollback. Keep the ability to fall back to the previous model and prompt version, or to static approved content, within minutes if error reports spike.

Decision guidance

Generated or retrieved answer: if a wrong answer creates a commitment (price, policy, legal position), retrieve approved text. Generate only where some imprecision is acceptable and the user can verify.
Human review or automatic validation: automatic validation scales and does not tire, but only checks what can be expressed as a rule; human review catches meaning but needs sampling to stay honest. High-consequence uses need both.
Small organisations: start with steps 1, 2, 5 and 6, plus an internal package proxy for developers. Approved-text responses and labelling are cheap and cover most customer-facing risk.
Large organisations: add a central evaluation service, groundedness scoring in production, reviewer quality sampling and contract terms with model providers covering accuracy claims and change notification.

Evidence and assessment

Assessors will want to see that the controls exist, that they run on every release, and that production behaviour is measured. Keep the golden set, its results per release, validator configurations, reviewer sampling records and user error reports.

Area	What to log	Signal worth investigating
Answers	Prompt, retrieved sources, response, citations, model and prompt version	Answers with no supporting source; citations that do not resolve
Validators	Each check, its result and whether the response was blocked	Rising block rates after a model or corpus change
Packages	Dependency requests through the registry proxy, including denied names	Requests for names that do not exist or were registered very recently
Agent actions	Tool call, parameters, agent identity, approval record	Calls with identifiers that failed system-of-record lookup
User feedback	Error reports and corrections	Clusters of reports on one topic or after one release

Metrics

Grounded-answer rate: share of production answers fully supported by retrieved sources, on a sampled basis.
Citation validity: share of citations that resolve and support the claim made.
Confident error rate: share of golden-set answers that are wrong and not hedged or refused, tracked per release.
Escaped errors: confirmed wrong answers reported by users or downstream teams per month.
Blocked hallucinated packages: dependency requests denied because the package does not exist or is not approved.
Review coverage: share of high-consequence outputs that received a recorded human or automatic check.

Mapping to other frameworks

IDs were checked against MITRE ATLAS 5.6.0, NIST SP 800-53 Rev. 5 and NIST CSF 2.0 datasets. OWASP's own entry points to ATLAS AML.T0048.002 Societal Harm.

Framework	Reference	How it relates to LLM09
MITRE ATLAS	AML.T0062 Discover LLM Hallucinations; AML.T0060 Publish Hallucinated Entities	The package hallucination attack chain: finding invented names, then registering them
MITRE ATLAS	AML.T0048 External Harms (AML.T0048.002 Societal Harm)	Harm caused by false output reaching the public
MITRE ATLAS	AML.T0070 RAG Poisoning	Deliberately induced misinformation through the retrieval corpus
NIST AI 600-1	Confabulation risk; actions MS-2.5-003, MS-2.3-002, MS-2.5-001, MS-2.6-005, GV-5.1-002	Verify sources and citations; validate capability claims empirically; avoid extrapolating from anecdotal tests; monitor outputs and recover from errors; document interactions for users
NIST SP 800-53 Rev. 5	SI-10, SA-11, SI-4, RA-3, AT-2, SR-3	Input and output validation, developer testing, monitoring, risk assessment, training, supply chain controls for dependencies
NIST CSF 2.0	ID.RA-01, PR.PS-06, PR.AT-01, DE.CM-09, DE.AE-02	Identify weaknesses, secure development, awareness training, monitoring and analysis
ISO/IEC 27001:2022 Annex A	A.8.28, A.8.25, A.6.3	Secure coding (including generated code), secure development life cycle, awareness and training
ISO/IEC 42001:2023	AI management system	Governance home for intended use, user information and performance monitoring

For the governance frameworks themselves, see the NIST AI RMF Guide and the AI Governance Guide.

Common pitfalls

Relying on a disclaimer. A "may contain errors" banner is OWASP's risk communication mitigation, not a substitute for verification. In the Air Canada case OWASP cites, the airline was successfully sued over what its chatbot told a traveller.
Letting the model paraphrase policy. Refund rules, eligibility criteria and legal terms drift when reworded. Retrieve and display approved text instead.
Checking conclusions but not references. Fabricated citations look plausible and are the easiest error to miss; resolve every citation automatically.
Installing suggested packages unchecked. Developers who copy an assistant's install command straight to a terminal are the target of OWASP's first scenario. Route installs through a vetted registry proxy. See the Malicious Package and Dependency Confusion Guide.
Measuring accuracy alone. A system that guesses instead of abstaining can score well on accuracy and still produce many confident errors. Track abstention and confident error rates too.
Agents acting on generated identifiers. An agent that takes a hallucinated ID as fact will act on the wrong record with real permissions. Resolve identifiers against the system of record and keep agent credentials narrowly scoped.

Practitioner checklist

Inventory every LLM use case with its output consumer and the decisions or actions it feeds.
Assign a verification requirement to each use case and record it with the system owner.
Build a golden evaluation set with known answers and refusal cases; rerun it on every model, prompt or corpus change.
Ground answers in an owned, approved corpus and validate every citation automatically.
Serve approved text, not generated paraphrase, for policy, pricing, legal and safety content.
Route all dependency installs through an internal registry proxy that blocks non-existent and unapproved packages.
Validate agent-generated identifiers and parameters against the system of record before any write action.
Label AI-generated content and give users a visible way to report wrong answers.
Train users and reviewers on model limitations, and sample reviewer decisions for quality.
Keep a tested rollback to the previous model, prompt version or static approved content.

OWASP LLM Top 10 LLM09 Misinformation Guide: Managing Hallucination and Overreliance Risk