AI Hallucination Business Risk: Liability and Controls

Generative AI systems produce confident, fluent answers that are sometimes false, and organisations that publish, file or act on those answers without checking them carry the consequences. Courts have sanctioned lawyers for citing cases that never existed, a tribunal held an airline responsible for what its chatbot told a customer, a consultancy partially refunded a government contract over invented references, and attackers register the software packages that coding assistants invent. This guide treats hallucination as a business and security risk rather than a curiosity: where it causes loss, how liability has been assigned so far, which controls reduce it, how to verify output at the right depth, and how to measure whether the controls work. It is written for security, risk and AI platform teams, and for the product and legal owners who sign off AI features. Where it discusses liability it describes published decisions and is not legal advice.

Key takeaways

Hallucination cannot be switched off. NIST AI 600-1 calls it confabulation, "the production of confidently stated but erroneous or false content", and lists it among twelve generative AI risks; plan controls on the basis that it will happen.
The deployer usually owns the output. In the published cases the organisation that put the AI in front of customers or a court was held responsible, not the model or its vendor.
Verification depth should follow consequence. Tier AI use cases by the harm a wrong answer can cause and require proportionate checks, from labelling to mandatory human sign-off against a primary source.
Some hallucinations are a supply chain attack surface. Invented package names, API endpoints and tool calls can be pre-registered or exploited by attackers, so they belong in security engineering, not only in content quality.
Measure it like any other defect. Track grounded-answer rates, citation validity, abstention and escaped errors per use case, and treat a hallucination that reaches a customer or a filing as an incident.

What hallucination is and why it happens

OWASP's Top 10 for LLM Applications 2025 places hallucination under LLM09:2025 Misinformation: the model produces false or misleading information that appears credible, by filling gaps with statistical patterns rather than understanding. OWASP pairs it with overreliance, where users trust generated content without verifying it, which is what turns a wrong answer into a wrong decision. NIST AI 600-1 (the Generative AI Profile of the AI RMF) uses the term confabulation and treats it separately from information integrity, which covers the wider ecosystem of false and misleading content.

OpenAI's September 2025 research note on why language models hallucinate argues that the cause is partly structural: pretraining predicts the next word with no true or false labels, so arbitrary low-frequency facts (a person's birthday, a case citation) cannot be learned reliably, and accuracy-only evaluations reward guessing over saying "I don't know". Its published SimpleQA comparison illustrates the trade-off: one model abstained on 52% of questions with a 26% error rate, while an older model abstained on 1% and was wrong 75% of the time, with similar accuracy. The practical lesson for buyers is that a headline accuracy figure says little about how often a system is confidently wrong.

Hallucination type	What it looks like	Typical business impact
Fabricated fact or policy	A chatbot states a refund rule, price or legal requirement that does not exist	Customer compensation, regulatory complaints, reputational damage
Fabricated citation or quote	Invented cases, papers, report references or quotations attributed to real people	Court sanctions, contract refunds, defamation exposure, loss of credibility
Package or dependency hallucination	A coding assistant recommends a library that does not exist in npm or PyPI	Malware installed if an attacker registers the name (slopsquatting)
Tool or API hallucination	An agent calls a non-existent function, invents parameters or fabricates a tool result	Failed or wrong transactions, silent data corruption, unsafe actions
Misrepresented reasoning or status	An agent reports that a task, test or check succeeded when it did not	False assurance in change, audit or security workflows
Summarisation drift	A summary adds, inverts or drops material facts from a source document	Wrong decisions in legal, medical, HR or incident triage contexts

Where it has caused real loss

The cases below were read from primary decisions or reliable reporting. They show a consistent pattern: the harm came from publishing or relying on output without verification, and responsibility landed on the organisation or professional that did so.

Case	What went wrong	Outcome
Moffatt v. Air Canada, 2024 BCCRT 149 (14 February 2024)	The website chatbot told a customer he could claim a bereavement fare within 90 days after buying a full-price ticket, contrary to the airline's policy	Negligent misrepresentation; total award of CAD 812.02; the tribunal rejected the argument that the airline was not responsible for its chatbot
Mata v. Avianca, No. 22-cv-1461 (S.D.N.Y., 22 June 2023)	A filing cited judicial opinions generated by ChatGPT that did not exist; the lawyers did not withdraw it when questioned	USD 5,000 penalty under Rule 11 or inherent authority, letters to the judges falsely named, firm held jointly responsible
Ayinde v. Haringey and Al-Haroun v. Qatar National Bank, [2025] EWHC 1383 (Admin) (6 June 2025)	Five non-existent cases in one set of grounds; 18 of 45 citations in the other did not exist	Wasted costs of GBP 2,000 each against counsel and a law centre, referrals to the Bar Standards Board and Solicitors Regulation Authority, warning to leaders of firms and chambers
Deloitte report for Australia's Department of Employment and Workplace Relations (2025)	An AUD 440,000 report contained invented references and a fabricated quote from a court judgment, found by an academic reviewer	Corrected report published and a partial refund; Deloitte disclosed an Azure OpenAI GPT-4o tool chain and said findings were unaffected
New York City MyCity business chatbot (reported March 2024)	Told businesses they could take workers' tips, refuse cash and refuse housing vouchers, all contrary to law, and gave inconsistent answers to the same question	Public reporting by The Markup; the city described it as a pilot that would improve

The Air Canada decision is the clearest statement of deployer responsibility so far: the tribunal found the chatbot was part of the airline's website, that the airline was responsible for all information on it whether static or generated, and that it had not taken reasonable care to ensure the chatbot was accurate. See the Air Canada chatbot ruling for the full case analysis.

Liability: how responsibility has been assigned

This section summarises published decisions and frameworks; it is not legal advice, and outcomes depend on jurisdiction, contract and facts.

Deployer responsibility for customer-facing output: the Air Canada tribunal treated chatbot statements as the company's own representations. The tribunal rejected the argument that the customer should have cross-checked the policy page elsewhere on the same site.
Professional duties are not delegated to the tool: the Mata court said using a reliable AI tool is not inherently improper but lawyers keep a gatekeeping duty to ensure filings are accurate. The English Divisional Court in Ayinde said lawyers must check AI-assisted research against authoritative sources, that relying on a client's research is no excuse, and that heads of chambers and managing partners must take practical steps so everyone understands these duties.
Contractual exposure: the Deloitte case shows that a client can seek money back when AI-assisted deliverables contain fabricated material, even where the conclusions are said to stand. Contracts should state whether AI tools may be used, what disclosure is required and what quality assurance applies.
Product liability: the EU's revised Product Liability Directive, Directive (EU) 2024/2853 of 23 October 2024, repeals the 1985 directive and brings software within its scope, which widens the routes by which defective AI output in a product can lead to claims. See the AI Liability and Product Liability Guide.
Vendor terms rarely cover you: model providers' terms typically disclaim accuracy and place responsibility for use on the customer. Do not assume an indemnity for output accuracy unless the contract says so in writing.

Hallucination as a security risk

Most hallucinations are quality failures, but several are directly exploitable, and attackers already look for them. MITRE ATLAS records AML.T0062 Discover LLM Hallucinations as an adversary technique, and OWASP LLM09 describes the scenario where attackers find package names that coding assistants commonly invent and publish malicious packages under those names.

Package hallucination (slopsquatting): Spracklen and colleagues' study for USENIX Security 2025 generated 576,000 code samples from 16 models and found average package hallucination rates of at least 5.2% for commercial models and 21.7% for open-source models, with 205,474 unique invented package names. Each is a name an attacker could register. See the Slopsquatting and Package Hallucination Guide and the Malicious Package and Dependency Confusion Guide.
Agentic tool hallucination: an agent that invents a tool result ("payment confirmed", "tests passed") or calls a real tool with invented parameters can take wrong actions under a real identity. The blast radius is set by the permissions of the agent's credentials, not by the model. See the Agentic AI Security Guide.
Hallucinated security findings: AI assistants in the SOC or code review can invent CVE numbers, misstate fixed versions or report non-existent vulnerabilities, which wastes analyst time or hides real issues. Require every finding to carry a verifiable source.
Prompt injection amplifies it: retrieved content can steer a model into confidently stating attacker-chosen falsehoods. Grounding only helps if the grounding sources are trusted. See the RAG Security Guide.

Controls that reduce hallucination risk

No single control is sufficient. OWASP LLM09 lists retrieval-augmented generation, fine-tuning, cross-verification and human oversight, automatic validation, risk communication, secure coding practices, interface design and training. The table organises them by what they actually reduce.

Control	What it reduces	Limits
Retrieval grounding (RAG) on curated sources, with answers restricted to retrieved content	Fabricated facts and policies in customer and internal assistants	Only as good as the corpus; stale or poisoned documents produce confident wrong answers
Mandatory citations with automated link and quote checks	Fabricated references and misquotes	A real citation can still be misread; check that the source supports the claim
Abstention and refusal design ("I don't know", hand-off to a human)	Confident guesses outside the system's scope	Lower answer rates; needs product owner acceptance
Structured output with schema validation	Invented fields, parameters and tool arguments	Validates form, not truth
Deterministic checks against systems of record (prices, policies, account status)	Wrong commitments to customers	Requires integration effort per fact type
Package allowlists, registry proxies and lockfiles	Installation of hallucinated or squatted packages	Needs a process for adding legitimate new dependencies
Human review against primary sources	High-consequence errors in filings, reports, advice	Reviewers drift into rubber-stamping; must be trained and sampled
Labelling and user communication	Overreliance by users	Does not, by itself, make a wrong answer harmless
Least-privilege credentials and confirmation gates for agent actions	Harm from agents acting on hallucinated plans	Adds friction to automation

Verification: matching depth to consequence

Verification is the control that the court and tribunal decisions turn on. The useful question is not "is this AI output correct" in general, but "what is the cost of this specific output being wrong, and who checks it before that cost is incurred".

Tier	Examples	Minimum verification
1. Low consequence, internal	Drafting internal notes, brainstorming, code explanation	User awareness training; AI-generated label
2. Internal decisions	Summaries feeding management decisions, triage suggestions	Source links required; spot checks by the owner; sampled quality review
3. External or customer-facing	Chatbots, marketing copy, customer emails, published reports	Grounding on approved content, deterministic checks for prices and policy, escalation path, ongoing evaluation set
4. Regulated, legal or safety-relevant	Court filings, regulatory submissions, medical or financial advice, security findings, code to production	Named human sign-off against primary sources, every citation opened and read, audit trail of what was checked
5. Autonomous actions	Agents that pay, change configuration, deploy or delete	Pre-execution validation against systems of record, human approval or hard policy limits, reversible actions, full logging

Citation checks: resolve every reference to a real document, confirm the quoted text exists at the cited location, and confirm the source supports the proposition. The Deloitte and Ayinde errors would each have been caught by opening the cited sources.
Reviewer independence: the person who prompted the output should not be the only reviewer for tier 4 material. Mata shows how a signer who did not read the cited cases inherits the error.
Evidence of review: record who verified what and against which source. That record is what demonstrates "reasonable care" if the output is later challenged.

How to implement a hallucination risk programme

Inventory AI use cases: list every place generative AI output reaches customers, regulators, courts, code or automated actions, including tools staff use on their own. Record owner, model, data sources and audience.
Tier each use case: apply the five tiers above and record the verification standard. Treat any agent with write access as tier 5 until shown otherwise.
Build an evaluation set per use case: collect 100 to 300 real questions with known correct answers, including questions the system should refuse. Score accuracy, error rate and abstention separately, as the OpenAI research recommends.
Pilot in shadow mode: run the assistant alongside the existing process, with humans answering and the AI output logged but not shown to customers. Compare error rates before go-live.
Deploy guardrails in audit mode first: turn on citation validation, schema checks and package allowlists in log-only mode, tune false positives, then enforce.
Release in rings: internal users, then a small customer segment, then general availability, with a kill switch that routes traffic back to humans or static content.
Define rollback: document how to disable the feature, how to correct statements already made to customers, and who authorises compensation, as the Air Canada case shows a wrong answer can bind you.
Re-evaluate on change: rerun the evaluation set on every model version, prompt change or corpus update. Model upgrades can change hallucination behaviour in either direction.

Decision guidance

Generative answer vs retrieval only: where the correct answer is a fixed policy, price or legal statement, return the approved text (or a link to it) rather than letting the model paraphrase. Use free generation where variation is acceptable and errors are cheap.
Build vs buy assurance: vendor features such as grounding checks and citation verification help, but test them on your own evaluation set; vendor benchmark figures are the vendor's.
Small organisations: focus on tiers 3 to 5: restrict customer-facing bots to approved content, ban unverified AI citations in external documents, and enforce a package allowlist or registry proxy for developers.
Large organisations: run hallucination metrics through the model risk process, require evaluation evidence before production approval, and include AI-generated deliverables in supplier contracts and audit scope. See the AI Model Risk Management Guide.

Detection and measurement

Log prompts, retrieved sources, outputs, tool calls and user feedback for tier 3 to 5 use cases, with retention that matches your complaint and litigation windows. Monitor for spikes in escalations, corrections and customer disputes that mention the assistant.

Grounded answer rate: share of answers fully supported by retrieved approved sources, measured by sampled review or automated entailment checks.
Citation validity rate: share of cited references that resolve and support the claim (target 100% for tier 4 before release).
Confident error rate and abstention rate: measured on the evaluation set, tracked per model version.
Escaped hallucinations: count of errors found after reaching customers, courts, regulators or production, each logged as an incident with root cause.
Blocked package installs: requests for packages not on the allowlist or not present in the registry, by source (human or AI assistant).
Review coverage: share of tier 4 outputs with a recorded human verification.

Framework mapping

Framework	Reference	Relevance
OWASP Top 10 for LLM Applications 2025	LLM09:2025 Misinformation	Hallucination, overreliance, package hallucination scenario, mitigations
NIST AI 600-1	Risk 2.2 Confabulation; 2.8 Information Integrity; actions such as MS-2.3-002, MS-2.5-001, MS-2.9-001	Measurement and governance actions tagged to confabulation
MITRE ATLAS	AML.T0062 Discover LLM Hallucinations; AML.T0010.001 AI Software (supply chain); AML.T0048.002 Societal Harm	Adversary use of hallucinations and resulting harms
NIST CSF 2.0	ID.RA-05, GV.SC-07, PR.AT-01, DE.CM-09	Risk prioritisation, supplier risk, user awareness, monitoring
NIST SP 800-53 Rev. 5	SI-10 Information Input Validation, SA-11 Developer Testing and Evaluation, SR-11 Component Authenticity, AT-2 Literacy Training and Awareness	Output validation, evaluation before release, dependency authenticity, training
ISO/IEC 27001:2022 Annex A	A.8.28 Secure coding; A.8.29 Security testing in development and acceptance	Controls on AI-generated code and its acceptance testing

Common failure patterns

Assuming the correct policy elsewhere is enough: Air Canada argued the customer could have found the real bereavement policy on another page; the tribunal looked instead at whether reasonable care was taken to make the chatbot accurate.
Signing what you have not read: in Mata the filing lawyer did not read the cited cases; in Al-Haroun the solicitor relied on research from the client. Responsibility followed the signature.
Checking the conclusions but not the references: Deloitte said the report's findings stood, yet invented references were enough to require a correction and a refund.
Letting the model paraphrase law or policy: the NYC MyCity bot gave different answers to the same question about housing vouchers. Fixed rules should come from approved text, not generation.
Installing whatever the assistant suggests: without a registry proxy or allowlist, a hallucinated package name becomes an install command an attacker can fulfil.
Measuring accuracy only: a system that never abstains can score well on accuracy and still be confidently wrong most of the time it errs.

Practitioner checklist

Inventory every generative AI use case and assign a consequence tier and owner.
Restrict customer-facing assistants to approved, versioned content and check prices and policies against systems of record.
Require citations for factual claims and validate that each one resolves and supports the claim.
Require named human verification against primary sources for legal, regulatory, financial, medical and security output.
Enforce a package allowlist or registry proxy and block installs of packages that do not exist or are newly registered.
Give agents least-privilege credentials, confirmation gates and reversible actions.
Maintain an evaluation set per use case and score accuracy, error and abstention on every model or prompt change.
Log prompts, sources, outputs and tool calls, and treat escaped hallucinations as incidents.
Add AI use, disclosure and quality assurance clauses to supplier and professional services contracts.
Train staff on overreliance, using real cases such as Mata and Air Canada.

Standards and references

Related NHI Mgmt Group resources: AI Liability and Product Liability Guide · AI Model Risk Management Guide · LLM Security Guide · Slopsquatting and Package Hallucination Guide · AI Risk Assessment Guide. This guide summarises published decisions on AI output liability and is not legal advice.

AI Hallucination Business Risk Guide: Liability, Controls and Verification