LLM hallucinations can mimic data leaks in enterprise AI

At a glance

What this is: Knostic research finds that enterprise LLMs can invent plausible but false sensitive information, including salary and personal data, when retrieval fails.

Why it matters: For IAM and security teams, fabricated answers blur the line between disclosure, misstatement, and policy failure, creating governance issues across human identity, NHI-assisted workflows, and regulated data handling.

By the numbers:

• 70% of employers have caught employees using AI assistants for salary research, with 38% saying these tools are pushing salary demands higher than ever.
• 63% of HR leaders report employees making salary requests based on completely inaccurate information they got from AI.
• 72% of organisations have experienced or suspect they have experienced a breach of non-human identities, with 46% confirmed and 26% suspected.

👉 Read Knostic's research on fabricated AI data and enterprise governance risk

Context

Enterprise AI creates a governance problem when systems answer with confidence despite weak or missing source data. In enterprise settings, that means a copilot can turn a retrieval failure into a plausible-sounding statement about salaries, personal details, or other sensitive information, which then behaves like a disclosure event even when no real record was accessed.

This is not just a content quality issue. For identity and access programmes, the risk sits at the intersection of human identity data, AI-assisted workflows, and policy enforcement, because downstream users may act on fabricated statements as if they were authoritative. That makes validation, provenance, and response boundaries part of the control model, not a separate AI concern.

Key questions

Q: What breaks when enterprise AI fabricates sensitive information?

A: When enterprise AI fabricates sensitive information, the main failure is trust collapse. Employees and managers may act on a false statement as if it were official, which can trigger disputes, compliance questions, and poor decisions. The risk is not limited to leaks. A made-up answer can be just as damaging because it still influences behaviour and governance outcomes.

Q: Why do hallucinations create compliance risk in AI assistants?

A: Hallucinations create compliance risk because regulated decisions depend on accurate, source-backed statements. If an assistant invents salary, privacy, or policy details, the organisation may distribute misleading information that looks authoritative. In practice, that can create audit problems, employee relations issues, and possible regulatory exposure if the false answer affects documented processes or internal controls.

Q: How can security teams measure whether AI responses are trustworthy?

A: Security teams should measure how often the assistant refuses ungrounded requests, how often answers can be traced to source documents, and how often sensitive prompts return unsupported claims. Those signals are more useful than generic accuracy scores because they show whether the system is staying inside its governance boundary when data is incomplete.

Q: Who is accountable when an AI assistant gives a false answer about employee data?

A: Accountability sits with the organisation that deployed the assistant, not with the model alone. Teams that expose employee or policy data through AI must define ownership for retrieval access, output approval, logging, and incident response. Under regulated conditions, the business remains responsible for what the assistant says and how users act on it.

Technical breakdown

Why LLMs fabricate sensitive information when retrieval fails

Large language models generate text by predicting likely next tokens, not by verifying truth. When retrieval-augmented generation fails to find a matching record, the model may still produce a confident answer if the prompt encourages completion and the system lacks a strong abstention rule. In enterprise copilots, incomplete indexing, weak document retrieval, and ambiguous prompts increase the chance that the model invents a name, salary, or policy detail that sounds credible. The failure mode is not random error. It is a predictable combination of probabilistic generation and missing guardrails around source grounding.

Practical implication: require abstention or refusal behavior when the system cannot ground a claim in retrievable source data.

Why fabricated data can be as damaging as a real leak

A fabricated statement can trigger the same organisational harm as an actual disclosure because recipients rarely distinguish between a real record, an outdated value, and a generated falsehood. That is especially true when the AI is perceived as an official enterprise interface. In practice, false salary figures, performance claims, or confidential project details can create disputes, compliance exposure, and trust collapse even if no protected source was exposed. Security controls that only look for exfiltration miss this class of harm because the damage comes from false authority, not just data movement.

Practical implication: classify hallucinated sensitive statements as governance incidents, not just quality defects.

How data governance for AI needs provenance and policy enforcement

AI governance in enterprise environments has to cover provenance, authorization, and output filtering together. Provenance establishes where an answer came from. Authorization determines whether the user should see the data. Policy enforcement decides whether the model may surface, summarise, or refuse the information. Without all three, an enterprise assistant can overreach even when the underlying content is incomplete or incorrectly indexed. This is where identity controls matter: access boundaries, least privilege, and lifecycle governance must extend into AI retrieval paths and downstream response handling.

Practical implication: tie copilot access, retrieval permissions, and output filtering to existing identity and data governance controls.

NHI Mgmt Group analysis

Fabricated sensitive information is a governance failure, not just an AI accuracy problem. The core issue is that enterprise users treat an AI response as authoritative even when it is generated from incomplete context. That breaks the boundary between data security, human trust, and compliance handling. For identity teams, the lesson is that AI output control now belongs in the same governance conversation as access control and data classification.

LLM hallucination creates a new class of false disclosure risk. Traditional data loss tools are built to detect real secrets leaving approved boundaries. They do not fully address situations where the model invents a salary, personal detail, or policy exception that never existed in the source systems. That means the control gap is not just leakage detection, but truth validation and response suppression when grounding fails. Practitioners should treat this as an AI governance debt issue.

Identity governance becomes relevant the moment AI speaks for the organisation. If a copilot can answer questions about employee data, access, or internal records, then the enterprise has effectively delegated part of its control surface to an AI-mediated interface. That makes identity, authorization, and output policy inseparable. The organisation must govern who can ask, what the model can retrieve, and when it must refuse. Practitioners should extend least privilege into AI response paths.

Data governance programmes need a named concept for this failure mode: hallucinated disclosure. This is the point where a false AI statement creates the same business impact as a leak, because users cannot reliably distinguish fiction from sensitive truth. The control objective is not perfect model honesty, which is unrealistic, but bounded response behaviour that prevents ungrounded sensitive claims from spreading. Security leaders should design for containment of false authority.

Regulated enterprises will increasingly be judged on AI-generated statements, not just data stores. If a model gives a wrong answer about compensation, privacy, or compliance and people act on it, the organisation still owns the outcome. That shifts accountability toward documented provenance, approval paths, and auditability for AI-assisted workflows. Practitioners should align AI governance evidence with the same control discipline used for regulated records and identity-linked decisions.

What this signals

Hallucinated disclosure: enterprise AI is now capable of producing false sensitive statements that behave operationally like data leaks, which means security teams need detection logic for ungrounded claims as well as exfiltration. That is where control models built around provenance, refusal, and output policy matter most, especially when identity-linked data is involved.

The next governance gap will be proving that an AI answer was safe to surface, not just that the underlying dataset was protected. If the assistant can answer employee, salary, or policy questions, then retrieval boundaries and identity permissions must be audited together. Use the NIST Cybersecurity Framework 2.0 as the broader control spine and keep AI output evidence tied to source accountability.

This is also a workload identity problem in practice, because the assistant is acting as an interface to sensitive enterprise systems. When the response path is over-broad, the model can reconstruct information users should never see, even without direct exfiltration. For identity teams, that makes AI access scope a first-class design issue rather than an afterthought.

For practitioners

• Implement grounded-response controls Require enterprise copilots to cite source documents or refuse when they cannot ground a sensitive claim in retrievable records. This is especially important for salary, personnel, and policy questions where false precision creates immediate governance risk.
• Classify fabricated statements as incidents Treat hallucinated sensitive information as a security and compliance event when it crosses into employee data, confidential projects, or regulated content. Route these events into the same review process you use for policy violations and misleading disclosures.
• Extend least privilege into AI retrieval paths Limit what the AI can search, summarise, or surface based on the user’s authorised access and the sensitivity of the underlying dataset. If the user should not see a record directly, the assistant should not be able to reconstruct it from broad retrieval access.
• Add provenance and refusal logging Log which sources were available, which were returned, and why the model answered or refused. That evidence is essential for auditability, especially when a hallucinated answer causes a dispute over compensation, access, or confidentiality.
• Test copilots with adversarial prompts Use red-team scenarios that ask for employee pay, personal data, and confidential project details when the source system is incomplete or missing records. The goal is to verify that the AI declines gracefully instead of inventing an answer.

Key takeaways

• Enterprise LLMs can invent sensitive information that looks authoritative enough to drive real workplace and compliance harm.
• The risk is not only leakage of real data, but false disclosure through fabricated details that users may trust and repeat.
• Security teams need provenance, refusal, and access-boundary controls so AI assistants cannot speak beyond their grounded evidence.

Key terms

• Hallucinated Disclosure: A false statement produced by an AI system that is treated like a real sensitive disclosure because it sounds authoritative. The risk is not only that the model is wrong, but that users act on the fabricated content as if it came from a trusted source.
• Grounded Response: An AI output that can be traced to verifiable source material the system actually retrieved during the interaction. Grounding reduces the chance of invention, but only when retrieval, access controls, and refusal behaviour are designed to stop unsupported claims from being surfaced.
• Provenance Control: The practice of recording where an AI answer came from, which sources were available, and whether the model had enough evidence to respond. In enterprise governance, provenance is essential for auditability, incident review, and deciding whether a response was trustworthy.
• Retrieval Boundary: The set of permissions and scope limits that determine what information an AI assistant can search, summarise, or expose on behalf of a user. When retrieval boundaries are too broad, the model can reconstruct information that the user should not directly access.

What's in the full article

Knostic's full research covers the operational detail this post intentionally leaves for the source:

• How the enterprise copilot behaved when it could not find HR data and why it fabricated a response
• The specific demo setup used to test salary and personal-information prompts inside an artificial company environment
• Examples of how false salary or personnel answers can affect employee relations, compliance handling, and managerial decisions
• The vendor’s framing of its own AI governance approach and product positioning

👉 Knostic's full post includes the demo scenario, source materials, and examples of how false AI answers spread through workplace decisions.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management. It helps practitioners connect identity controls to the broader security decisions that govern AI-assisted access and response.