Chatgpt data leaks expose the gap between policy and control

At a glance

What this is: Riot frames ChatGPT risk as a three-part problem covering input leaks, output leakage, and hallucination-driven misuse, with real incidents showing that policy alone does not stop exposure.

Why it matters: IAM, security, and data protection teams need to govern how employees use generative AI because the main failure mode is uncontrolled data movement, not just unsafe software adoption.

By the numbers:

• Google DeepMind and Cornell researchers found that repeated-word prompting could make ChatGPT leak training data 164 times more often than a neutral control word.

Context

Generative AI changes data handling because employees can paste source code, contracts, customer records, and other sensitive material directly into a chat interface without the friction of a file upload. That makes the problem primarily one of governance and control design, not simply user awareness. For identity and security teams, the relevant question is how to keep data, privilege, and policy aligned when users interact with external AI services outside conventional exfiltration paths.

Riot separates ChatGPT risk into input risk, output risk, and hallucination risk. That distinction matters because a CASB or DLP control may reduce pasted data leaving the organisation, but it does not address model memorisation, republishing of training fragments, or employees acting on incorrect AI output. The article also shows that shadow AI is now part of the identity and access conversation because sanctioned use, account settings, and detection policies determine where the control boundary sits.

The Amazon and Samsung examples are not outliers so much as proof that well-intentioned employee use can become a data exposure channel quickly when organisations lack approved alternatives and AI-specific controls.

Key questions

Q: What breaks when employees paste sensitive data into generative AI tools without controls?

A: The control break is that data can leave the organisation outside normal exfiltration patterns, with no attachment, upload event, or obvious malware signal to catch it. That creates a governance gap between user intent and data handling, so DLP, CASB, and policy need to cover chat-style interactions, not just conventional file transfer.

Q: Why do generative AI tools increase data security risk?

A: Generative AI tools increase risk because they expand the number of places where sensitive content can be ingested, copied, surfaced, or misused. They also consume unstructured data that legacy classification tools often misread, which weakens policy enforcement. The result is a larger blast radius when access is over-permissioned or data visibility is incomplete.

Q: What are the signs that shadow AI is becoming a governance problem rather than a productivity aid?

A: The clearest signs are widespread use outside IT visibility, repeated sharing of corporate data with GenAI tools, and no consistent approval path for new applications. If employees are using AI because no sanctioned option exists, or if security teams cannot see which tools are active, the issue has moved from isolated behaviour to unmanaged risk.

Q: When should organisations restrict ChatGPT use instead of allowing it broadly?

A: They should restrict it whenever the workflow involves sensitive data, regulated records, source code, or operational instructions that could cause harm if exposed or misused. Broad approval works only when the organisation can classify inputs, govern outputs, and enforce acceptable-use rules consistently across teams.

Technical breakdown

Why pasted prompts create a new exfiltration path

Input risk occurs when an employee pastes sensitive material into a generative AI tool that sits outside the organisation's direct control. Unlike classic file exfiltration, there may be no attachment, no upload event, and no obvious malware-like signal for legacy DLP rules to catch. The result is a governance blind spot in which code, contracts, or customer data can cross a trust boundary in seconds. Because the user is acting with legitimate intent, the activity often looks like productivity rather than theft, which makes detection and policy design harder.

Practical implication: DLP and CASB controls need to inspect chat-style pastes and sanctioned AI endpoints, not just email and file-transfer channels.

How model memorisation turns output into a governance problem

Output risk is different from user input because the model itself can be manipulated into reproducing fragments of its training data. That means a user can encounter information they never owned, and an attacker can potentially extract memorised content through repeated prompts or other divergence-style techniques. The security issue here is not only confidentiality but boundary control, since the system may surface data from many conversations or sources at once. This is why opt-outs, enterprise data settings, and model-use policy all matter together rather than separately.

Practical implication: govern enterprise and consumer AI settings as part of the identity and data control plane, not as a one-time user policy.

Why hallucinations are a separate control failure

Hallucination risk is not the same as data leakage. Here the model produces plausible but incorrect output that a worker may treat as factual, which can distort legal, operational, or technical decisions. The failure mode is trust in an unverified machine response, not disclosure of sensitive data. That makes this a training and verification issue as much as a security one. Organisations that separate input controls from output controls often miss this third problem entirely, even though it can still create significant business harm.

Practical implication: require verification for AI-generated work products in high-risk workflows, especially legal, security, and engineering tasks.

Threat narrative

Attacker objective: The objective is to gain access to confidential company data or training fragments through ordinary user interaction with a generative AI service.

1. Entry occurs when an employee pastes sensitive code, records, or contracts into a generative AI tool during normal work.
2. Credential or data exposure follows if the prompt content is retained, trained on, or made available through downstream model behaviour.
3. Impact emerges when confidential material leaves organisational control or when leaked model fragments influence another user's session.

Breaches seen in the wild

• Samsung ChatGPT leak 2023: Samsung staff pasted chip source code and meeting notes into ChatGPT weeks after it was allowed, leading Samsung to restrict generative AI tools.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

ChatGPT risk is now a data-governance problem, not just an acceptable-use problem. The article shows that employees do not need malicious intent to create exposure, because paste-based workflows let sensitive material cross the trust boundary instantly. In practice, this means AI use policy must be tied to detection, approved alternatives, and data classification if it is to be enforceable.

Input controls and output controls solve different problems, and organisations that treat them as interchangeable will fail. DLP can help with pasted content, but it does nothing to stop model memorisation or later extraction of training data. Conversely, training-data controls do not stop a user from pasting a contract into a public chatbot, so the control set has to be layered by risk type.

Shadow AI is the right named concept for this problem space. The article describes undiscovered or unmanaged AI use that bypasses normal security review and creates a policy gap between what the organisation allows and what employees actually do. The practitioner takeaway is that discovery, sanctioned access, and monitoring must be treated as one programme, not separate initiatives.

The Amazon and Samsung cases show that productivity pressure is often the real attacker helper. People use generative AI to move faster, and that incentive can override caution unless the organisation gives them a safer approved path. The practical implication is that governance must accommodate workflow reality, not just state prohibitions.

AI data leakage is becoming an identity and access issue because the user session is now part of the control surface. When workers authenticate to a third-party AI service, the account, settings, and session context determine what data can be retained or reused. Practitioners should therefore treat AI accounts, retention settings, and access pathways as governed identity assets rather than informal productivity tools.

From our research library:

• 43% of security professionals are concerned about AI systems learning and reproducing sensitive information patterns from codebases, according to the State of Secrets in AppSec.
• Read next: AI Agent Identity Security Buyer's Guide

What this signals

Shadow AI governance will increasingly be decided at the account and session level. The practical boundary is no longer the perimeter alone, but the combination of user intent, account settings, and what data an employee can paste into an external model. Programmes that do not inventory sanctioned and unsanctioned AI use will struggle to prove control.

AI data exposure is now a policy, identity, and detection problem at once. The teams that do best will align acceptable use, DLP, and approved tool access instead of treating them as separate workstreams. That is especially true where workers handle code, contracts, and other sensitive material in browser-based AI tools.

43% of security professionals are concerned about AI systems learning and reproducing sensitive information patterns from codebases, according to the State of Secrets in AppSec.

For practitioners

• Classify promptable data before you set policy Inventory the data types most likely to be pasted into generative AI tools, including source code, contracts, PII, financial data, and M&A material, then define handling rules for each.
• Deploy AI-aware DLP and CASB controls Extend monitoring to chat interfaces and browser paste events so sensitive text can be flagged or blocked before it reaches third-party AI services.
• Turn off training-data reuse by default Verify enterprise and consumer account settings so employee prompts are not reused for model training unless the organisation has explicitly approved that behaviour.
• Provide a sanctioned AI alternative Offer an approved internal or enterprise AI service so employees do not route sensitive work into shadow AI when public tools are restricted.
• Train for the helpful assistant blind spot Make awareness training specific to the productivity mindset that leads people to paste confidential material while believing they are simply working faster.

Key takeaways

• Generative AI creates a three-part risk model covering sensitive input, model output leakage, and hallucination-driven misuse.
• Real incidents at Amazon and Samsung show that well-intentioned employee behaviour can become a data exposure path quickly when AI use is not governed.
• The controls that matter most are AI-aware detection, sanctioned alternatives, and explicit training-data settings, not policy statements alone.

Key terms

• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
• Input Risk: The risk that sensitive data is entered into an external AI system during normal work and then leaves organisational control. The concern is not only leakage but retention, reuse, and policy mismatch across consumer and enterprise account settings.
• Effect Risk: Effect risk is the possibility that an AI system will cause real-world actions, not just produce flawed text. It matters when outputs can trigger payments, changes, postings, or system commands, because the security boundary moves from content quality to operational control.
• Hallucination Risk: Hallucination risk is the chance that an AI system produces a confident but incorrect explanation because the input data is incomplete, inconsistent, or misleading. In security operations, it usually appears when context is missing and the model has to infer more than the telemetry can prove.

Deepen your knowledge

NHI Mgmt Group's NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security and identity practitioners translate identity controls into real-world governance decisions for AI and cloud-adjacent systems.