Generative AI Data Loss Prevention Guide

Generative AI has opened a new data loss channel that most DLP programmes were not built for: staff paste sensitive data into AI tools, copilots surface data people should never have been able to reach, and AI agents can be talked into sending data out. This guide covers the GenAI-specific controls: tiering AI tools, browser and endpoint controls, prompt and response inspection, sensitivity labels honoured by copilots, RAG permission hygiene, training opt-outs, prompt logging and retention, and agents with tool access. It is for security engineers, data protection teams and AI platform owners. General DLP mechanics are in the Data Loss Prevention (DLP) Guide; product features are named as documented examples only.

Key takeaways

• Give people a safe tool before you block the unsafe one. An enterprise AI service with no training on business data, admin-controlled retention and a DPA removes most reasons to use consumer tools.
• Copilots inherit your permission debt. An assistant that "only surfaces organizational data to which individual users have at least view permissions" (Microsoft) will still find every overshared file. Fix sharing and labels before rollout.
• Inspect prompts, uploads and responses, not just files. Pasted text, attached files and generated output each need a control point, and vendor controls often cover only one.
• Prompt logs are a new sensitive data store. They collect everything users typed, so restrict access and set a deletion period.
• Agents turn leakage into exfiltration. An agent with mail, web or API tools can send data out on an attacker's instructions; least-privilege tools, egress allowlists and approvals matter more than prompt filters.

How data leaks through generative AI

OWASP lists this as LLM02:2025 Sensitive Information Disclosure and warns that data users enter may later reappear in responses. NIST AI 600-1, the Generative AI Profile, names Data Privacy as one of its twelve GenAI risks: "Impacts due to leakage and unauthorized use, disclosure, or de-anonymization of biometric, health, location, or other personally identifiable information or sensitive data." In practice there are six distinct leak paths.

Tier your AI tools before writing rules

AI DLP rules need a published list of which tools are allowed for which data. Discovery comes first; see the Shadow AI Discovery Guide. Then assign every tool to a tier.

The terms that move a tool into the sanctioned tier are specific and should be checked in writing, not assumed:

• Training default: OpenAI states that data from ChatGPT Business, Enterprise, Edu and the API Platform "isn't used for training" by default; Microsoft that "Prompts, responses, and data accessed through Microsoft Graph aren't used to train foundation LLMs"; Google that Workspace content is not used for training outside your domain without permission. Consumer tiers differ: in ChatGPT, training depends on the user's "Improve the model for everyone" setting, and submitting feedback can make the whole conversation eligible.
• Retention: whether admins set retention (Google lets admins auto-delete Gemini app conversations after 3, 18 or 36 months), how long deleted content persists (OpenAI: "within 30 days"), and API retention (OpenAI keeps API data up to 30 days for abuse monitoring unless zero data retention is approved for eligible endpoints).
• Contract and audit: a data processing agreement, sub-processor list, data location, SSO with SCIM deprovisioning, and an export or compliance API so prompts can be investigated in your own systems.

The controller remains accountable for personal data staff send to a provider; the EDPB's expert report on LLM privacy risks tells deployers to "Work with providers to ensure they adhere to data protection regulations and do not retain or misuse" it. Legal basis and DPIA questions are in the AI and Privacy Guide.

Controls on public AI tools: browser and endpoint

Much consumer AI use happens in a browser tab, so the control point is the browser, the endpoint agent or the network path. Each has gaps.

• Endpoint DLP with a browser extension: detects paste and upload of sensitive content to listed AI sites in managed browsers. Microsoft Purview's preconfigured policy "DSPM for AI: Detect sensitive info added to AI sites" "discovers sensitive content pasted or uploaded in Microsoft Edge, Chrome, and Firefox to AI sites" in audit mode, and "DSPM for AI - Block sensitive info from AI sites" applies block-with-override to elevated-risk users, created in test mode. Third-party AI site coverage needs the Purview browser extension and onboarded devices.
• Enterprise browser: restricts paste, upload and download per site and profile, and can separate personal and work accounts on the same AI service. See the Enterprise Browser and Browser Security Guide.
• Network inspection (SSE, proxy): sees prompt payloads from any app, including desktop clients, if TLS inspection is in place. Purview can ingest these through a SASE or SSE integration. See the TLS Inspection Guide.
• Account restrictions: allow the corporate workspace of an AI service and block personal logins to the same domain where the provider supports tenant restrictions.

Choosing an approach

• Small organisations: one sanctioned AI service, a web filter warning on consumer AI categories, and the DLP built into the productivity suite. Coaching messages do most of the work.
• Large or regulated organisations: endpoint DLP on managed devices, inline SSE for everything else, exact data match for customer records, and risk-adaptive rules for users with insider risk indicators, agreed with HR and privacy.
• Unmanaged devices: no agent applies, so use conditional access to keep corporate data off them or deliver it through an enterprise browser or virtual desktop. See the Conditional Access Guide.

Prompt and response inspection

For sanctioned and internal AI services, inspect what goes in and what comes out. Detection methods are the usual DLP ones; the actions differ.

• Redact, do not just block: OWASP lists tokenisation and pattern-based redaction before processing as mitigations; replacing a card number with a token keeps the user productive.
• Know what is not inspected: Microsoft documents that DLP for Microsoft 365 Copilot "only checks the text you type into the prompt itself", not files uploaded into prompts.
• Inspect output: NIST AI 600-1 action MP-4.1-009 says to "Leverage approaches to detect the presence of PII or sensitive data in generated output". It catches retrieved data the prompt never contained.
• Do not rely on the system prompt: OWASP notes that instructions restricting which data a model returns may not hold up against prompt injection.

Gateway placement, classifiers and LLM judges are covered in the AI Guardrails and AI Firewalls Guide.

Copilots, sensitivity labels and oversharing

Copilots search everything the user can open. They create no new access, but make latent oversharing (sites shared with "everyone", stale groups, organisation-wide links) instantly discoverable. Rollout and agent risks are in the Enterprise AI Copilot Security Guide; below is the DLP layer.

• Labels and encryption are honoured: Microsoft states that for content encrypted by Purview Information Protection, Copilot "honors the usage rights granted to the user". Google states that where IRM stops a user downloading, printing or copying a file, "Gemini will not retrieve those files".
• Exclude labels from AI processing: the Purview DLP location "Microsoft 365 Copilot and Copilot Chat" (Custom template only) offers the action "Prevent Copilot from processing content". Labelled items "still appear in the citations of the response", but their content is not used. It covers files and emails sent on or after 1 January 2025, and changes can take up to four hours to apply.
• Block sensitive prompts: the same location can match sensitive information types in typed prompts so that "The experience doesn't respond to the prompt", or block web search grounding for that prompt (in preview at the time of writing).
• Scope search while you fix permissions: SharePoint Restricted Content Discovery hides a site from organisation-wide search and Copilot (Set-SPOSite -Identity <site-url> -RestrictContentOrgWideSearch $true). It "doesn't change existing permissions", is described as a temporary governance control, and can take more than a week on sites over 500,000 items.
• Assess oversharing: DSPM for AI runs a weekly default data risk assessment of the top 100 SharePoint sites by usage and offers remediation by label-based DLP, Restricted Content Discovery, auto-labelling or retention. The method is the same in any suite: find widely shared sites holding sensitive data, then fix the sharing.

The EchoLeak flaw (CVE-2025-32711, published 11 June 2025, described by NVD as "Ai command injection in M365 Copilot") showed that an external email could instruct Copilot to pull internal data into a response that leaked it. Reducing what the assistant can reach limits what any such flaw can expose.

RAG and custom AI applications

Internal chatbots and agents built on retrieval-augmented generation move documents into vector stores, often under a single service identity that can read everything. The EDPB expert report on LLMs flags RAG "knowledge bases containing sensitive data or identifiable personal data without implementing proper safeguards".

• Carry permissions into the index: store source access control lists or labels as metadata with each chunk and filter at query time by the caller's identity, not the application's. See the Permission-Aware RAG Guide.
• Curate before indexing: index named, owned collections rather than whole drives, exclude the highest labels, and keep one namespace per customer in multi-tenant products.
• Re-sync deletions and permission changes: a document removed or restricted at source must leave the index on a defined schedule, or the assistant keeps answering from stale copies.
• Protect the pipeline identities: embedding jobs and connectors hold broad read access and API keys; store them in a secrets manager and scope them to the collections they index. See the LLM Security Guide.

Logging and retention of prompts

Prompt logs are needed for investigations, subject access requests and record-keeping, and they concentrate whatever staff typed. The EDPB expert report warns that "Logs from monitoring systems may retain personal data such as user interactions" and advises to "Enforce strong access controls, encrypt log data, and monitor access logs". The DeepSeek database exposure, which leaked chat logs and API secrets from an open database, shows the risk.

AI agents with tool access

When a model can call tools, leakage becomes exfiltration: an outbound email, a web request whose URL carries data, an image link in a chat client that loads an attacker URL with data in the query string, or a write to an attacker-readable location. In ATT&CK terms the result is usually T1567 Exfiltration Over Web Service, performed by a trusted identity.

• Injected instructions: the ForcedLeak and SalesBleed cases turned CRM agents into data exfiltration tools through poisoned records. See the Prompt Injection Guide.
• Malicious tools: the postmark-mcp malicious MCP server copied every email an agent sent to an attacker. Vet and pin MCP servers; see the MCP Security Guide.
• Least-privilege tools: give each agent its own identity with only the scopes its task needs, read-only by default; never reuse a human's session or a shared API key. See the AI Agent Authorisation Guide.
• Egress allowlists: restrict outbound fetch and send tools to approved domains and recipients; block or proxy rendering of external images and links in AI output.
• Approval and volume limits: require confirmation before an agent sends data outside the organisation, cap records per tool call, and alert on unusual volume per agent identity.

Implementing GenAI DLP in phases

1. Discover and decide: inventory AI use from proxy, SSE, browser and OAuth consent logs; publish tool tiers and the data classes allowed in each.
2. Provide the sanctioned tool: contract an enterprise AI service with the terms above, enforce SSO, set retention, and switch on audit logging before announcing it.
3. Audit public AI use: run paste and upload detection to AI sites in audit mode for at least two business cycles.
4. Prepare copilots: run oversharing assessments, fix organisation-wide sharing on sensitive sites, label the highest-risk content and exclude those labels from AI processing.
5. Coach, then enforce in rings: enable warnings with a link to the sanctioned tool for a pilot group (IT and security first), then block-with-override, then block for secrets and exact-match customer data.
6. Secure custom AI and agents: add prompt and output inspection at the AI gateway, permission-aware retrieval, and egress controls for agent tools.
7. Roll back safely: version each rule and be able to return it to audit the same day, allowing for propagation delays.

Detection and measurement

Send AI DLP events (paste, upload, prompt match, blocked Copilot processing, AI site visits; Purview logs "AI interaction" and "AI website visit" events) to the SIEM with user, device and destination. Treat any secret or regulated data confirmed in a third-party prompt as an incident: contain, rotate, request deletion from the provider where the contract allows, and assess notification duties. See the Incident Response Guide.

• Sanctioned share: percentage of AI sessions going to sanctioned tools, trending up.
• Sensitive prompt rate: sensitive matches per 1,000 prompts to public tools, by data type, trending down after coaching.
• Override and repeat rate: share of warnings overridden and users with repeat events after coaching.
• Oversharing exposure: number of sites or files with sensitive labels shared organisation-wide or externally, trending down.
• Label coverage: percentage of crown-jewel repositories with sensitivity labels applied.
• Agent egress: number of agents with unrestricted outbound tools, target zero.

Framework mapping

In CIS Controls v8.1 the same work sits under Control 3 Data Protection, notably 3.3 (data access control lists), 3.12 (segment processing by sensitivity) and 3.13 (deploy a data loss prevention solution). Wider AI risk management is in the AI Governance Guide.

Common failure patterns

• Banning without an alternative: after the Samsung ChatGPT source code leak, in which staff pasted code into a public AI tool, the response was a ban. Bans without a sanctioned tool push use onto personal devices.
• Copilot before permissions: organisation-wide links and stale groups turn the assistant into a search engine for salaries, HR cases and board papers.
• Enterprise contract, consumer logins: personal accounts on the same AI domain are never blocked, so sensitive prompts still reach the consumer tier.
• Keeping every prompt forever: full-content logging with no retention period or access control creates the most sensitive dataset in the company.
• Giving agents broad tools: a support agent with unrestricted email and web tools can be steered by one poisoned record into sending data out, as the EchoLeak and ForcedLeak cases show.

Practitioner checklist

• Publish AI tool tiers and the data classes allowed in each, and review them quarterly.
• Contract a sanctioned enterprise AI service with no training on business data, admin retention control and a DPA.
• Block personal accounts on AI services where a corporate workspace exists.
• Deploy paste and upload detection to AI sites in audit mode, then coach, then block secrets and exact-match customer data.
• Run an oversharing assessment and fix organisation-wide sharing on sensitive sites before copilot rollout.
• Exclude the highest sensitivity labels from copilot processing and confirm label coverage on crown-jewel repositories.
• Enforce permission-aware retrieval and deletion sync in every RAG application.
• Set retention, access control and encryption for prompt logs, and record them in the record of processing.
• Give each AI agent its own least-privilege identity, an egress allowlist and approval for outbound actions.

Standards and references

• OWASP: LLM02:2025 Sensitive Information Disclosure
• NIST: AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
• EDPB Support Pool of Experts: AI Privacy Risks and Mitigations, Large Language Models (LLMs)
• ICO: How should we assess security and data minimisation in AI?
• Microsoft Learn: Considerations for Microsoft Purview DSPM for AI
• Microsoft Learn: Learn about using DLP to protect interactions with Microsoft 365 Copilot
• Microsoft Learn: Restricted Content Discovery
• OpenAI: Enterprise privacy

Related NHI Mgmt Group resources: Data Loss Prevention (DLP) Guide · Shadow AI Discovery Guide · Enterprise AI Copilot Security Guide · AI Guardrails and AI Firewalls Guide · Data Classification Guide