AI data leaks expose a governance gap in enterprise security

At a glance

What this is: AI data leaks are a distinct enterprise risk because sensitive information can flow into AI systems through sanctioned and shadow usage, model memorization, and autonomous agent actions.

Why it matters: IAM, NHI, and security teams need a governance model that sees AI usage, constrains tool access, and controls data movement across human, machine, and agentic workflows.

By the numbers:

• In 2025, 20% of organizations that suffered a data breach said the security incidents involved shadow AI.
• The global average breach cost was $4.44 million in 2025.
• The EU AI Act can penalize prohibited AI practices up to €35 million or 7% of global revenue.

👉 Read WitnessAI's full guide on AI data leaks and shadow AI governance

Context

AI data leaks happen when confidential information moves into or out of AI systems in ways the organisation did not intend and cannot reliably control. That includes employees pasting sensitive material into public tools, models exposing memorised training data, and autonomous agents moving data through APIs and MCP-connected services.

The governance gap is wider than traditional data protection programmes assume. DLP, CASB, and SSE controls were designed around labels, perimeter inspection, and deterministic traffic patterns, while AI workflows are contextual, probabilistic, and often invisible at the browser or network layer. That is why AI data leaks sit at the intersection of IAM, NHI, and emerging agentic control models.

Key questions

Q: What breaks when AI data loss controls rely only on DLP and CASB?

A: They miss the main AI leakage paths because prompts, responses, embeddings, and agent actions are not ordinary file transfers. Traditional DLP and CASB depend on labels, regexes, and perimeter visibility, but AI workflows are contextual and often occur inside approved tools. Teams need controls that inspect the interaction itself, not just the network or file.

Q: Why do AI workflows complicate least-privilege access models?

A: AI workflows can move data across multiple systems in one session, especially when agents have API, database, and file-system access. Least privilege becomes harder to define because the access path is dynamic and the risk depends on intent, not only role. Practitioners need to bind permissions to task scope and to the exact data path.

Q: How do security teams know if shadow AI governance is working?

A: Look for evidence that unsanctioned AI use is visible, attributable, and enforceable. If teams cannot inventory approved tools, identify users and agents, or log prompt-response activity, governance is not working. A mature programme can show where data moves, who moved it, and which policy controlled the interaction.

Q: Who is accountable when an AI system leaks regulated data?

A: Accountability usually spans the business owner of the workflow, the security team responsible for policy, and the platform team that controls access and logging. When autonomous agents are involved, accountability must also cover the identity issuing the actions and the approvals that allowed tool access. That is why audit trails matter.

Technical breakdown

Why DLP and CASB miss AI data leaks

Traditional DLP and CASB tools look for known patterns such as labels, file types, regular expressions, and sanctioned network paths. AI interactions break those assumptions because the sensitive content may appear inside natural language prompts, be transformed into embeddings, or be returned as model output rather than as a file transfer. In retrieval-augmented generation, data can be vectorized into representations that legacy inspection tools cannot interpret. The security problem is therefore not just exfiltration, but semantic exposure across a conversational workflow.

Practical implication: security teams need controls that inspect AI context and intent, not just files and network sessions.

How agentic AI changes the leak path

Agentic AI changes leakage from human-driven misuse to software-driven action. An autonomous agent can call APIs, query databases, access file systems, and combine outputs across tools in a single workflow, often at machine speed and without a human approval gate between steps. When the agent is connected through MCP, the access problem becomes sharper because the protocol does not inherently preserve user context for downstream servers. That means the system may grant broad, shared access even when the initiating user should not have it.

Practical implication: treat agent tool access as a governed identity boundary, not as a simple application integration.

Why prompt and response protection both matter

AI leakage is bidirectional. Sensitive data can leave the organisation in a prompt, but it can also return in a model response, especially when the system memorises training data or has access to internal systems. Pre-execution controls reduce outbound exposure before the model acts, while response protection inspects what the model returns before users can copy, commit, or distribute it. If either direction is left open, the control model is incomplete.

Practical implication: enforce policy at both the prompt and response stages, not just at submission time.

Threat narrative

Attacker objective: The objective is to obtain sensitive enterprise data through AI-enabled workflows without triggering the controls built for conventional exfiltration.

1. Entry occurs when an employee, developer, or autonomous agent passes confidential data into an external AI system, or when a model has been trained on sensitive internal data and can reproduce it later.
2. Escalation happens when the AI system expands access through broad repository permissions, API calls, MCP connections, or context reuse that exposes more data than the original user should have reached.
3. Impact is the disclosure of customer records, source code, credentials, strategic documents, or other regulated information through prompts, outputs, or downstream tool actions.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI data leaks are now an identity governance problem, not just a content protection problem. The article shows that exposure can come from humans, service-style AI workflows, or autonomous agents, which means the control boundary is the actor and its access path, not only the document or field being protected. That shifts the programme question from "what data is sensitive?" to "which identities can move that data into AI systems?" Practitioners should govern AI usage as an access problem with data consequences.

Intent-based control is the named gap this category exposes. Content labels and keyword matching were designed for deterministic artefacts, but AI usage is conversational and context dependent. The result is an intent visibility gap: security teams can often see that a tool was used, but not whether the interaction was legitimate analysis, policy violation, or data exfiltration. Practitioners need to recognise that the failure mode is semantic blindness, not simply missing policy.

Shadow AI turns governance into an assurance problem. The article’s 20% breach linkage shows that unsanctioned AI is not a fringe exception, it is part of the active incident mix. That makes regular audits, approved tool inventories, and usage attribution the difference between policy on paper and control in practice. Practitioners should assume unseen AI use exists until they can prove otherwise.

Agentic data movement creates a broader trust boundary than human copy and paste. When an autonomous agent can query databases, call APIs, and reach external services through MCP, the issue is no longer just accidental disclosure by an employee. The governance model must account for machine-speed propagation across multiple systems in one session. Practitioners should treat those workflows as privileged NHI behaviour with direct data-loss potential.

OpenAI-style model exposure, coding assistants, and AI workflow tools all point to the same identity lesson: access without contextual restraint creates portable leakage. In NHI terms, the problem is not only credential theft but uncontrolled use of an identity to move sensitive data between systems. The practitioner conclusion is straightforward: governance must bind the identity, the context, and the data path together.

From our research:

• The average organisation believes more than 1 in 5 of their non-human identities are insufficiently secured, according to The 2024 ESG Report: Managing Non-Human Identities.
• Enterprises that have experienced a compromised NHI averaged 2.7 separate incidents in the past 12 months, according to The 2024 ESG Report: Managing Non-Human Identities.
• For teams building a broader AI governance posture, the Ultimate Guide to NHIs , Lifecycle Processes for Managing NHIs helps connect access, rotation, and offboarding to day-to-day control design.

What this signals

Intent-based visibility will become the deciding control category for AI governance. Teams that can see prompt content, response content, and agent tool calls will have a defensible operating model; teams that can only see app access will keep missing the actual leakage event. The practical signal is simple: if you cannot attribute AI interactions to a user or non-human identity, you do not yet have governable AI.

Shadow AI should be treated as an identity sprawl problem with data-loss consequences. In our research, more than 1 in 5 non-human identities are believed to be insufficiently secured, which is why governance cannot stop at policy creation. The programme response needs discovery, inventory, and enforcement across browsers, desktop apps, coding assistants, and agents, not just approved enterprise tools.

Agentic workflows will force security teams to align AI controls with NHI lifecycle thinking. Once agents can call APIs and access internal systems, the same questions arise that already govern service accounts: who created it, what can it reach, when does it expire, and how is it removed. The organisations that answer those questions consistently will be able to scale AI without normalising invisible data movement.

For practitioners

• Map AI usage by identity type Inventory which human users, service accounts, copilots, and autonomous agents can reach external AI services, internal models, and MCP-connected tools. Classify each path by data sensitivity and approval model so you can see where sanctioned use and shadow use overlap.
• Replace label-only DLP with intent-aware controls Use controls that inspect prompts, responses, and workflow context instead of relying only on keywords, file labels, or network destinations. Prioritise the surfaces where employees paste sensitive content and where agents can chain access across APIs.
• Constrain agent tool access to least-privilege scopes Require task-scoped credentials, explicit tool allowlists, and policy checks before agents query databases, file systems, or external services. Review whether MCP-connected servers can distinguish users or whether shared access is widening exposure.
• Add response-side protections for model output Inspect model responses for regulated data, source code, secrets, and memorised content before the output reaches the user or downstream systems. Rehydrate tokenised values only when policy permits and log the full prompt-response pair for audit.
• Audit shadow AI as a breach precursor Track unsanctioned AI use as an incident signal, not just a policy violation. Cross-check breach investigations against approved AI inventories, because the article’s data shows shadow AI is already present in real breach events.

Key takeaways

• AI data leaks are a control gap between how people use AI and how legacy security tools were built to detect loss.
• The cited breach and cost data show that shadow AI is already producing measurable financial and compliance exposure.
• The most effective response is to govern AI as an identity-and-data problem, with visibility, intent-aware controls, and agent scope limits.

Key terms

• Shadow AI: Shadow AI is the use of AI tools, models, or agents that security and governance teams have not formally discovered or approved. It matters because unseen AI usage creates invisible data paths, unreviewed access, and audit gaps that traditional control frameworks cannot reliably measure or contain.
• Intent-based classification: Intent-based classification evaluates what a person or agent is trying to do, not just which words, files, or labels appear in the interaction. In AI governance, that distinction matters because the same content can be harmless analysis in one session and an active disclosure event in another.
• Model Context Protocol: Model Context Protocol is a way for AI systems to connect to tools and data sources at runtime. It expands capability, but it also widens the governance problem because downstream services may not receive enough user context to apply precise, per-session access decisions.
• Prompt-response audit trail: A prompt-response audit trail records both what was sent to an AI system and what the system returned. It is essential for proving compliance, investigating leakage, and distinguishing normal productivity from policy violation, especially when agents and human users interact with the same model estate.

Deepen your knowledge

AI data leaks and shadow AI governance are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building a control model for human, machine, and agentic access to AI systems, it is worth exploring.

This post draws on content published by WitnessAI: AI data leaks and how to prevent them. Read the original.