AI systems expose sensitive data through lifecycle gaps and shadow use

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Orca Security's blog post: AI systems expose sensitive data across the full lifecycle, with traditional controls missing conversational, stored, and regenerated data flows.

Why it matters: It matters because IAM, data security, and AI governance teams need controls that follow data into models, prompts, and shadow tools, not just file systems and network choke points.

By the numbers:

• 56% of security professionals confirm employees use AI tools without formal approval, according to Mindgard survey findings cited by Orca Security.
• 22% suspect unsanctioned AI use is happening but cannot prove it, according to Mindgard survey findings cited by Orca Security.

Context

Sensitive data in AI pipelines is not just a storage problem. Once PII, PHI, source code, credentials, or contract text enters an AI workflow, it can be retained in prompts, encoded in model weights, or resurfaced in outputs outside the original control point.

The governance gap is that most security programmes still assume data moves through stable, inspectable boundaries such as file uploads, email, or databases. AI breaks that assumption because data can travel through browser sessions, third-party models, and unsanctioned tools without a clean transfer event.

For IAM and NHI teams, the question is not only who can access the data, but which service identities, models, and tools can ingest it, persist it, or regenerate it later.

Key questions

Q: How should security teams stop sensitive data from being uploaded into public AI tools?

A: Security teams should enforce endpoint controls that block sensitive files and clipboard content before they reach public AI tools. The policy should be based on data classification, application destination, and user context, so the control works at the moment of transfer rather than after the data has already left the enterprise boundary.

Q: Why do public AI tools create data leakage risk even when employees are acting in good faith?

A: Public AI tools create leakage risk because the employee is trying to get work done, not trigger a security event. Sensitive data can be pasted into a system the company does not control, and that transfer may be indistinguishable from routine productivity. The risk is data loss without obvious malice, which makes traditional alerting and user behavior assumptions less effective.

Q: What breaks when AI service identities have broad permissions?

A: Broad service identities turn one pipeline into many attack paths. If the same identity can read training data, modify model artefacts, and access logs, a single misuse can expose sensitive content across the whole lifecycle. Least privilege has to be enforced at the workload level, not assumed from the application boundary.

Q: How can organisations tell whether shadow AI is becoming a material risk?

A: Look for three signals: frequent use of personal accounts, multiple AI tools inside the same workflow, and extensions or connectors that request broad permissions. When those patterns overlap, the organisation has moved beyond isolated usage into unmanaged access sprawl. That is the point where governance needs to shift from awareness to control.

Technical breakdown

Why AI data flows defeat perimeter-style controls

Traditional DLP and perimeter controls look for fixed transfer points such as attachments, uploads, or endpoint writes. AI workflows often bypass those points entirely because the user pastes data into a chat interface or sends it through an API call that looks like ordinary application traffic. The deeper problem is that AI systems convert input into durable internal state, whether through prompts, embeddings, caches, or training data. That means the control point moves from the edge of the system to the point of ingestion and model use, where many legacy tools have no visibility.

Practical implication: treat AI ingestion and prompt handling as first-class control points, not as extensions of file or endpoint security.

How service identity overreach widens AI exposure

AI pipelines frequently reuse a small number of service identities across training, inference, logging, and retrieval layers. That is convenient for engineering but dangerous for governance because one identity can gain read access to training data, write access to logs, and broad permissions on model artefacts. In NHI terms, this is overprivileged workload identity, not just a misconfigured app. When the same identity spans multiple stages, exposure in one layer becomes access to all the others, which makes blast radius much larger than the original use case suggests.

Practical implication: scope each AI pipeline stage to a separate service identity with permissions that match actual function, not deployment convenience.

Why prompt storage and output regeneration create persistence risk

Prompt histories, conversation logs, and model outputs create a second life for data that many teams assume has already been handled. If a third-party model stores prompts, the organisation may no longer control deletion, retention, or residency. If a model memorises training fragments, the data can reappear later even when the original source is removed. That makes AI risk different from conventional data retention, because the exposure is not limited to where data is stored. It can also surface through model behaviour itself.

Practical implication: govern prompt retention, output filtering, and training-data handling as one lifecycle, not as separate operational concerns.

Threat narrative

Attacker objective: The objective is to persist, reuse, or surface sensitive data outside the organisation's authorised control boundary, whether through direct leakage or shadow AI use.

1. Entry occurs when sensitive content is pasted into a consumer AI tool, sent to a third-party model, or submitted through an unsanctioned workflow that bypasses approved controls.
2. Credentialed or regulated data is then retained in prompts, training sets, logs, or embeddings, creating persistence outside the organisation's original governance boundary.
3. The data can later resurface through model outputs, be accessed through overprivileged AI service identities, or remain exposed in shadow AI tools that lack enterprise oversight.
4. Impact follows when confidential material, credentials, or regulated records are disclosed to other users, stored in uncontrolled systems, or used in ways that violate policy and compliance obligations.

Breaches seen in the wild

• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.
• Samsung ChatGPT leak 2023: Samsung staff pasted chip source code and meeting notes into ChatGPT weeks after it was allowed, leading Samsung to restrict generative AI tools.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI data governance is now a lifecycle problem, not a point-control problem. Sensitive information does not simply cross a boundary and disappear inside AI systems. It can persist in prompts, be retained by providers, or resurface in outputs long after the original transaction. The governance model therefore has to follow the data from classification through ingestion, storage, and retrieval, or it will miss the real exposure path.

Overprivileged AI service identities turn one data exposure into a platform-wide problem. When training, inference, logging, and retrieval share the same workload identity, least privilege stops being meaningful in practice. The identity has more authority than any single task requires, so a compromise or misuse in one layer expands into multiple adjacent systems. Practitioners should read this as an NHI scoping failure, not as a generic cloud hygiene issue.

Shadow AI is a visibility failure before it is a policy failure. Organisations cannot govern data they do not know has entered an approved or unapproved model. The article's strongest signal is that employee behaviour is outrunning enterprise inventory, which means discovery and classification must extend into AI usage patterns, not just storage estates. The practical consequence is that governance must begin with discovery, or compliance and access policy become aspirational.

Prompt retention creates an unowned data copy that existing deletion models do not cleanly remove. That breaks the assumption that erasure, revocation, or storage deletion ends exposure. Once prompt history lives in a third-party system or a model has encoded sensitive patterns, the original owner may no longer be able to prove deletion or prevent regeneration. Teams should treat retention, residency, and model memory as one control surface.

Dynamic AI behaviour demands control points that operate before and after the model, not only around it. Input validation, output filtering, encryption, and continuous monitoring each address a different stage of the AI lifecycle, and none substitutes for the others. The field is moving toward continuous posture management for AI workflows because static review cycles cannot keep pace with how quickly data enters, moves, and reappears in these systems.

From our research library:

• 76% of organisations cite shadow AI as a definite or probable problem, up from 61% in 2025, according to HiddenLayer's 2026 AI Threat Landscape Report.
• Read next: Shadow AI and AI Agent Discovery Guide

What this signals

Shadow AI is the early warning signal. Once employees can route sensitive data into tools outside approved control points, the issue is no longer just user behaviour. It becomes an identity, discovery, and governance failure that requires continuous inventory of AI assets and the service identities behind them.

Lifecycle governance has to extend to prompt history and model memory. Revocation and deletion logic that works for files does not reliably work for stored prompts or memorised training data. Security teams should assume that AI introduces durable secondary copies unless retention, residency, and output handling are explicitly governed.

Least privilege for AI workloads is now a data-protection control. When one service account can reach training data, logs, and model artefacts, a data issue becomes an access-control issue. That makes AI workload identity scope part of the sensitive-data defence model, not an afterthought.

For practitioners

• Classify sensitive data before AI ingestion Inventory PII, PHI, source code, credentials, and regulated records across cloud storage and collaboration tools before they can enter prompts, training sets, or retrieval pipelines.
• Scope each AI workload identity separately Separate training, inference, logging, and retrieval permissions so a single service account cannot read all data stores, modify model artefacts, and alter forensic logs.
• Harden prompt and output handling Treat every prompt as untrusted input and every model response as unverified output, then apply sanitisation, filtering, and anomaly detection before either is reused downstream.
• Govern shadow AI discovery continuously Monitor for unsanctioned tools, unapproved models, and browser-based AI usage so governance teams can see where sensitive data is flowing outside approved contracts and residency rules.
• Encrypt and monitor across the full AI lifecycle Apply encryption to training data, prompts, outputs, and logs, then feed model activity into SIEM or SOAR for continuous detection of abnormal access or drift.

Key takeaways

• AI systems expose sensitive data through multiple lifecycle stages, so the main risk is persistence and reuse rather than a single point of exfiltration.
• Shadow AI and broad service identities amplify the problem because they move data beyond approved oversight and widen the blast radius of any misuse.
• The most effective response is to combine discovery, least privilege, prompt and output controls, encryption, and continuous monitoring across the full AI lifecycle.

Key terms

• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
• AI Pipeline: An AI pipeline is the chain of systems that moves data from ingestion to training, inference, logging, and retrieval. In security terms, it creates multiple persistence points where sensitive data can be copied, transformed, stored, or resurfaced beyond the original request.
• Service Identity: A service identity is a non-human identity used by applications, workloads, or automation to authenticate and access resources. It may be a role, token, key, or certificate, and it needs the same lifecycle discipline as any privileged identity because it can directly expose data.
• Prompt retention: Prompt retention is the storage of user prompts, chat history, or model context after a conversation ends. It matters because sensitive data can persist outside the original system boundary, creating privacy, compliance, and deletion challenges when organisations cannot prove what was kept, for how long, or where it was copied.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.