PHI exposure in GenAI tools is a governance problem, not just DLP

At a glance

What this is: Cyberhaven argues that PHI exposure in GenAI tools is a governance problem because healthcare employees are sending patient data into approved and unapproved AI systems faster than conventional controls can see or stop it.

Why it matters: This matters because IAM, security, and compliance teams need policies that govern data flows, vendor agreements, and tool usage together, not as separate controls.

Context

PHI exposure in GenAI tools is a governance gap when patient data moves into systems the organisation does not control. In healthcare, the problem is not simply whether an AI tool exists, but whether data is visible, authorised, and constrained once staff use it in real workflows.

The security issue is amplified by shadow AI and embedded AI features in approved platforms. That creates an identity and access problem as well as a data protection problem, because staff are effectively extending organisational trust to external services without the same governance that applies to internal systems.

Key questions

Q: What fails first when employees paste PHI into GenAI tools?

A: The first failure is usually governance visibility. Organisations often know which tools are approved, but not which prompts carry PHI or where those prompts are sent. Once patient data enters a consumer or embedded AI service without enforced destination control, legal, compliance, and breach exposure can arise before the security team has usable evidence.

Q: When should healthcare teams prioritise BAA enforcement over tool allowlisting?

A: They should prioritise BAA enforcement when the main risk is patient data leaving the organisation through legitimate work tools. Allowlisting helps reduce exposure, but it does not stop PHI from reaching an approved application that lacks the legal and technical conditions required to process it. The control must operate at the data destination, not just the procurement layer.

Q: What are the signs that PHI governance in GenAI is failing?

A: Common signs include repeated use of personal AI accounts, clinicians relying on consumer tools for documentation, AI features appearing inside approved platforms without review, and no reliable audit trail for PHI prompts. If the organisation can only see application access but not data movement, the governance model is already out of date.

Q: How should healthcare organisations respond when PHI is sent to a non-compliant AI service?

A: They should treat it as a compliance and incident-review event, then determine whether the disclosure is reportable under HIPAA and whether the destination should be blocked at the data layer. The response should also feed back into policy updates, staff guidance, and tool approval criteria so the same path is not repeated.

Technical breakdown

Why natural-language PHI is harder to control than file uploads

PHI entering a prompt is often invisible to controls built for structured file transfer. Traditional DLP works best when it can inspect known file types, fixed patterns, or explicit destinations, but GenAI use is frequently free text that carries identity, clinical, and payment context in ordinary language. That means the control problem shifts from blocking a file to understanding intent, context, and destination. When a user types patient details into an external model, the disclosure can occur before a perimeter tool sees a conventional exfiltration event.

Practical implication: controls need content awareness at the point of data movement, not only at the network edge.

Why approved tools still create PHI governance risk

An approved application is not automatically an approved data destination. Embedded AI features inside EHRs, productivity suites, and collaboration tools can move PHI to external services without a separate procurement event or visible user choice. That creates a governance blind spot where the identity of the application is known, but the downstream processing path is not. For healthcare teams, the real question is whether the tool has authority to receive PHI under the organisation's policy and legal posture, not whether the application itself was once sanctioned.

Practical implication: governance must classify destinations and AI features, not just applications.

Why BAA enforcement has become a data control issue

A business associate agreement is a legal prerequisite, but it is not a technical safeguard by itself. If PHI can still be sent to a tool before policy enforcement intervenes, the organisation has already accepted exposure. This is why BAA checks must be tied to data-level control and audit evidence. The governance model has to prove that PHI was either routed only to authorised services or prevented from leaving the environment at all.

Practical implication: link approval status, destination checks, and audit logging into one enforcement path.

Threat narrative

Attacker objective: The practical effect is uncontrolled disclosure and retention of patient data outside the organisation's governance boundary.

1. Entry occurs when clinicians, billing staff, or nurses paste PHI into consumer or embedded GenAI tools during routine work.
2. Credentialed trust is then extended to an external service that may retain prompts, use them for training, or expose them under terms the organisation did not approve.
3. Impact follows when the disclosure violates HIPAA obligations, creates breach-notification exposure, and widens reputational risk around patient data handling.

Breaches seen in the wild

• Samsung ChatGPT leak 2023: Samsung staff pasted chip source code and meeting notes into ChatGPT weeks after it was allowed, leading Samsung to restrict generative AI tools.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

PHI in GenAI is a governance problem because the control boundary moved from the tool to the data path. Healthcare programmes that still think in terms of approved versus blocked applications are already behind the real risk. The decisive issue is whether patient data can be recognised, constrained, and audited after it enters AI workflows. For practitioners, that means policy has to follow the data, not just the application inventory.

Shadow AI in healthcare is often a productivity response, not a policy failure alone. Staff use external GenAI because the tools reduce friction in documentation, appeals, and clinical lookup tasks. If security only blocks without providing safe alternatives, usage will continue in unapproved channels. The named concept here is PHI governance drift: the gradual loss of control that occurs when sanctioned workflows no longer match how clinical work actually happens. Practitioners should treat that drift as an operating-model issue, not a one-off exception.

BAA enforcement must become machine-enforced if the organisation expects it to hold at scale. Manual legal review is too slow for everyday AI interactions, and an approved-vendor list is not enough when data can still leak into a destination before review. The real governance failure is assuming authorisation can be checked after the disclosure. Practitioners need controls that bind AI destination approval to the data itself.

Identity governance now extends to who is allowed to move PHI into which AI context. That is an access decision, a policy decision, and a compliance decision at the same time. In mixed human and machine workflows, the organisation must govern not only the user but also the AI service acting as a data recipient. Practitioners should align AI use policies with IAM, data loss controls, and legal approvals as one control plane.

Healthcare AI governance will increasingly be judged on evidence, not intent. Saying that staff were instructed not to use consumer tools will not matter if the organisation cannot show where PHI went, who sent it, and whether the destination was authorised. That pushes monitoring, auditability, and exception handling into the centre of AI governance. Practitioners should prioritise provable control over policy statements.

From our research library:

• 28% of secrets incidents now originate outside code repositories, in Slack, Jira, and Confluence, and are 13% more likely to be categorised as critical than code-based leaks, according to the State of Secrets Sprawl 2026.
• Only 23% of IT leaders were very confident in their organisation's ability to manage security and governance for GenAI deployments, according to a 2025 Gartner survey of 360 IT leaders.
• Read next: Shadow AI and AI Agent Discovery Guide

What this signals

PHI governance drift: healthcare AI risk now emerges when sanctioned workflows no longer match how staff actually work, especially when embedded copilots and consumer tools blur the boundary between approved and unapproved processing. Security teams should assume that visibility gaps will widen before policy catch-up does, and design controls around data movement rather than tool names.

One-third of employees access AI tools via personal accounts, according to Cyberhaven. That behaviour matters in healthcare because personal-account usage weakens auditability, complicates BAA enforcement, and makes PHI handling harder to prove after the fact.

For practitioners

• Implement data-level PHI visibility Track where patient data moves after it leaves source systems so security teams can see AI prompts, embedded copilots, and external services in one control view.
• Bind AI approval to BAA status Require that any tool receiving PHI is checked for a valid business associate agreement before data reaches it, and block the destination if the agreement is absent.
• Enforce minimum-necessary prompts Apply controls that flag or block prompts containing structured PHI patterns or full patient records when a narrower query would satisfy the task.
• Provide approved alternatives for clinical work Offer sanctioned AI workflows for documentation and summarisation so staff are not pushed toward consumer tools that sit outside compliance oversight.
• Log AI data movement for audit readiness Retain user, tool, destination, and timestamp evidence so compliance teams can demonstrate how PHI was handled if a disclosure must be reviewed.

Key takeaways

• PHI exposure in GenAI tools is not just a DLP problem because the main control failure is loss of governance over where patient data goes once staff start using AI in everyday work.
• The evidence points to a widening gap between approved tools and actual usage, especially where embedded AI features and personal accounts bypass normal review paths.
• Healthcare teams need data-level policy, BAA enforcement, audit trails, and approved alternatives if they want to reduce shadow AI without blocking useful clinical work.

Key terms

• Protected Health Information: Protected Health Information is any health-related data that can identify a person and is covered by HIPAA protections. In practice, PHI can flow through applications, integrations, service accounts, and cloud systems, which is why identity governance matters as much as data governance.
• Business Associate Agreement (BAA): A BAA is a contract that governs how a third party may handle PHI on behalf of a covered entity. It is not just paperwork. It is an enforceable control that defines permitted use, safeguards, and accountability across vendor relationships.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
• Data-Level Policy Enforcement: The practice of applying governance rules directly to the data or field being used instead of relying on each application to implement the rule separately. This is the difference between portable control and fragmented enforcement, especially in AI and analytics environments.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore nhimg.org for resources that connect identity governance to the broader security disciplines your programme depends on.