The main breakdown is governance. Teams often lose a clear audit trail, cannot prove minimum necessary access, and may not be able to demonstrate compliant handling during an investigation. That weakens breach response, makes policy enforcement inconsistent, and creates exposure around unauthorized disclosure, especially when users copy clinical notes or identifiers into prompts.
Why This Matters for Security Teams
When ChatGPT enters a workflow that touches PHI, the issue is rarely just “someone used an AI tool.” The real problem is that the organisation may not be able to prove where data went, who approved the use, or whether disclosure stayed within policy. That creates a governance gap across privacy, security, and legal review, especially when staff treat a conversational interface like a controlled clinical system. The NIST Cybersecurity Framework 2.0 is useful here because it forces teams to think about governance, data handling, and accountability together rather than as separate after-the-fact checks.
For PHI, the failure is not only accidental exposure. It is also the loss of defensible process. If a team cannot show purpose limitation, access restriction, retention control, and review of outputs, it becomes difficult to demonstrate that the AI use was appropriately governed. That matters during incident response, compliance review, and vendor due diligence. It also matters because AI workflows often spread informally before security teams are aware of them, which means the technical risk arrives after the behaviour is already embedded in operations. In practice, many security teams encounter PHI leakage only after staff have already normalised prompt-based workarounds instead of through intentional control design.
How It Works in Practice
Most breakdowns happen at the boundary between workflow convenience and control enforcement. Staff paste clinical notes, discharge summaries, insurance details, or patient identifiers into a general-purpose chatbot to draft messages, summarise records, or speed up administrative work. If the platform is not approved for PHI, the organisation may lose visibility into logging, retention, model training use, and data residency. Even when a platform claims strong privacy features, the security team still has to validate the actual configuration, contractual terms, and user behaviour.
A practical control set usually includes:
- data classification rules that explicitly treat PHI as restricted input for public or unmanaged AI services;
- approved use cases that define what can be summarised, transformed, or analysed;
- technical guardrails such as DLP, SSO, logging, and prompt filtering where feasible;
- workflow redesign so staff use de-identified or minimum-necessary data instead of raw records;
- incident playbooks that address AI-generated disclosure, not just classic exfiltration.
From a governance perspective, this aligns with privacy-by-design expectations and the need to document access, purpose, and oversight. Where AI is used to support regulated health workflows, teams should also examine whether the system creates a new processing activity that needs legal review, retention controls, and user training. Guidance from OWASP Top 10 for Large Language Model Applications is helpful for identifying prompt injection, sensitive data exposure, and output handling risks that often appear in real deployments.
These controls tend to break down when PHI is embedded in free-form unstructured text pipelines, because staff cannot reliably separate clinical convenience from approved disclosure paths.
Common Variations and Edge Cases
Tighter PHI controls often increase workflow friction, requiring organisations to balance clinical speed against privacy assurance. That tradeoff is real, especially in environments where staff need rapid turnaround for documentation, triage, or patient communications. Best practice is evolving, but there is no universal standard for allowing unsanitised PHI into general-purpose AI tools; policy decisions still depend on risk appetite, jurisdiction, and contractual safeguards.
Some teams try to solve the problem by banning AI entirely, but that can push usage into shadow IT. Others permit broad use with vague warnings, which is worse because it creates the appearance of control without the evidence. The better approach is to define approved patterns for de-identified inputs, restrict human-entered PHI, and require review before any AI-generated text is reused in official records. Where AI outputs are used to draft patient-facing content, organisations should also validate that the output does not infer additional sensitive information beyond the original input. For healthcare environments that already struggle with identity governance, the issue can extend to who is authorised to submit PHI into an AI tool in the first place, not just what the tool returns.
The operational edge case is highest-risk when the workflow spans multiple systems, such as EHR export, browser-based AI use, and manual copy-back into a record system, because control ownership becomes fragmented and auditability degrades.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 address the attack surface, NIST CSF 2.0, NIST AI RMF and NIST SP 800-63 set the technical controls, and PCI DSS v4.0 define the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | GV.RM-01 | PHI-in-AI workflows need governance and risk ownership, not ad hoc tool use. |
| OWASP Agentic AI Top 10 | Sensitive Data Exposure | Chat interfaces often expose PHI through prompts, outputs, or retained conversation history. |
| NIST AI RMF | GOVERN | AI governance is required to show accountability, policy, and oversight for PHI use. |
| NIST SP 800-63 | Identity assurance matters when deciding who may input PHI into AI workflows. | |
| PCI DSS v4.0 | 3.2 | Although built for payment data, the control logic mirrors restrictions on sensitive data exposure. |
Treat PHI like restricted data and prevent unnecessary storage or transmission in AI tools.
