The covered entity remains accountable for how its workforce uses AI, even when the provider signs a BAA. That includes product-tier selection, identity configuration, training, monitoring, and response to misuse. The vendor has obligations under the BAA, but the organisation owns the controls that prevent disclosure in the first place.
Why This Matters for Security Teams
PHI exposure through ChatGPT Enterprise is rarely a model problem alone. It is usually an identity, access, and governance problem that lands on the covered entity because the organisation decided how the tool was deployed, who could use it, and what data could reach it. A BAA can define vendor obligations, but it does not replace workforce policy, access control, or incident response. Current guidance suggests that the practical risk comes from misconfigured accounts, overbroad sharing, and weak supervision, not from the interface itself. NHIMG research shows that secrets and identity controls fail at scale when visibility is poor, and that pattern maps closely to enterprise AI misuse. See The 52 NHI breaches Report and Ultimate Guide to NHIs — Why NHI Security Matters Now for the control failures that most often precede disclosure. In practice, many security teams encounter PHI leakage only after a user has already pasted sensitive content into an approved AI workspace, rather than through intentional misuse testing.
How It Works in Practice
Accountability follows control, not convenience. If a covered entity enables ChatGPT Enterprise, it must decide whether the product tier, tenant settings, retention options, and identity bindings are appropriate for PHI. That includes deciding who can use the service, whether role-based access control is narrow enough, whether single sign-on and conditional access are enforced, and whether employees have been trained on what constitutes PHI. The vendor may process data under contract, but the organisation remains responsible for preventing unnecessary disclosure.
Operationally, the safest pattern is to treat the AI workspace like any other privileged system. Require workforce identity, enforce least privilege, and separate general productivity use from workflows that might involve PHI. Use policy-as-code and request-time checks where possible, because static rules age quickly once users start experimenting with prompts and file uploads. Anthropic’s report on an AI-orchestrated cyber espionage campaign shows how quickly autonomous or semi-autonomous tool use can amplify mistakes, which is why identity boundaries matter even when the platform is enterprise-grade. For identity and access design, the issues align closely with the abuse patterns discussed in 52 NHI Breaches Analysis and with external controls guidance such as the Anthropic — first AI-orchestrated cyber espionage campaign report.
- Bind access to named users and device posture, not shared accounts.
- Minimise prompt and file exposure paths for PHI.
- Monitor usage, exports, and unusual sharing patterns.
- Document incident steps for accidental disclosure and retention concerns.
These controls tend to break down in highly distributed environments where users can move PHI between sanctioned SaaS tools, browser extensions, and copied text without a single enforcement point.
Common Variations and Edge Cases
Tighter controls often increase friction, requiring organisations to balance clinical productivity against data minimisation. That tradeoff is real, especially where employees need AI assistance for summarisation, drafting, or triage. Best practice is evolving, but there is no universal standard that says every PHI use case must be banned; instead, many organisations restrict the data classes allowed in the tool and route high-risk workflows through approved, audited systems.
The accountability question can also shift when the AI is used by a business associate, a contractor, or an internal team building agentic workflows around the chatbot. In those cases, responsibility may be shared contractually, but the covered entity still owns due diligence, access review, and breach response coordination. This is where NIST AI RMF, OWASP-AGENTIC, and CSA-MAESTRO become useful: they push teams to define who can act, under what intent, with what logging, and with what revocation path. If the organisation allows uploaded documents, custom connectors, or automated downstream actions, then the governance model must cover the full data path, not just the chat window. The underlying lesson from the NHI guidance remains the same: identity and secret sprawl create exposure long before an incident is reported, as reflected in the broader patterns documented in Ultimate Guide to NHIs — Why NHI Security Matters Now and the breach trends in The 52 NHI breaches Report.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and CSA MAESTRO address the attack and risk surface, while NIST AI RMF set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | Defines governance and accountability for AI risk management. | |
| OWASP Agentic AI Top 10 | Covers misuse paths when AI tools can act or chain actions. | |
| CSA MAESTRO | Provides security controls for agentic and workflow-based AI systems. |
Map ChatGPT Enterprise workflows to control points for identity, intent, telemetry, and revocation.
