How should organisations govern PHI in generative AI workflows?

They should govern PHI as a data-flow problem, not just an application problem. That means enforcing identity controls, selective logging, encryption, retention rules, and auditability across prompts, retrieval systems, embeddings, outputs, and telemetry. If any one layer can copy or expose patient data without policy checks, the overall workflow is not sufficiently controlled.

Why This Matters for Security Teams

PHI in generative ai workflows is not just sensitive content, it is regulated healthcare data that can be copied, transformed, retrieved, and exposed in ways that traditional application controls do not fully capture. The security problem expands across model prompts, retrieval-augmented generation, vector stores, caches, transcripts, and monitoring systems. Governance therefore has to cover the entire data path, not only the user interface or the model endpoint.

Security teams often underestimate how quickly PHI leaves the original control boundary once it enters search indexes, prompt histories, or telemetry pipelines. That creates privacy, compliance, and breach notification risk, especially where access decisions are implicit rather than policy-driven. Current guidance from the NIST Cybersecurity Framework 2.0 supports this broader view by tying governance, data protection, and monitoring together rather than treating them as separate tasks. In practice, many security teams encounter PHI leakage only after logs, exports, or model outputs have already been shared beyond the intended clinical workflow.

How It Works in Practice

Effective governance starts by classifying PHI before it enters any generative AI flow, then enforcing policy at each processing stage. That means deciding whether PHI may be used at all, whether it can be sent to external services, and whether it must be redacted, tokenised, minimised, or kept entirely out of the model path. The control objective is consistent: only authorised identities should be able to submit, retrieve, store, or observe PHI-bearing content.

A practical control set usually includes:

• Identity-aware access control for clinicians, analysts, developers, and service accounts
• Prompt filtering and output inspection to prevent unnecessary disclosure
• Encryption in transit and at rest for transcripts, embeddings, caches, and audit records
• Retention limits for prompts, completions, and telemetry, with explicit deletion workflows
• Audit logging that is selective enough to be useful without creating a secondary PHI repository

For model risk governance, the NIST AI 600-1 Generative AI Profile is useful because it translates AI risk management into operational practices such as traceability, monitoring, and documentation. In healthcare environments, that should be paired with data lineage controls so teams can answer where PHI came from, who accessed it, which model touched it, and whether it was reproduced in an output or log. Best practice is evolving around whether embeddings themselves should always be treated as PHI, but many organisations now treat them as sensitive derived data when they can be linked back to a patient record. These controls tend to break down when third-party plugins, ad hoc analyst notebooks, or unmanaged exports bypass the approved workflow because policy enforcement no longer follows the data.

Common Variations and Edge Cases

Tighter PHI controls often increase operational overhead, requiring organisations to balance clinical usability against privacy, latency, and support costs. That tradeoff becomes more visible in workflows where staff expect fast conversational access to patient context, yet governance requires redaction, approval gates, or restricted retention.

There is no universal standard for every generative AI use case, so the right approach depends on whether the system is assisting diagnosis, summarising records, answering administrative questions, or powering a back-office copilot. A low-risk administrative workflow may tolerate more automation than a clinical decision-support workflow, where PHI exposure has higher consequence. In some environments, the safest design is to keep PHI out of the model entirely and use retrieval from a governed record system instead of prompt injection of raw patient data.

Identity governance also matters here because service accounts, API keys, and agent credentials can become non-human identities with broad access to records and telemetry. That intersection is often missed until an integration is granted more visibility than a human clinician would receive. Where vendors handle hosting or fine-tuning, organisations should verify contractual limits on data reuse, logging, and training. The NIST AI 600-1 GenAI Profile reinforces the need for ongoing monitoring, but it does not remove the need for local policy decisions about PHI retention and disclosure. The pattern fails most often in environments with fragmented ownership, because privacy, security, and clinical teams each assume another control layer is already handling the PHI.

Related resources from NHI Mgmt Group

• How should healthcare organisations govern AI chatbots that can access PHI?
• How should healthcare organisations govern AI tools that handle PHI?
• How should organisations govern RAG-based AI workflows?
• How should organisations govern AI marketing workflows that touch customer data and claims?