Independent analysis by NHI Mgmt GroupSource article: WitnessAI, “AI and patient privacy: Risks, challenges, and safeguards in healthcare” (May 16, 2026)

TL;DR: AI is moving Protected Health Information into prompts, ambient documentation, copilots, and agents in ways HIPAA-era controls were not designed to see, while 58% of frontline staff already use generic AI tools at least monthly, according to WitnessAI. The gap is no longer theoretical: privacy programmes must govern live interactions, not just stored records, because inference-time data and shadow AI break older assumptions.

At a glance

What this is: NHI Mgmt Group's analysis, drawing on WitnessAI's blog post: AI is changing patient privacy from a records problem into a live interaction problem, with PHI now flowing through prompts, ambient documentation, copilots and agents.

Why it matters: Healthcare IAM and security teams need to govern who and what can expose PHI at runtime, because legacy access logging and static privacy controls do not cover AI-mediated data movement.

Context

AI in healthcare now moves Protected Health Information through live model interactions rather than only through databases and file shares. That matters because the control point has shifted from storage location to runtime behaviour, where prompts, transcripts, and agent actions can expose PHI before traditional monitoring sees it.

The governance gap is not limited to one workflow. Ambient scribing, chat assistants, and AI agents that query EHR systems each create a different privacy failure mode, but they all strain controls built for static records, known custodians, and clear request boundaries.

For healthcare organisations, the core question is whether privacy programmes can create auditability and policy enforcement at the moment PHI is handled by AI. If they cannot, then compliance, patient trust, and incident response all start from incomplete visibility.

Key questions

Q: What breaks when healthcare teams rely on traditional security controls to protect PHI in AI workflows?

A: Traditional controls often miss the context needed to protect PHI across modern data paths. They can leave misconfigurations, overexposed data, and ROT sprawl unchecked, while AI pipelines introduce new leakage and governance gaps. The result is weaker control over sensitive records, slower compliance response, and greater chance of an incident spreading across systems.

Q: Why does AI use in healthcare create regulatory risk even when a BAA exists?

A: A BAA helps define permitted use, but it does not fully govern every AI training, inference, and subprocessor path. The risk grows when PHI moves through multiple vendors and runtime interactions that the contract does not instrument or audit in real time.

Q: What are the signs that patient data is leaking into shadow AI workflows?

A: Look for staff using unapproved chat tools, ambient transcription systems joining meetings without notice, and AI integrations that appear in network traffic but not in approved inventories. Those are strong indicators that PHI may be moving outside governed workflows.

Q: How should healthcare teams govern AI agents that access clinical systems?

A: Treat AI agents as managed identities with named ownership, scoped permissions, audit trails, and revocation. In healthcare, the governance bar should be higher than for ordinary automation because agents can touch regulated workflows, patient data, and legacy systems. Combine least privilege with human oversight for actions that could affect care delivery or privacy.

Technical breakdown

Why conversational AI changes the PHI control point

Conversational AI treats patient information as context, not as a record in a database. That means PHI can exist in prompts, context windows, logs, embeddings, and downstream summaries at the same time. Traditional controls were built to protect data at rest and in transit, but AI introduces data in use during inference, where the model and surrounding tooling can process or expose sensitive content before legacy monitoring has a chance to act. In healthcare, that creates a control boundary problem as much as a privacy problem.

Practical implication: Treat AI interactions as governed data events, not just user input.

Ambient scribes, copilots and shadow AI create invisible PHI pathways

Ambient documentation tools and general-purpose copilots expand the number of places PHI can travel outside approved systems. When staff paste notes into consumer AI tools or when transcription systems send encounter data to external services, the organisation often loses direct oversight of where the information goes next. The privacy issue is not only unauthorized disclosure, but also unaudited reuse, subprocessor expansion, and the creation of secondary copies that sit outside standard EHR controls. The workflow may look routine to clinicians while still being outside governance.

Practical implication: Inventory every AI workflow that can receive PHI and classify its approval status.

AI agents introduce a non-human identity layer into clinical systems

AI agents are different from simple chat tools because they can act across systems, including EHR-connected workflows. That creates a non-human identity problem: the agent may have credentials, scopes, and tool access that were granted for one purpose but can be used repeatedly at runtime. In this setting, the privacy risk is not only what the model sees, but what the agent is authorised to reach, combine, and disclose on behalf of the organisation. The governance question becomes who can let the agent act, what it can touch, and what evidence proves each action was legitimate.

Practical implication: Apply identity, scope and audit controls to agents that can access clinical systems.

Threat narrative

Attacker objective: The objective is to obtain or propagate PHI through AI-mediated workflows in a way that bypasses the organisation's normal privacy visibility and accountability.

  1. Entry occurs when clinicians, staff, or connected AI tools place PHI into prompts, transcripts, or EHR-linked workflows that sit outside the organisation's traditional privacy boundary.
  2. Credential or access abuse follows when shadow AI, ambient services, or agents use approved integrations, OAuth scopes, or external processing chains to move the data onward.
  3. Impact appears as PHI exposure, unauthorised disclosure, compliance evidence gaps, and potential legal scrutiny when the organisation cannot reconstruct who handled the data and where it went.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI privacy is becoming a runtime governance problem, not a records-management problem. The article shows that PHI can be exposed during prompts, transcription, and agent execution, long before any downstream storage control matters. That breaks the old assumption that privacy is mainly about protecting known repositories. For healthcare teams, the practical conclusion is that policy must be enforced where AI handles data, not only where data is stored.

Shadow AI is the privacy control gap most organisations still underestimate. Frontline use of generic tools makes AI exposure possible even when formal procurement and legal review exist. The control failure is not simply unsanctioned software use, but the absence of visibility into the exact interactions where PHI enters external systems. That means privacy programmes need discovery and evidence, not just acceptable-use language.

Agentic access turns PHI governance into a non-human identity issue. Once an AI agent can query the EHR or connected systems, the organisation must govern machine-initiated access with the same seriousness it applies to privileged service accounts. The old assumption that only humans create meaningful disclosure risk no longer holds. The implication is that healthcare identity programmes now need to cover both user behaviour and delegated machine access in one governance model.

Intent-based control is the more durable privacy concept here than content-only filtering. The article points to semantic prompts, ambient documentation, and re-identification risk that simple pattern matching will miss. A named concept emerges from that pattern: inference-time PHI exposure. That is the window in which sensitive data is handled by the model itself, and it is the point at which healthcare privacy teams must decide what can be seen, tokenized, blocked, or audited.

BAAs and HIPAA paperwork are necessary but no longer sufficient on their own. The article makes clear that modern AI stacks involve multiple vendors, subprocessors, and processing layers that exceed a simple covered-entity-to-business-associate relationship. The governance implication is that accountability has to follow the data flow, not just the contract boundary. Healthcare organisations should expect audit-ready evidence of runtime controls, not only contractual assurances.

From our research library:

What this signals

Inference-time PHI exposure is the governance concept healthcare teams now need to operationalise. Privacy programmes built only around stored records will miss the moment when a prompt, transcript, or agent call turns sensitive data into model input, so the control boundary must move to runtime.

The more AI is embedded in clinical work, the less useful purely static privacy controls become. Discovery, intent-aware policy, and audit evidence need to operate together so that compliance teams can prove what happened inside the interaction, not only what happened after the fact.

For practitioners

  • Map every PHI-bearing AI interaction Build an inventory of prompts, transcripts, ambient documentation, copilots, agent calls, and EHR-connected AI workflows that may process PHI.
  • Classify and restrict shadow AI use Identify consumer AI tools and unmanaged desktop applications that staff use for clinical or administrative work, then separate approved from unapproved paths for PHI.
  • Apply intent-based policy to clinical workflows Define what each role, department, and workflow is allowed to do with PHI in AI systems, including allow, warn, block, and route decisions.
  • Tokenize sensitive content before model exposure Substitute PHI with reversible tokens before data reaches external models, and keep rehydration inside the controlled organisational boundary.
  • Create audit evidence for AI handling of PHI Retain interaction-level logs, policy decisions, and rehydration events so privacy, compliance, and legal teams can reconstruct what happened after the fact.

Key takeaways

  • AI in healthcare is turning patient privacy into a live interaction problem, because PHI now moves through prompts, transcripts, and agent actions that older privacy controls were not built to track.
  • The article ties that risk to widespread shadow AI use, ambient documentation, and EHR-connected agents, all of which expand the number of exposure points beyond conventional record systems.
  • Healthcare teams need runtime visibility, intent-based policy, tokenization, and audit evidence if they want privacy controls to cover the way AI actually handles PHI.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

OWASP Non-Human Identity Top 10 and MITRE ATT&CK address the attack and risk surface, while NIST CSF 2.0 and NIST AI RMF set the governance and control requirements practitioners need to meet.

FrameworkControl / ReferenceRelevance
OWASP Non-Human Identity Top 10NHI-10 — Human Use of NHIClinicians and staff are using AI tools and agents in ways that can expose PHI outside approved governance.
NHI-04 — Insecure AuthenticationAI agents and connected tools accessing EHRs rely on delegated access that must be tightly authenticated and scoped.
Recommendation — Govern every AI tool that handles PHI as an identity-controlled workflow and restrict human use to approved paths. Bind AI system access to strong authentication and explicit authorization boundaries for each clinical workflow.
NIST CSF 2.0PR.AA-05 — Access Permissions, Entitlements and AuthorizationsThe article centres on who and what can access PHI through AI workflows.
Recommendation — Review AI entitlements against PHI handling use cases and remove access that is broader than the workflow requires.
MITRE ATT&CKTA0006;TA0008 — Credential Access; Lateral MovementAgents and connected tools can use legitimate access to move PHI across systems and services.
Recommendation — Map AI workflow abuse to credential access and lateral movement patterns so detections cover delegated access paths.
NIST AI RMFGOVERN — AI Governance and AccountabilityThe article is fundamentally about governing AI use in healthcare and proving accountability.
Recommendation — Establish AI governance ownership for PHI handling and require evidence that controls operate in live workflows.

Key terms

  • Inference-Time PHI Exposure: Exposure that occurs while an AI system is actively processing patient information, rather than when data is stored in a database. In healthcare, this is the point where prompts, context windows, and model outputs can reveal PHI before legacy privacy controls or logging processes can see it.
  • Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
  • Ambient AI Scribe: An ambient AI scribe is a speech-to-text system that listens to clinician patient conversations and drafts clinical notes in real time. It is designed to reduce documentation burden, but it still requires human review because transcription errors, context gaps, and workflow issues can affect the quality of the medical record.
  • Non-Human Identity (NHI): A digital identity assigned to a non-human entity such as a software application, service account, API key, bot, machine, or AI agent that enables it to authenticate and interact with systems without direct human involvement. NHIs now outnumber human identities in most enterprises by 25 to 50 times.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.
NHIMG Editorial Note
Published by the NHIMG editorial team on June 5, 2026.
Updated on October 8, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org