AI risk management in healthcare is outpacing IAM oversight

At a glance

What this is: This is an independent analysis of AI risk in healthcare, showing that unsanctioned tools, clinical decision failures, and autonomous agent actions are outpacing governance.

Why it matters: It matters because healthcare AI now touches PHI, clinical decisions, and operational workflows, so IAM, NHI, and human governance all need enforceable controls at runtime.

By the numbers:

• In 2025, 20% of organizations suffered a breach specifically tied to Shadow AI.
• 86% of healthcare IT executives reported shadow IT instances in their health systems.
• Only 16% of healthcare systems had an enterprise-wide AI governance strategy.
• The average IT budget devoted to governance was roughly 4.2% in 2025.

👉 Read WitnessAI's analysis of AI risk management in healthcare

Context

Healthcare AI risk is no longer a future concern. Unsanctioned tools are already touching patient data, influencing decisions, and automating workflows faster than health systems can supervise them, which leaves policy, audit, and accountability controls behind the operational reality. The primary keyword here is AI risk management, because the governance problem is broader than security alone.

The control gap is structural. Traditional DLP and after-the-fact review were built for static data movement and human-paced decisions, not for conversational AI, agentic APIs, or clinical systems that can act before a reviewer intervenes. That is why the article focuses on runtime oversight, inference-level logging, and policy enforcement at the point of use.

For healthcare leaders, the question is not whether AI can create value. The question is whether the organization can map where AI is used, decide which actions require human approval, and prove that PHI, clinical recommendations, and downstream system calls were governed as they happened.

Key questions

Q: How should healthcare teams govern AI use that touches patient data?

A: They should start with discovery, then enforce policy at the point of use, and finally require auditability for every consequential interaction. That means mapping all AI apps, prompts, model calls, and downstream actions that can touch PHI, then applying runtime controls and identity-linked logs so the organisation can prove who used what, when, and for which workflow.

Q: Why does shadow AI create such a serious risk in healthcare?

A: Shadow AI creates risk because the organisation cannot govern what it cannot see. Unsanctioned tools can move PHI outside approved channels, bypass review, and create unlogged decisions in clinical or administrative workflows. The result is a gap between policy intent and operational reality, which is especially dangerous in regulated environments where evidence matters.

Q: How do runtime guardrails reduce AI risk in clinical workflows?

A: Runtime guardrails reduce risk by inspecting prompts and outputs before they reach the user or downstream systems. That lets teams block PHI exfiltration, warn on risky requests, redirect sensitive prompts to approved models, and apply tokenization where needed. They work because they act in real time, not after harm has already propagated.

Q: Who is accountable when an AI agent takes a harmful action in healthcare?

A: Accountability should remain with the human or team that deployed and authorised the agent, not with the model itself. The organisation needs named ownership, scope definitions, and logs that tie each action to an identity. Without that chain of responsibility, agentic behaviour becomes operationally opaque and difficult to defend in audits or investigations.

Technical breakdown

Shadow AI and PHI exposure in healthcare workflows

Shadow AI is unsanctioned use of AI tools outside approved governance channels. In healthcare, the technical risk is not just that a model may see protected health information, but that the organization loses visibility into where data went, what prompts were entered, and which responses were retained or reused. Legacy DLP often inspects static files and known destinations, while conversational AI exchanges are contextual, dynamic, and often embedded in ordinary work. That creates a blind spot at the exact point where PHI can leak into external model training, browser sessions, or uncontrolled workflows.

Practical implication: treat AI discovery as a visibility problem first, and inventory every AI app, agent, and conversation before policy enforcement.

Prompt injection and runtime defence for clinical AI

Prompt injection is a manipulation technique that steers a model through crafted input, often to reveal data or produce unsafe output. In healthcare, the important distinction is that the attack surface includes both directions of the AI conversation, not just the user prompt. A malicious instruction can ride inside an apparently normal request, while the model response can become the carrier for harmful guidance, unsafe recommendations, or data exfiltration. Runtime defence therefore has to inspect prompts and outputs in real time, because post-hoc review cannot stop a bad recommendation from entering a clinical workflow.

Practical implication: enforce bidirectional runtime controls at the model boundary, not just content review after the fact.

Inference-level audit trails and accountable agent governance

Inference-level audit trails capture the prompt, response, model version, identity, policy action, and downstream system calls tied to each AI interaction. That matters because healthcare governance needs evidence, not only policy statements. The article also extends that logic to AI agents, which can perceive, plan, and act through APIs without a human click at every step. Once an agent can schedule, route, or modify workflows, the audit trail has to show who deployed it, what it could access, and which human identity remains accountable for its actions.

Practical implication: log every AI decision with identity attribution and scope, then require approval checkpoints for high-impact agent actions.

Threat narrative

Attacker objective: The attacker objective is to use AI systems inside healthcare workflows to expose PHI, alter decisions, or trigger harmful downstream actions while avoiding timely oversight.

1. Entry occurs when unsanctioned AI tools, prompt injection, or agentic integrations are allowed into clinical workflows without consistent oversight or discovery.
2. Credential access or abuse happens when models, plugins, or agent connections can read PHI, influence recommendations, or call downstream systems without enforced policy gates.
3. Escalation follows when a manipulated prompt or autonomous agent action propagates into billing, care coordination, or clinical decision processes before a human reviews it.
4. Impact is realized as PHI exposure, unsafe clinical recommendations, compliance failure, legal liability, and operational decisions that cannot be cleanly audited after the fact.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI risk management in healthcare is really an identity governance problem with clinical consequences. The article is not just describing unsafe tooling, it is describing a control environment where identities, models, and agents are making consequential decisions without consistent attribution or review. That pushes the issue into IAM, NHI, and lifecycle governance at once. The practitioner conclusion is simple: if you cannot govern who or what acted, you cannot govern the clinical workflow.

Shadow AI creates a governance gap because discovery fails before policy can begin. The article shows that organizations cannot enforce controls over AI usage they have not discovered, and that is a visibility failure before it is a security failure. In NHI terms, the problem is unmanaged access paths into model services and AI-enabled workflows that were never brought under inventory. The practitioner conclusion is to treat AI discovery as the prerequisite to every other control.

Inference-level auditability is the new evidence standard for AI in regulated environments. The article correctly ties defensible governance to prompt, response, model version, and action logging, because post-hoc application logs do not explain AI behaviour well enough for healthcare oversight. This aligns with NIST Cybersecurity Framework thinking on traceability and accountability, but it also exposes a practical reality: if the system cannot prove what happened at inference time, compliance claims are fragile. The practitioner conclusion is to demand audit evidence at the interaction layer, not only at the application layer.

Autonomous AI agents collapse the assumption that a human is the stable operator behind each consequential action. Human approval workflows were designed for conditions where access persists long enough to be reviewed, attributed, and certified. That assumption fails when an agent can perceive, plan, and act through APIs before a reviewer intervenes, because the decision loop is no longer human-paced. The implication is that healthcare governance has to rethink accountability, not just add a control.

Bidirectional runtime defence is becoming the practical boundary between controlled and uncontrolled AI use. The article shows why governance must inspect both prompts and responses, because the risk is not confined to input filtering. For healthcare teams, this means policy has to operate at the point of interaction, across employee use, models, and agents. The practitioner conclusion is that runtime enforcement is now part of operational identity control, not an optional AI overlay.

From our research:

• 72% of organisations have experienced or suspect they have experienced a breach of non-human identities, 46% confirmed and 26% suspected, according to The 2024 ESG Report: Managing Non-Human Identities.
• Enterprises that have experienced a compromised NHI averaged 2.7 separate incidents in the past 12 months, a pattern that shows identity failure compounds quickly once controls are weak.
• That is why the NHI Lifecycle Management Guide is the next step for teams building inventories, rotation, and offboarding discipline.

What this signals

Healthcare AI governance is converging with identity governance because the real control question is no longer only what the model says, but what identity was allowed to act, access, or delegate in the first place. The programmes that will hold up are the ones that can connect policy to runtime enforcement, audit trails, and accountable ownership across employees, models, and agents.

Inference accountability gap: healthcare leaders should expect pressure to prove AI decisions at the interaction level, not only at the application level. As unreviewed AI use spreads through EHRs, revenue cycle tools, and patient communication systems, the burden shifts from policy creation to evidence generation, and that changes how compliance, security, and clinical safety teams work together.

If your organisation is still treating AI as a sidecar to existing security controls, the programme is already behind. A practical response is to pair discovery with enforcement, then use standards-based governance such as the NIST Cybersecurity Framework 2.0 to connect visibility, protection, detection, response, and recovery.

For practitioners

• Inventory every AI access path Map every AI application, model endpoint, agent integration, and conversational workflow that can touch PHI or operational systems. Include shadow AI, browser-based tools, vendor-enabled features, and MCP-style connections so discovery is complete before enforcement begins.
• Enforce bidirectional runtime controls Inspect both prompts and responses at the point of use, and block, warn, redirect, or tokenize sensitive content before it reaches external models. Pair those controls with policy exceptions for approved internal workflows so clinicians are not forced into shadow usage.
• Bind AI actions to accountable identities Require identity attribution for every prompt, model call, and downstream action, including autonomous agent activity. Record who deployed the agent, what it can access, and which human owner remains responsible for consequential outputs.
• Separate low-risk assistance from high-impact decisions Define which AI uses may assist drafting, summarization, or routing, and which uses must stop before care plans, claims, or patient-facing communications. Keep approval checkpoints for the actions that change outcomes, not just the ones that process information.

Key takeaways

• AI risk in healthcare is a governance and identity problem as much as a model-safety problem, because unsanctioned tools, agents, and workflows can act without consistent oversight.
• The scale of the issue is already measurable, with shadow AI, weak governance coverage, and breach exposure showing that current controls are lagging operational reality.
• Healthcare teams need discovery, runtime policy enforcement, and inference-level audit trails if they want to use AI without losing accountability or compliance evidence.

Key terms

• Shadow AI: Shadow AI is the use of AI tools, models, or agents outside approved governance channels. In healthcare, that means the organisation may lose control over PHI, auditability, and acceptable use before security or compliance teams even know the tool exists.
• Inference-level audit trail: An inference-level audit trail records the prompt, response, model version, policy action, and downstream system calls for each AI interaction. It is the evidence layer that lets regulated organisations prove what happened at the moment the model or agent acted.
• Runtime guardrails: Runtime guardrails are controls that inspect and shape AI behaviour while the interaction is happening, not after the fact. They can warn, block, redirect, or redact based on policy, which makes them essential when AI is embedded in live clinical or administrative workflows.
• Agentic AI: Agentic AI is software that can perceive, plan, and act across tools or APIs with a degree of independent execution. In governance terms, it requires accountability, scope, and monitoring because its decisions can propagate into systems before a human reviews them.

Deepen your knowledge

AI risk management in healthcare is covered in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building governance for agents, models, and shadow AI in a regulated environment, it is worth exploring.

This post draws on content published by WitnessAI: AI risk management in healthcare and the practical controls needed to govern it. Read the original.