AI Security in Healthcare Guide

Clinical AI now listens to consultations, drafts notes, summarises records and suggests decisions, which puts protected health information into prompts, transcripts and model logs that most hospital security programmes were never designed to see. The risks are specific to healthcare: a hallucinated medication in a discharge summary is a patient safety incident, an AI vendor that stores audio is a HIPAA business associate, and a generative summarisation feature may make a product a regulated medical device. This guide covers the main clinical AI patterns (ambient scribes, chart summarisation, patient-facing assistants, AI-enabled devices and agents), the regulatory duties that apply in the US, UK and EU, the security controls that differ from general AI security, and how to roll them out and measure them. It is for CISOs, clinical safety officers, information governance leads and architects in providers, payers and health technology suppliers. Clinical identity and access are covered in the Healthcare Identity Security Guide.

Key takeaways

Every AI vendor touching PHI is a business associate. HHS guidance on cloud services states that lacking an encryption key does not exempt a provider from business associate status; get a BAA before the first pilot.
Some clinical AI is a medical device. NHS England guidance says simple transcription is likely not a device but generative summarisation likely is; classification drives MHRA, FDA and EU obligations.
Clinicians must review outputs. NHS England requires users to review and approve AI scribe outputs before further action; make that review a logged, enforced step, not an assumption.
Prompt injection is a clinical risk. Records, referral letters and patient messages are untrusted input; NHS guidance names indirect prompt injection explicitly.
AI identities need clinical-grade scoping. Agents and scribes should access only the current patient and task, through their own identities, with orders and prescriptions gated by a clinician.

Clinical AI patterns and their risks

Pattern	Data involved	Main security and safety risks	Priority controls
Ambient scribe (ambient voice technology)	Consultation audio, transcripts, draft notes	Audio retention and secondary use, hallucinated or omitted findings, recording without proper notice, supplier access	BAA or DPA, retention limits per data type, clinician sign-off, clinical safety case
Chart summarisation and inbox drafting	Full record extracts, messages	Over-broad record access, sensitive information disclosure in outputs, prompt injection from documents	Patient-scoped retrieval, output review, input sanitisation
Patient-facing chatbot or triage assistant	Symptoms, identifiers, appointment data	Unsafe advice, impersonation, data leakage to other users, transparency duties	Strong guardrails, escalation to humans, AI disclosure, authentication before PHI
AI-enabled medical device (imaging, monitoring)	Images, signals, device data	Model tampering, adversarial inputs, insecure updates, vulnerable software components	Manufacturer SBOM and patch commitments, segmentation, change control
Clinical and operational agents (coding, prior authorisation, scheduling)	Records plus write access to systems	Excessive agency, wrong-patient actions, credential misuse	Dedicated identities, least privilege, human approval for consequential actions
Staff use of public chatbots	Whatever is pasted in	PHI disclosed to a service with no BAA	Approved tools, data loss prevention, training

Regulatory duties that apply

Jurisdiction	Requirement	What it means for AI
US: HIPAA Privacy and Security Rules	Business associate agreements; Security Rule safeguards; minimum necessary use	AI vendors that create, receive, maintain or transmit ePHI need a BAA; the narrow conduit exception covers only transmission-only services
US: Section 1557 nondiscrimination, 45 CFR 92.210	Patient care decision support tools	Covered entities have a continuing duty to make reasonable efforts to identify tools that use race, colour, national origin, sex, age or disability as inputs and to mitigate discrimination risk
US: FDA	Cybersecurity in Medical Devices final guidance (current version February 2026, covering section 524B cyber devices); AI-Enabled Device Software Functions draft guidance (January 2025)	Manufacturers document cybersecurity and AI lifecycle risk management in submissions; hospitals should ask for that evidence. The FDA's AI-Enabled Medical Device List says it will explore tagging devices built on large language models
UK: NHS England ambient scribe guidance	DTAC, DCB0129 (supplier) and DCB0160 (deploying organisation) clinical safety standards, DPIA, MHRA classification, DSPT and Cyber Essentials	Assign a Clinical Safety Officer, keep a safety case and hazard log, limit pilots to four months, confirm medical device status independently
EU: AI Act	High-risk requirements for AI that is a safety component of, or is, a product under medical device law; Article 50 transparency	The Commission states high-risk rules for AI in regulated products will apply from 2 August 2028 following the AI omnibus agreement; Article 50 disclosure duties apply from 2 August 2026
EU and UK: GDPR	Health data is special category data; DPIA for high-risk processing	Lawful basis and condition for processing, retention limits, data subject rights over transcripts

The full regulatory picture is in the Healthcare Cyber Regulation Map, and device security in the FDA Medical Device Cybersecurity Guide and EU MDR Cybersecurity Guide. EU AI Act dates above reflect the Digital Omnibus amendments (Regulation (EU) 2026/1744, in force 27 July 2026); check the Commission's AI Act Service Desk before relying on them.

Protecting PHI in prompts, transcripts and logs

Map every copy: for each AI tool, list where audio, transcripts, prompts, retrieved record extracts, outputs and logs are stored, by whom, and for how long. NHS England asks organisations to set retention for each data type: interaction audio, transcripts, user interactions and final outputs.
Delete audio early: once a note is signed, raw audio rarely needs to persist. Short retention removes the most sensitive copy.
Control secondary use: contracts should prohibit training on your PHI unless you have explicitly agreed, and should state where data is processed. Under HIPAA, any use outside the BAA is a potential breach.
Apply minimum necessary to retrieval: a summarisation tool should retrieve only the current patient's relevant records, not the whole chart or other patients. See the RAG Security Guide.
Keep logs useful but safe: prompt and output logs are PHI. Store them in the same protected tier as the record, with access logged, and avoid sending them to general observability tools without a BAA.
Use de-identification carefully: de-identified data used for model development must meet HIPAA de-identification standards; re-identification risk from free text is high.
Block unapproved tools: provide an approved assistant under a BAA, and use DLP and web controls to stop PHI being pasted into consumer chatbots. See the Shadow AI Discovery Guide.

AI identities and access in clinical systems

Scribes, summarisers and agents reach the EHR through integrations, FHIR APIs and service accounts. These are non-human identities with access to some of the most sensitive data in the organisation.

Identity	Recommended access	Controls
Scribe integration writing draft notes	Write drafts to the encounter in progress, no read beyond what is needed	Dedicated client credentials per supplier, scoped to draft note creation, signed by the clinician
Summarisation service	Read the current patient's record in the context of the clinician's session	Delegated access tied to the clinician's session and patient context; no standing bulk read
Operational agent (scheduling, coding, prior authorisation)	Task-specific APIs	Own identity, least privilege, rate limits, human approval for submissions and orders
Supplier support access	Time-limited, approved sessions	Brokered remote access with MFA and recording

Never share clinician credentials with tools: a scribe acting as the clinician's account makes every AI action indistinguishable from the clinician's in the audit trail.
Enforce limits outside the model: patient context and allowed actions should be enforced by the API and authorisation layer, not by the prompt. See the AI Agent Authorisation Guide.
Gate consequential actions: orders, prescriptions, referrals and claims submissions require clinician or staff approval. See the Human-in-the-Loop for AI Agents Guide.
Remove anonymous paths: the OpenAI agent Medicare portal incident saw an agent reach non-public aggregate health statistics, with archived code suggesting an automatic guest sign-in path. Agents will find any unauthenticated route that humans ignore.
Protect remote access: AI suppliers add new remote access paths. The Change Healthcare breach began with a stolen login on a remote access portal without MFA and disrupted claims and pharmacy payments across the US for weeks.

Clinical safety, prompt injection and output integrity

In healthcare an integrity failure is a safety failure. Treat security threats to AI outputs as clinical hazards in the safety case.

Indirect prompt injection: instructions hidden in a referral letter, scanned document or patient portal message can steer a summariser to omit findings or an agent to take actions. Sanitise and label untrusted content, restrict tool use when it is in context, and test with injected documents. See the Prompt Injection Guide.
Hallucination and omission: record known failure modes (wrong drug, wrong dose, invented history, missed negation) in the hazard log and design review screens that show the source for each statement.
Wrong-patient errors: verify that the patient context in the AI session matches the open record, especially when clinicians switch quickly between patients.
Model and configuration changes: supplier model updates can change behaviour overnight. Require notice of changes, re-run your evaluation set and record the change in the safety case, mirroring the change control expected of devices.
Bias: for decision support tools, Section 1557 requires reasonable efforts to identify and mitigate discrimination risk, which needs evaluation across patient groups.

Supplier assurance questions for clinical AI

Most health organisations buy rather than build clinical AI, so supplier assurance carries much of the security load. Add these questions to your standard security questionnaire and contract schedule.

Area	Question	Good answer
Data use	Is any customer PHI, audio or transcript used to train or improve models?	No, unless the customer opts in in writing; contractually enforced
Retention	How long are audio, transcripts, prompts and outputs retained, and can we set it?	Configurable per data type, with audio deleted after note sign-off
Sub-processors	Which model providers and hosting regions process our data?	Named list, all covered by downstream BAAs or DPAs, notification of changes
Regulatory status	What is the product's medical device classification in each market, and on what basis?	Documented classification with registration evidence where it is a device
Model changes	How are model and prompt changes tested and communicated?	Advance notice, release notes, customer re-validation window
Security testing	Has the product been tested for prompt injection and cross-patient data leakage?	Recent independent test report with findings and fixes
Integration identity	How does the product authenticate to our EHR, and can access be scoped per function?	Dedicated client credentials, scoped tokens, no shared clinician accounts
Logging	Can we receive audit logs of user and AI actions in our SIEM?	Yes, in a documented format, retained to our policy

Implementing an AI security programme in a health organisation

Inventory AI in use, including features switched on inside existing EHR, imaging and productivity products, and staff use of public tools.
Classify each use by data (PHI or not), function (administrative, clinical support, device) and autonomy (drafts only, acts with approval, acts alone).
Complete the paperwork before the pilot: BAA or data processing agreement, DPIA, DTAC and DCB0160 in England, medical device classification, security questionnaire with AI-specific questions.
Design identities and access for each integration as above, with logging into the SIEM.
Pilot in a limited setting (NHS England caps pilots at four months), with clinician review mandatory and accuracy audited against a sample of encounters.
Test security and safety together: prompt injection through documents, data leakage across patients, behaviour under supplier outage, and rollback to manual documentation.
Scale with monitoring: track the metrics below, review incidents in clinical governance, and re-assess on every major model or product change.

Small practices versus health systems: a GP or dental practice should choose tools from assured supplier routes, insist on a BAA or DPA and short audio retention, and rely on the supplier's evidence. A health system needs a central AI register, an AI review board joining security, clinical safety, privacy and informatics, and its own evaluation set. See the Cyber Security for Dental and GP Practices Guide.

Detection and measurement

Log: AI service identity, user, patient context, records retrieved, actions taken and approvals, plus supplier access sessions.
Detect: AI identities accessing patients outside the clinician's current context, bulk retrieval, actions without a matching approval, PHI flowing to unapproved AI domains, and changes to integration credentials.

Metrics: percentage of AI tools touching PHI with a signed BAA or DPA (target 100 percent); percentage of AI integrations using dedicated scoped identities; clinician sign-off rate on AI-drafted notes; audited accuracy and omission rate on a monthly sample; median audio retention period; number of PHI uploads to unapproved AI services blocked.

Common failure patterns

Pilot before paperwork: a department trials a scribe with live patients before the BAA, DPIA and clinical safety case exist.
Supplier self-declaration accepted: a product described as "not a medical device" that generates clinical summaries, contrary to the classification view in NHS England guidance.
Clinician account reuse: AI tools acting through clinicians' credentials, destroying accountability.
Indefinite audio storage: raw consultation recordings retained by default in a supplier cloud.
Review that is not a review: notes auto-filed if the clinician does not reject them within a time window.
Unscoped retrieval: a summariser able to query any patient, so one injected instruction can exfiltrate other records.

Lessons from incidents

Published AI-specific healthcare breaches are still few, but two live cases carry the lessons. The OpenAI agent incident on the Australian Medicare statistics portal shows that agents will use unauthenticated paths to non-public health data and that notification can lag: OpenAI took 84 days to notify. The Change Healthcare breach shows how one remote access login without MFA can halt a national health payment system, which is why every new AI supplier connection deserves the same remote access controls as any other third party.

Framework mapping

Control area	HIPAA Security Rule	NIST CSF 2.0	Threat reference
Business associate assurance	45 CFR 164.308(b)	GV.SC-05	Third-party risk
Access control for AI identities	45 CFR 164.312(a)	PR.AA-05	OWASP LLM06 Excessive Agency
Audit controls	45 CFR 164.312(b)	DE.CM-09	MITRE ATLAS AML.M0024 AI Telemetry Logging
Integrity of clinical outputs	45 CFR 164.312(c)	PR.DS-01	OWASP LLM01 Prompt Injection; ATLAS AML.T0051
Sensitive information disclosure	Minimum necessary (45 CFR 164.502(b))	PR.DS-02	OWASP LLM02 Sensitive Information Disclosure

Practitioner checklist

Build a register of every AI tool and feature touching PHI, including embedded EHR features.
Sign a BAA or DPA with every AI supplier before any patient data is used.
Classify each tool's medical device status independently of the supplier's claim.
Complete DPIA and clinical safety documentation (DCB0160 in England) before pilots.
Set retention per data type and delete consultation audio once notes are signed.
Give each AI integration its own scoped identity, tied to patient context, with no clinician credential sharing.
Require logged clinician approval for AI-drafted notes and any order, prescription or submission.
Test for indirect prompt injection through documents and messages.
Block PHI uploads to unapproved AI services and provide an approved alternative.
Re-evaluate tools on every supplier model change.

AI Security in Healthcare Guide: Protect PHI, clinicians and patients across clinical AI