Who is accountable when AI recommendations conflict with patient preferences or a clinician’s judgment?

Why This Matters for Security Teams

When AI recommendations conflict with patient preference or clinician judgment, the risk is not just a bad suggestion. It is a governance failure that can distort informed consent, delay escalation, and create ambiguity about who owned the final decision. NIST’s NIST SP 800-53 Rev 5 Security and Privacy Controls is clear that accountability and decision responsibility must be assigned, not implied.

In healthcare, this problem is especially sensitive because AI outputs can influence triage, medication choices, and follow-up actions while still carrying uncertainty. The operational question is not whether the model was “right,” but whether the organisation defined who reviews, who overrides, and how exceptions are documented. That is why NHIMG treats AI accountability as part of broader NHI and agent governance, not as a model quality issue alone. The same control discipline that matters in Gemini CLI Breach — Silent Code Execution and DeepSeek breach also applies when AI is inserted into clinical decision support. In practice, many security and governance teams discover accountability gaps only after an override dispute, not through deliberate workflow design.

How It Works in Practice

Accountability should remain with the health care organisation, the treating clinician, and the governance process that permits AI use. The model can generate recommendations, but it cannot accept responsibility, explain policy exceptions, or weigh patient values. That means the organisation must define a human decision owner, a review path for contradictory outputs, and an auditable record of why the final choice was made.

In practice, strong programs separate three functions: recommendation generation, clinical evaluation, and final disposition. The AI may flag a risk or suggest an intervention, but the clinician decides whether the recommendation fits the patient’s condition, consent status, and treatment goals. Where a recommendation conflicts with patient preference, the record should show whether the patient declined, whether alternatives were offered, and whether the AI output was considered or rejected.

• Assign a named clinical owner for every AI-assisted workflow.
• Require explicit override capability when the AI conflicts with patient preference or clinician judgment.
• Log the AI output, the human review, the rationale, and any escalation.
• Use policy and training to reduce automation bias, especially in high-pressure care settings.

This governance should align with access and audit expectations in NIST SP 800-53 Rev 5 Security and Privacy Controls and with emerging AI accountability guidance from NIST AI risk practice. It also benefits from lessons in NHIMG research, especially when AI-facing systems expose secrets or tool access that can amplify bad decisions if the workflow is not constrained. Current guidance suggests the safest model is not “AI decides, human approves,” but “human decides, AI informs.” These controls tend to break down in emergency care pathways with overloaded staff because review steps become informal and exceptions go undocumented.

Common Variations and Edge Cases

Tighter review controls often increase clinical friction, requiring organisations to balance safety against speed of care. That tradeoff is unavoidable in emergency medicine, specialist consults, and remote triage where delays can harm patients. Best practice is evolving, but there is no universal standard for how much autonomy an AI recommendation may have before it becomes a de facto clinical decision.

One common edge case is soft conflict, where the AI does not directly contradict the clinician but nudges the team toward a higher-risk intervention. Another is patient refusal, where the recommendation is clinically sound but the patient declines after informed discussion. A third is delegated review, where a nurse, pharmacist, or care coordinator sees the AI output before the attending clinician. In all of these cases, accountability still sits with the organisation’s policy chain, but documentation must show who had authority to act and who merely relayed the recommendation.

Healthcare organisations should also be careful with vendor language that implies “clinical AI independence.” That framing weakens responsibility. The better pattern is to define the system as decision support unless a regulated, validated workflow explicitly authorises otherwise. NHIMG analysis of incident patterns shows that ambiguous ownership is what creates avoidable drift between recommendation and care action, not the model output itself.

Related resources from NHI Mgmt Group

• Who is accountable for auditability when agentic AI activity is used in regulated environments?
• Who is accountable when an event registration process mishandles personal data or tracking preferences?
• Who should be accountable for securing API and AI platform traffic in an enterprise environment?
• Who is accountable when AI risk, privacy, and security controls are managed in separate programmes?