Should healthcare organisations prioritise runtime baselines over EHR audit logs for AI agents?

Why runtime baselines should lead when AI agents touch clinical workflows

Runtime baselines are the better primary control when the question is whether an AI agent is behaving safely enough to influence care. They let teams compare actual agent behaviour against an expected pattern, including tool use, timing, scope, and escalation path. EHR audit logs still matter, but they mostly tell you what was recorded after the fact.

The practical distinction is that an audit trail can confirm access, while a runtime baseline can surface abnormal decision-making before the wrong output becomes part of a patient-facing workflow. For healthcare organisations, that difference is material because the harm often comes from the agent’s action path, not just from the record it touched.

Runtime baselines also fit the way AI agents fail in practice: drifting prompts, unexpected tool calls, overbroad context, or silent changes in output quality. A well-tuned baseline gives you a stable reference for what “normal” looks like across the agent lifecycle, which is more useful for prevention than reconstructing intent after the event.

What EHR audit logs are good for, and what they miss

EHR audit logs remain essential for accountability, investigation, and compliance. They help answer who accessed which record, when a write occurred, and which user or system path was involved. That is valuable for incident response, legal review, and post-incident reconstruction.

What they do not reliably answer is whether the agent’s recommendation, summary, or generated note was clinically sound at the moment it was produced. An audit log can show the transaction, but it usually does not reveal whether the model was hallucinating, whether the tool output was stale, or whether the agent crossed an unsafe boundary while still appearing “authorized.”

That is why audit logs are better treated as a forensic layer than a safety layer. They are strongest after an event, while runtime monitoring is strongest before the unsafe output propagates into orders, notes, handoffs, or clinician decisions.

How to split the control stack for AI-mediated harm

The right design is to treat runtime baselines and EHR audit logs as complementary controls with different jobs. Runtime baselines should watch for abnormal agent behaviour, such as unexpected escalation, unusual tool sequences, or output patterns that diverge from the approved clinical use case. EHR audit logs should preserve a trace of access and changes for attribution and review.

Healthcare teams get into trouble when they use record logs as a proxy for model safety. A logged write to the EHR does not prove that the preceding AI step was trustworthy, and a clean log does not guarantee that the agent did not introduce downstream clinical risk through summary distortion, instruction misuse, or unsafe automation.

This is where runtime monitoring pairs well with established AI agent observability and incident response practices, because the goal is to detect behavioural drift early and retain enough evidence to explain it later. For organisations formalising agent boundaries, the AI Agent Authorisation Guide is the relevant model for keeping access scoped to the task.

Risk and Threat Considerations

When AI agents influence clinical workflows, the main risk is not just unauthorized access to the EHR, it is unsafe action that still looks legitimate in logs. A system can be fully auditable and still produce harmful recommendations, wrong chart updates, or inappropriate downstream actions if runtime behaviour is not constrained.

Failure mechanism: The agent operates within its apparent permissions, but its prompts, tool calls, context, or output quality drift away from the approved clinical baseline, so the unsafe behaviour is only visible after it has affected the workflow.

Impact: Clinicians may inherit incorrect information, patient records may contain misleading content, and incident response may be delayed because the audit trail shows an authorized transaction rather than a clearly anomalous clinical failure.

Practitioner Guidance

What to prioritise: Use runtime baselines for the control decision and audit logs for the accountability decision. If you must choose where to invest first, start with the signals that can stop or quarantine bad agent behaviour before it reaches the EHR.

What to verify: Confirm that the baseline covers the behaviours that matter clinically, not just system uptime, especially tool invocation patterns, escalation paths, output shape, and when human review is required. A baseline that cannot distinguish expected summarisation from unsafe synthesis is too weak to protect care delivery.

Practitioner takeaway: Audit logs answer who did what, but runtime baselines answer whether the agent was safe enough to do it in the first place.

Related resources from NHI Mgmt Group

• When should organisations prioritise audit trails over more secret rotation for AI agents?
• When should organisations prioritise runtime attribution over posture scanning for AI agents?
• When should organisations prioritise runtime controls over policy documentation for AI agents?
• When should organisations prioritise governance over more AI pilots in healthcare?