What do organisations get wrong when assessing whether an AI medical device is high-risk?

Why This Matters for Security Teams

AI medical devices sit at the intersection of product regulation, patient safety, and software assurance, so a misread classification can distort both compliance effort and risk acceptance. The key mistake is treating “high-risk” as a generic AI label rather than asking whether the product falls into a regulated medical-device use case and how that use case is already governed under the MDR. If teams skip that step, they can miss when a conformity assessment is already expected, or fail to justify why a system is below the threshold for significant risk. The EU Cyber Resilience Act is relevant here because product-security obligations increasingly sit alongside sector-specific safety rules, not instead of them.

That distinction matters operationally. A model can be technically “AI-enabled” without being the thing that drives the regulatory decision, and a medical workflow can still be high-consequence even when the model itself looks narrow. Teams that collapse those layers often over-engineer the wrong controls, or under-document the basis for their decision. In practice, many organisations only discover the classification gap when a regulator, notified body, or procurement review asks for the written rationale they never completed.

How It Works in Practice

The assessment should start with the intended medical purpose, the clinical context, and the product boundary. From there, organisations need to determine whether the device is already captured by medical-device regulation, whether the AI function changes the risk profile, and whether the system’s outputs influence diagnosis, triage, treatment, or another safety-critical decision. In other words, the question is not “does the system use AI,” but “what does the system do in the real world, and what harm can result if it is wrong, unavailable, biased, or poorly controlled?”

Teams usually get better results when they separate the review into three layers:

• the regulated product and its intended use;
• the AI component and its failure modes;
• the evidence trail showing why the chosen risk classification is defensible.

That evidence trail should be written, current, and easy to retrieve. It should show the clinical assumptions, the basis for any claim that the system does not present significant risk, and the reason any third-party assessment was or was not required. Where the device relies on external data, model updates, or changing usage patterns, the classification should be revisited whenever the clinical function, user population, or decision impact changes. OWASP API Security Top 10 is useful as a supporting reference when the device depends on service integrations or remote calls that can alter the security posture of the product.

These controls tend to break down when a team assumes the vendor’s AI description is enough to define the regulatory position, because the decisive issue is the product function, not the marketing label.

Common Variations and Edge Cases

Tighter classification usually increases documentation and review overhead, so organisations have to balance speed against the cost of getting the regulatory basis wrong. One common edge case is a device that is not novel in its AI technique but is high impact in its clinical setting, which can make the risk profile much more serious than the model architecture suggests. Another is software that assists clinicians rather than replacing them, because decision support can still be material if it shapes the final clinical outcome.

Borderline cases often involve systems that sit between general-purpose software and a regulated medical device, or tools that are deployed in one workflow but later extended into a more safety-critical one. Current guidance suggests treating those transitions carefully, because the classification can change when the intended use changes, even if the code does not. Organisations should also be cautious about relying on a single control such as human review to down-classify the system; if the output is influential, that review may not reduce the risk enough to change the regulatory view. Where the device is part of a broader connected environment, product-security requirements and clinical-risk requirements should be assessed together rather than in separate silos. The NIST AI Risk Management Framework can help structure the broader governance conversation around reliability, transparency, and impact, even though the legal classification still turns on the medical use case.

In practice, the hardest calls are usually made where the clinical impact is obvious but the organisation has not documented how that impact translates into a formal risk assessment.

Risk and Threat Considerations

The main risk is misclassification, which can leave a medical AI product under-assessed, under-documented, or governed with the wrong level of scrutiny. That creates exposure in both directions: over-classification can waste time and resources, while under-classification can leave a safety-critical system without the review, traceability, and evidence needed to defend its deployment.

Failure mechanism: The error usually happens when teams use model type, vendor claims, or generic AI language as the deciding factor instead of the intended medical purpose and the actual clinical decision path. Once that happens, the organisation may miss a required conformity assessment, omit a written rationale for a non-high-risk decision, or fail to notice that a later workflow change has made the product materially more consequential.

Impact: The result can be regulatory challenge, delayed approval, weak auditability, and avoidable patient-safety exposure if the system’s limitations are not evaluated at the right level. In a live setting, the bigger consequence is often not a single bad classification, but a governance trail that cannot support the decision when it is later questioned.

Practitioner Guidance

What to prioritise: Anchor the assessment in intended medical purpose, clinical use, and impact on decisions, then map the AI function to that context. If the product’s output can influence diagnosis, triage, or treatment, treat the classification as a clinical-risk question first and an AI question second.

What to verify: Keep a dated written record that shows why the system is or is not high-risk, what evidence was reviewed, and whether a third-party conformity assessment was already required by the product’s medical-device status. Re-check that record whenever the workflow, user group, or model behaviour changes.

Decision rule: If the organisation cannot explain the classification in terms of intended use, risk to the patient, and regulatory route, the assessment is too weak to rely on. In that case, pause rollout until the rationale is made explicit and reviewable.

Practitioner takeaway: The safe position is not to label every AI medical device as high-risk, but to ensure that every borderline case has a defensible, written, and clinically grounded reason for the classification chosen.

Related resources from NHI Mgmt Group

• What do organisations get wrong about AI-driven cyber risk?
• What do organisations get wrong about AI identity risk?
• What do organisations get wrong when they separate AI risk from identity risk?
• What do organisations get wrong about AI prompt injection risk?