High-risk AI systems under the EU AI Act demand stronger governance

At a glance

What this is: Holistic AI explains how the EU AI Act classifies AI systems as high-risk and which obligations then apply, with emphasis on Annex III, harmonised product legislation, transparency duties, and GPAI overlays.

Why it matters: For AI governance teams, the practical issue is inventory and classification discipline, because once a system is in scope the organisation must be ready for documentation, oversight, transparency, and cybersecurity evidence.

Context

The core governance problem is classification. Under the EU AI Act, organisations must first decide whether an AI system falls into a high-risk category before they can work out which obligations apply, and that decision depends on the system's purpose, deployment context, and legal setting. For AI governance teams, the risk is misclassification at intake, not only weak controls later in the lifecycle.

High-risk status can arise from Annex III use cases, from being a safety component or product covered by harmonised sector legislation, or from being a general-purpose AI model integrated into a high-risk system. That means AI inventories, use-case descriptions, and deployment relationships all become compliance artefacts, not just architecture records.

Key questions

Q: What breaks when an organisation continues using an AI system that falls within the EU AI Act prohibited practices?

A: The main failure is legal and governance failure, not just technical noncompliance. The organisation risks using a system that is no longer lawful in the EU, which can invalidate deployment decisions, create immediate remediation pressure, and expose the business to penalties. In practice, this also weakens trust in AI governance because the system should have been stopped earlier.

Q: When should organisations treat a GPAI model as part of a high-risk system?

A: They should do so when the model is integrated into a broader AI system whose use case or deployment context places it in the high-risk category. At that point, model-only governance is insufficient because both the model obligations and the system obligations apply together.

Q: What are the signs that an AI inventory is too weak for EU AI Act readiness?

A: Common signs include missing use-case descriptions, unclear ownership, no record of whether Annex III applies, and no documented exception analysis for borderline systems. If those fields are absent, the organisation is not ready to prove its classification decisions or defend them during review.

Q: Which teams should own EU AI Act high-risk compliance decisions?

A: Ownership should be shared across AI governance, legal, security, privacy, product, and operational teams, with a named decision owner for each system. The reason is simple: classification, documentation, oversight, and cybersecurity are connected obligations, so no single function can evidence readiness alone.

Technical breakdown

How the EU AI Act classifies high-risk AI systems

The EU AI Act uses a risk-based model rather than a one-size-fits-all AI control set. A system may be high-risk because it is listed in Annex III or because it is a product, or safety component of a product, covered by Union harmonisation legislation that requires a third-party conformity assessment. In practice, classification is driven by function, deployment context, and whether the system affects health, safety, or fundamental rights. This is a governance test as much as a technical one, because the same model can be low risk in one use case and high risk in another.

Practical implication: Build a classification workflow that records the use case, legal basis, and deployment context before development or procurement proceeds.

Why Annex III use cases need human oversight and documentation

Annex III captures AI used in sensitive contexts such as employment, education, critical infrastructure, law enforcement, migration, public services, and judicial processes. The key governance point is that high-risk status is not just about the model, but about how the system can influence decisions affecting people. The article also notes exceptions where a system performs only a narrow procedural task or other limited function, but those exceptions must be assessed and documented before placement on the market or deployment. That makes the assessment itself part of the control set.

Practical implication: Document the exception analysis for every borderline use case so that the organisation can defend why a system was or was not treated as high-risk.

How GPAI inside high-risk systems changes the control boundary

When a general-purpose AI model is integrated with interfaces, feedback loops, or other system elements, the result is an AI system that can inherit high-risk obligations. That creates a two-layer compliance problem: the GPAI model has its own documentation, copyright, summary, and cooperation duties, while the combined system may also need the high-risk controls for risk management, logging, human oversight, and cybersecurity. The control boundary therefore shifts from model-only governance to system-level governance, which is where many programmes are currently weakest.

Practical implication: Treat integrated GPAI as part of the regulated system inventory and map both model obligations and system obligations before deployment.

NHI Mgmt Group analysis

Classification, not deployment, is the first governance failure mode: many AI programmes focus on model quality and user experience, but the EU AI Act starts by asking what the system does and where it is used. If the inventory is incomplete or the use case is under-described, the organisation can miss a high-risk classification entirely. That makes intake governance the decisive control point, and practitioners should treat system classification as a documented compliance gate.

High-risk AI governance is becoming a control stack, not a single policy: the article shows that risk management, data governance, technical documentation, record-keeping, transparency, human oversight, and cybersecurity are linked obligations rather than separate workstreams. The practical consequence is that AI teams need shared evidence across legal, security, privacy, and product functions. The practitioner conclusion is simple: compliance will fail where these disciplines operate in silos.

Integrated GPAI creates a dual-obligation problem that many programmes have not modelled: once a general-purpose model sits inside a high-risk system, organisations must account for both the model obligations and the system obligations. That means the governance model must track dependency, integration point, and accountability separately from the model itself. Practitioners should classify the combined system, not just the foundation model.

Regulatory evidence becomes operational evidence: the article makes clear that assessments, technical documentation, and transparency outputs are not paperwork after the fact. They are the artefacts authorities may ask for, and they need to reflect how the system was actually evaluated and controlled. The practitioner takeaway is to make evidence generation continuous, not retrospective.

Cybersecurity is now a compliance attribute of AI governance: the EU AI Act explicitly requires appropriate accuracy, robustness, and cybersecurity for high-risk systems, and systemic-risk GPAI has further cybersecurity expectations. That means AI security teams cannot sit outside the governance process. Practitioners should align AI control testing, incident handling, and documentation so cybersecurity evidence can support regulatory readiness.

What this signals

High-risk AI governance will be won or lost at inventory quality: practitioners need a complete view of use case, deployment context, and model integration before they can trust any classification decision. If the inventory does not capture those relationships, compliance will be reactive rather than defensible.

EU AI Act readiness is forcing security and governance convergence: the same programme now has to evidence risk management, record-keeping, transparency, human oversight, and cybersecurity for the same system. That means AI security controls should be designed as compliance evidence from the outset, not retrofitted after deployment.

For practitioners

• Map every AI use case to its legal classification Record whether the system is prohibited, falls under Annex III, is covered by harmonised product legislation, or uses a GPAI model inside a broader system. Keep the classification decision, rationale, and owner with the inventory entry.
• Document the high-risk exception test For borderline use cases, capture why the system performs only a narrow procedural task, supports rather than replaces human assessment, or otherwise falls outside high-risk status. Keep that assessment available for competent authority review.
• Separate model governance from system governance Track the GPAI model, the integration layer, the user interface, and the feedback mechanisms as distinct control objects so that high-risk obligations are not lost at integration time.
• Build evidence for transparency and oversight Retain technical documentation, record-keeping outputs, human oversight records, and cybersecurity evidence in a form that can be reused for compliance review and deployment approval.

Key takeaways

• The main governance issue is not just what the model does, but whether the system falls into a high-risk category that triggers specific obligations.
• The article links classification, documentation, oversight, transparency, and cybersecurity into one compliance workflow rather than separate tasks.
• AI teams should start with inventory accuracy and legal classification, because those decisions determine the rest of the control model.

Key terms

• High-Risk AI System: A high-risk AI system is one whose outputs can materially affect a person’s rights, opportunities, or safety. These systems need stronger oversight because errors, bias, or unauthorized actions can create legal exposure as well as security and trust problems.
• Annex A: Annex A is the control catalogue associated with ISO 27001. It provides the recommended control set organisations can use to support their ISMS, along with the requirement to justify whether each control is applied, excluded, or out of scope.
• General Purpose AI System: A general purpose AI system is a model or service that can be used across multiple tasks rather than for one narrow function. Under regulation, that broad usability increases governance pressure because the same system can be repurposed into different business, privacy, and security contexts.
• Conformity assessment: A conformity assessment is the formal process used to show that a high-risk AI system meets the obligations required before it is placed on the market. It combines documentation review, technical verification, and evidence of operational controls, rather than relying on policy statements alone.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and machine identity security. It is designed for practitioners who need to connect identity governance to broader security programmes and control evidence.