High-risk AI systems under the EU AI Act demand stronger governance

At a glance

What this is: This is a practitioner guide to classifying AI systems under the EU AI Act and identifying when high-risk obligations, transparency duties, and GPAI requirements apply.

Why it matters: It matters because AI teams need a defensible way to classify systems early, map the correct obligations, and avoid missing compliance duties that arise from the system's function, context, or embedded model.

👉 Read Holistic AI's guide to identifying high-risk AI systems under the EU AI Act

Context

The core governance problem is not whether an AI system uses advanced models, but whether its function places it inside a regulated risk category. Under the EU AI Act, classification depends on use case, sectoral product rules, and whether the system can affect fundamental rights, health, or safety. That makes inventory and role clarity essential for AI governance, compliance, and security teams.

For identity and access practitioners, the important intersection is that AI systems can trigger obligations around human oversight, record-keeping, transparency, and cybersecurity, while integrated general-purpose AI models add another layer of control and accountability. This is a compliance classification problem first, but it also has real consequences for how AI systems are governed across identity, data, and operational security.

Key questions

Q: How should organisations classify AI systems for EU AI Act compliance?

A: Start with intended use, not technical complexity. If the system is used for hiring, credit, performance evaluation, or another Annex III activity, it may be high risk even if it looks routine. Classification should be tied to the actual business function, the people affected, and whether the organisation is acting as provider or deployer.

Q: When do AI systems move into high-risk territory under the EU AI Act?

A: AI systems move into high-risk territory when their use case can materially affect health, safety, or fundamental rights, or when they are safety components covered by regulated product rules. In practice, this often affects hiring, education, essential services, biometric use, critical infrastructure, and judicial support systems. The decision depends on what the system does, not on the vendor's marketing label.

Q: What do security teams get wrong about AI compliance?

A: They often treat AI compliance as a model review exercise and miss the surrounding identity and access layer. In practice, regulators care about data handling, delegated permissions, logging, and accountability. If service accounts, tokens, and approvals are not governed, the control story is incomplete even when the model documentation looks strong.

Q: How should teams govern GPAI inside regulated AI systems?

A: Treat the model and the application as one governance chain. If a general-purpose AI model is integrated into a high-risk system, the provider must manage both sets of obligations, including documentation, oversight, cybersecurity, and incident handling. Teams should map responsibilities across model owners, deployers, and security functions before go-live.

Technical breakdown

How high-risk classification works under the EU AI Act

An AI system becomes high-risk through two main routes. First, it can be part of a product covered by Union harmonised legislation and require third-party conformity assessment. Second, it can appear in Annex III as a use case such as biometric systems, employment, education, critical infrastructure, or access to essential services. The classification question is therefore functional and contextual, not purely technical. A system may sit outside high-risk status if it performs only a narrow procedural task or supports a human decision without replacing it, but that exception depends on documented assessment and the actual deployment context.

Practical implication: Build a classification workflow that records use case, sector, and exception rationale before deployment.

Why transparency and documentation are part of AI governance

The EU AI Act treats high-risk AI as a governed system, not just a model. Providers must maintain technical documentation, keep records, disclose AI use where required, and support human oversight. These requirements create evidence trails that let regulators, deployers, and internal reviewers understand what the system does, what data it uses, and how decisions are controlled. For systems that generate synthetic content or interact directly with users, transparency can extend to labels, notices, and machine-readable disclosure. Governance failures here are often traceability failures first.

Practical implication: Treat documentation, logging, and disclosure as operational controls, not paperwork.

How GPAI inside a high-risk system changes the control model

When a general-purpose AI model is integrated into a high-risk AI system, the combined system inherits obligations from both regimes. That means the provider must think about model documentation, copyright policy, training data summaries, and systemic-risk duties alongside the high-risk system's own oversight, accuracy, robustness, and cybersecurity requirements. In practice, the model boundary becomes a governance boundary. Teams can no longer assess the interface layer alone if the embedded model influences output, risk, or human decision-making.

Practical implication: Map model, application, and deployment obligations together rather than treating them as separate compliance tracks.

NHI Mgmt Group analysis

High-risk AI classification is now a governance inventory problem, not a policy memo. The article shows that organisations must identify where AI systems sit in Annex III, where product legislation applies, and where exceptions might remove them from the high-risk category. That means classification is an evidence exercise, not a one-time legal interpretation. AI governance teams should expect regulators to ask for the rationale, not just the label.

The EU AI Act makes documentation a control surface. Technical documentation, record-keeping, and deployer information are not secondary tasks. They are the proof that oversight, accuracy, and cybersecurity obligations were designed into the system rather than added after deployment. Practitioners should treat documentation as a live control that supports auditability and incident response.

AI systems increasingly carry identity-like governance requirements. When a system interacts with users, generates content, or includes a GPAI model, teams need to know who is accountable, what role each party plays, and what access or oversight boundaries exist. That intersection matters for IAM and security leaders because AI governance now depends on clear ownership, traceability, and controlled delegation.

Regulated AI will force convergence between compliance and security operations. The obligations in the EU AI Act connect risk management, human oversight, robustness, and cybersecurity in one control plane. Organisations that keep legal review, model risk, and security operations separate will struggle to evidence compliance. The practical conclusion is that AI governance needs an operating model, not just policy language.

AI governance debt will accumulate where teams cannot explain why a system was excluded from high-risk status. The article's exception logic matters as much as the main classification rules. If organisations cannot document why a system was considered narrow, preparatory, or human-reviewed, they will inherit regulatory exposure later. Practitioners should preserve classification evidence at the point of design, not after challenge.

What this signals

AI governance programmes will increasingly fail at the handoff between legal classification and operational control. Classification drift: once a system changes use case, model, or user interaction pattern, the original high-risk judgment may no longer hold. Teams should build review triggers into change management so AI inventory, risk classification, and oversight stay aligned with deployment reality.

For identity and access teams, the key signal is ownership clarity. When AI systems rely on human review, content generation, or integrated GPAI models, responsibility must be explicit across system owners, deployers, and security operators. That makes AI governance closer to access governance than many organisations expect, especially where decision support shapes real-world outcomes.

For practitioners

• Create a classified AI system inventory Record every AI system, its business function, deployment context, and whether it may fall under Annex III or harmonised product legislation. Keep the evidence for any exception you rely on so the classification can survive challenge.
• Document high-risk exception decisions For systems you believe are not high-risk, capture why the system performs only a narrow procedural task, supports rather than replaces human review, or fits another exception. Store the rationale with the system record and review it when the use case changes.
• Align documentation, oversight, and security controls Make technical documentation, record-keeping, human oversight, and cybersecurity requirements part of the same governance workflow. This avoids separate approvals that drift apart from the actual deployed system.
• Map GPAI dependencies before deployment Identify whether any integrated general-purpose AI model introduces additional obligations for training data summaries, model documentation, or systemic-risk handling. Track the model boundary so responsibilities do not disappear inside the application stack.

Key takeaways

• The EU AI Act turns AI classification into an evidence-backed governance task, not a one-time legal label.
• Documentation, oversight, and cybersecurity are the controls that make high-risk AI defensible in practice.
• Integrated GPAI models expand the compliance boundary, so teams must govern the model and the application together.

Key terms

• High-Risk AI System: A high-risk AI system is one whose outputs can materially affect a person’s rights, opportunities, or safety. These systems need stronger oversight because errors, bias, or unauthorized actions can create legal exposure as well as security and trust problems.
• Conformity assessment: A conformity assessment is the formal process used to show that a high-risk AI system meets the obligations required before it is placed on the market. It combines documentation review, technical verification, and evidence of operational controls, rather than relying on policy statements alone.
• General Purpose AI System: A general purpose AI system is a model or service that can be used across multiple tasks rather than for one narrow function. Under regulation, that broad usability increases governance pressure because the same system can be repurposed into different business, privacy, and security contexts.
• Human Oversight: Human oversight is the requirement that a person remains responsible for reviewing, approving, or correcting AI-driven output before it causes a material action. In governance terms, it is the control that prevents automation from becoming unowned authority.

What's in the full article

Holistic AI's full blog covers the operational detail this post intentionally leaves for the source:

• Step-by-step AI Act classification examples across Annex III use cases and harmonised product categories
• Detailed breakdown of when the narrow-task and human-review exceptions may apply in practice
• Obligation mapping for providers, deployers, importers, distributors, and authorised representatives
• GPAI-specific documentation and systemic-risk duties for integrated models

👉 The full Holistic AI post covers classification steps, exception checks, and AI Act obligation mapping.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, human identity, and secrets management in a way that helps security teams connect access control to broader identity risk. It is suitable for practitioners building governance across identity, AI, and adjacent security programmes.