High-Risk AI System

Expanded Definition

High-risk AI systems are those whose outputs can affect eligibility, safety, access, or rights in ways that create material operational, legal, or security consequences. In practice, the label is often used in policy, procurement, and governance rather than as a single technical class, and definitions vary across vendors and regulations. The European Union AI Act is the clearest example of a formal risk-tiering model, while the NIST Cybersecurity Framework 2.0 frames the issue through governance, protection, detection, and response outcomes instead of using the same legal category. For NHI security teams, the term matters because an AI system can be high-risk even when the model itself is not malicious; the risk often comes from connected agents, privileged tools, exposed secrets, or poor human oversight. That is why the OWASP NHI Top 10 is useful for mapping technical abuse paths around identity and access. The most common misapplication is treating any generative model as high-risk by default, which occurs when teams ignore actual decision impact, user exposure, and the privileges granted to the surrounding agent workflow.

Examples and Use Cases

Implementing high-risk AI controls rigorously often introduces friction in release cycles and user experience, requiring organisations to weigh trust and safety against speed and automation.

• A recruiting agent that ranks applicants can become high-risk if it influences hiring outcomes, especially when it calls external tools or uses service credentials to fetch internal candidate data.
• An AI-assisted credit decision workflow is high-risk because inaccurate outputs can change access to loans, rates, or approvals, making auditability and appeal paths essential.
• A clinical triage assistant is high-risk when its recommendations affect care priority, which is why identity controls, logging, and human review must align with governance expectations.
• A public-sector benefits bot may be high-risk if it can alter eligibility decisions or trigger downstream case actions, especially when tied to shared secrets or overbroad permissions.
• Security research on the DeepSeek breach shows how exposed records and embedded secrets can expand an AI incident beyond model quality into a broader identity and data exposure event.

In these scenarios, practitioners often pair policy controls with access discipline and reviewable workflows. The same pattern is reflected in Top 10 NHI Issues, where over-privileged identities and weak secret handling turn an otherwise useful agent into an enterprise liability.

Why It Matters in NHI Security

High-risk AI systems become an NHI problem when agents, pipelines, and service accounts are allowed to act with authority that exceeds the business need. Once that happens, a model error is no longer just a quality defect; it can become an access-control failure, a compliance event, or an incident involving credentials and data. That is especially relevant because the 2024 ESG Report: Managing Non-Human Identities found that 72% of organisations have experienced or suspect they have experienced a breach of non-human identities, with two-thirds enduring a successful cyberattack resulting from compromised NHIs. For high-risk systems, that means governance must extend beyond model evaluation into secret hygiene, privilege boundaries, and continuous monitoring. In the NHI context, the right questions are not only whether the output is accurate, but also whether the agent had MCP access, whether the NHI was assigned only JIT access, and whether the workflow fits Zero Trust Architecture principles. The operational lens aligns with the Ultimate Guide to NHIs — Why NHI Security Matters Now and the Ultimate Guide to NHIs — Key Challenges and Risks. Organisations typically encounter the full impact only after an agent has made an unauthorized decision or leaked secrets, at which point high-risk AI system controls become operationally unavoidable to address.

Related resources from NHI Mgmt Group

• How can organizations prioritize high-risk AI agents?
• When should organisations treat an AI agent as a high-risk identity?
• AI Agent Authentication
• Shadow AI