Sustainable AI governance needs broader regulation and trust

At a glance

What this is: This is a Vinnova-funded analysis of sustainable AI governance that finds legal systems must adapt to black-box AI by broadening regulation, collaboration, and trust controls.

Why it matters: It matters to identity and security practitioners because AI governance increasingly intersects with accountability, access decisions, and responsibility for automated systems that affect people and data.

Context

Sustainable AI is fundamentally a governance problem, not only a model-design problem. When AI systems cannot be explained clearly, existing legal and organisational controls struggle to assign responsibility, evidence compliance, and justify decisions that affect individuals or operations. For IAM, PAM, and broader security teams, the key issue is how accountability is preserved when decision-making becomes partially opaque.

The report is framed as an inventory of knowledge rather than a prescriptive control manual, so its value lies in the policy and governance direction it sets. That makes it relevant to identity teams where AI is used in decision support, trust workflows, or automated access-related processes. The article's starting point is typical of current AI governance debates, but the implications are increasingly operational.

The primary keyword here is sustainable AI, and the report treats it as a cross-disciplinary governance challenge spanning law, ethics, and implementation. That places it squarely in the space where AI assurance, risk ownership, and human accountability meet.

Key questions

Q: How should organisations govern AI systems that can make consequential decisions?

A: Organisations should govern consequential AI systems with the same discipline used for high-risk identities: defined ownership, least privilege, logging, approval boundaries, and human override. The critical requirement is to connect model behaviour to real access paths so legal review, security review, and audit evidence all describe the same system.

Q: Why do black-box AI models become a compliance problem in regulated sectors?

A: They become a compliance problem when the organisation cannot justify outputs to auditors, regulators, or affected users. In regulated sectors, a model that cannot explain its reasoning can still be operationally useful, but it is difficult to defend when decisions are challenged. That gap often becomes visible only after deployment, when remediation is more costly.

Q: What do security teams get wrong about AI visibility?

A: They often assume licence data or static configuration data is enough to understand AI risk. In practice, the important question is what identities actually do at runtime, which services they reach, and what data they share. If you cannot observe that behaviour, you cannot govern it reliably.

Q: How can teams reduce AI governance risk before deployment expands?

A: Set baseline guardrails early for data handling, acceptable use, and escalation. Then require every new AI use case to pass through a repeatable intake process that captures purpose, dependencies, and control expectations. That approach makes scaling safer because teams are not inventing rules case by case.

Technical breakdown

Why black-box AI creates a governance gap

A black-box AI system is one whose internal reasoning cannot be readily inspected or explained in a way that supports oversight. That matters because legal and security controls depend on traceability, reproducibility, and the ability to justify outcomes. When a model is used in socially applied settings, the problem is no longer only whether it is accurate, but whether decision-makers can evidence why a result occurred and who is accountable for it. This shifts the control problem from pure model performance to governance around decision rights, documentation, and review.

Practical implication: build AI governance requirements around explainability evidence, decision ownership, and reviewability, not just output quality.

Transparency is not the same as accountability

Transparency tells you more about how a system works; accountability tells you who is responsible when it affects people. The report's central tension is that organisations often assume greater visibility automatically creates responsibility, but those are different control problems. In practice, a system can be documented and still leave unclear who approves use, accepts residual risk, or handles harm. For identity and access programmes, that distinction matters whenever AI supports trust decisions, entitlement decisions, or policy enforcement.

Practical implication: assign explicit human accountability for AI-supported decisions and make it part of governance, not a side effect of documentation.

Multidisciplinary governance is a control requirement, not a nice-to-have

The report argues for collaboration across law, technical disciplines, and domain experts because no single team can fully govern socially applied AI. That is a practical acknowledgement that AI risk spans compliance, model behaviour, data quality, and operational use. In security terms, the control boundary crosses traditional silos: identity teams, legal, data governance, and product owners all have parts of the decision chain. Without shared governance, organisations create gaps between policy intent and real-world deployment.

Practical implication: create cross-functional AI governance forums with clear decision rights for risk acceptance, escalation, and exception handling.

NHI Mgmt Group analysis

Sustainable AI is an accountability problem before it is a model problem. The report correctly centres the governance gap created by black-box systems, where outputs can influence decisions without clear line-of-sight into the basis for those decisions. For identity-led programmes, that is especially relevant when AI touches access, trust, or policy enforcement. Practitioners should treat accountability mapping as a control, not an afterthought.

Transparency without responsibility creates false assurance. Organisations often equate better documentation with better control, but the report shows why that is incomplete. A system may be observable and still remain poorly governed if nobody owns the decision outcome, exception process, or escalation path. The practical lesson is that governance has to bind explanation to named accountability.

Multidisciplinarity is the operating model AI governance now requires. The report's emphasis on collaboration reflects a real structural issue: AI risk crosses legal, technical, ethical, and operational boundaries. That is also where identity programmes often become the enforcement layer for policy, access, and trust decisions. Practitioners should design governance so that no single function is expected to carry AI assurance alone.

Sustainable AI should be treated as an emerging governance discipline, not a one-off compliance exercise. The article's value is not in a tool or product claim, but in highlighting that AI oversight needs durable processes, shared vocabulary, and repeatable review. That aligns with broader NIST AI RMF thinking: govern, map, measure, and manage as a continuing cycle. For practitioners, the conclusion is straightforward. If governance is episodic, AI risk will outpace it.

From our research library:

• Only 5.7% of organisations have full visibility into their service accounts, according to the Ultimate Guide to NHIs.
• Read next: Agentic AI Compliance Guide

What this signals

Sustainable AI governance will increasingly resemble identity governance. Once AI systems influence access, trust, or operational decisions, the question is no longer only whether the model works but who is accountable for the decision path it creates. That pushes practitioners toward governance structures that resemble lifecycle control, approval, and exception management rather than one-time model review.

AI governance debt will grow where organisations separate transparency from responsibility. The deeper issue is not model opacity alone, but the absence of durable ownership across legal, security, and business teams. If accountability is not designed into the operating model, organisations will accumulate risk faster than they can explain it.

Only 5.7% of organisations have full visibility into their service accounts, according to the Ultimate Guide to NHIs. That lack of visibility is a useful analogue for AI governance, where unknown or weakly owned system behaviour creates the same accountability problem in a different form.

For practitioners

• Define accountable AI decision ownership Document which business role, technical owner, and governance function accepts responsibility for AI-supported decisions, exceptions, and escalation paths.
• Separate transparency from control evidence Require audit-ready evidence for why a model was used, what data influenced the outcome, and who approved the decision, rather than relying on model visibility alone.
• Build cross-functional AI governance workflows Bring legal, security, data, and domain owners into a single review process for socially applied AI systems so accountability does not disappear across handoffs.
• Apply recurring risk review to deployed AI Set a periodic reassessment cycle for AI use cases, with explicit review of policy changes, model drift, and operational exceptions.

Key takeaways

• Sustainable AI governance is really about making opaque decision systems accountable through ownership, review, and escalation.
• The report's core warning is that transparency alone does not close the gap between model insight and human responsibility.
• Practitioners should treat AI governance as a continuing operating model, not a one-time compliance check.

Key terms

• Black Box AI: Black Box AI is an AI system whose internal reasoning is not transparent to users or operators. Its inputs and outputs can be observed, but the decision path, feature weighting, or intermediate logic cannot be easily inspected, which complicates validation, accountability, security review, and incident investigation.
• AI Accountability Boundary: The point at which responsibility for an AI system's outputs shifts from abstract policy to named operational owners. It defines who must evidence controls, review incidents, and justify decisions across the model, application, and infrastructure layers when harmful behaviour occurs.
• Transparency: Transparency is the ability to inspect how an AI system uses data, makes outputs, and influences decisions. In practice, it provides the evidence needed for audit, approval, and challenge. Without transparency, governance relies on trust rather than verifiable control.
• AI Governance: AI governance is the set of controls used to discover, classify, approve, restrict, monitor, and revoke AI-enabled access. It connects identity, data, and policy so organisations can manage what AI can reach, what it can share, and when it should be stopped.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, helps practitioners build strong foundations in NHI governance, human identity, and secrets management. It is a useful next step for security teams aligning identity controls with broader AI and governance risk.