AI governance fails when controls cannot enforce human oversight

At a glance

What this is: This is Saidot’s analysis of AI governance and control, using the Tromsø municipality incident and the EU AI Act to show that visibility alone does not stop harmful AI use.

Why it matters: It matters because IAM, NHI, and AI governance teams need enforceable oversight, not just inventories or policy statements, when AI outputs can influence regulated, auditable business decisions.

By the numbers:

• 673 pages of chat logs from Tromsø municipality became public in 2025 and exposed unverified AI-generated report content.
• The EU AI Act entered into force on 1 August, 2024, with governance rules for general-purpose AI models becoming applicable on 2 August, 2025.
• The most significant milestone for full applicability is scheduled for August 2026, when a significant part of high-risk system obligations will be enforced.

Context

AI governance is the set of decisions, controls, and evidence that determines whether AI is used within defined boundaries. The problem in this article is not AI itself, but the common gap between policy intent and enforceable oversight, especially when AI outputs can enter regulated documents or operational workflows.

The article links that gap to both human review and system-level control. In identity and security programmes, the same pattern appears whenever access, approval, or delegation is documented but not technically enforced. That is why the article’s governance model is relevant to human identity, NHI, and agentic AI programmes alike.

Key questions

Q: What breaks when AI governance exists on paper but not in enforcement?

A: Policy-only governance fails when teams cannot technically restrict access, log decisions, or revoke permissions in the systems AI actually uses. That creates a false sense of control while data exposure, privilege creep, and untracked actions continue. Effective governance must be enforced through identity, access, and audit mechanisms, not only through policy documents.

Q: Why do organisations need trigger-based governance for AI systems?

A: AI systems change after retraining, model updates, tool additions, and data changes, so a one-time approval quickly becomes stale. Trigger-based governance forces reassessment when the system changes, which is how teams avoid hidden risk accumulation and missed compliance obligations. Without it, the organisation governs yesterday’s version of today’s system.

Q: How do security teams know if AI governance is working?

A: Look for evidence that access decisions are reviewable, permissions are revocable, and exceptions are not becoming permanent. If the team cannot explain who owns an AI workflow, what it can reach, and when its access was last reviewed, governance is incomplete. Control maturity shows up in traceability, not adoption volume.

Q: Who should be accountable for AI-generated content that enters official records?

A: Accountability should sit with the business owner of the use case, the control owner for review and logging, and the governance function that sets risk thresholds. If AI-generated content is permitted into official records, someone must be responsible for the review step, the approval criteria, and the evidence trail. Shared ownership without clear control responsibility usually means nobody owns the risk.

Technical breakdown

Why AI inventorying is the first governance control

AI inventorying is the baseline control that makes AI systems visible enough to classify, assess, and govern. In practice, that means identifying models, agents, datasets, tools, and integrations across the enterprise, then mapping them to risk tiers and required oversight. Without that inventory, organisations cannot tell whether a system is prohibited, high risk, or low risk, and they cannot prove that controls were applied to the right assets. The mechanism matters because AI risk often comes from composition, not just the model itself. A retrained model, a new plug-in, or a third-party API can change the risk profile without any obvious business change.

Practical implication: Build a governed AI inventory before expanding use cases, or risk applying controls to the wrong systems.

How trigger-based governance closes AI lifecycle gaps

Trigger-based governance re-evaluates AI risk when something material changes, such as retraining, new data, new tools, or third-party updates. This is necessary because AI systems are not static assets. They evolve through model refreshes, context changes, and dependency updates that can alter behaviour and compliance posture after approval. The operational lesson is that governance cannot be a one-time gate at launch. It has to re-open when the system changes, or hidden risk accumulates across the AI supply chain. This is especially relevant where AI outputs feed regulated processes, official documents, or decision-support workflows.

Practical implication: Tie risk reassessment and control revalidation to change events, not just initial approval.

Why human-in-the-loop control is more than review

Human-in-the-loop control is an enforcement mechanism, not a courtesy review. The article shows why human oversight must be technically embedded so AI outputs cannot progress without the right approval step. That distinction matters because policy-only oversight assumes people will pause, recognise risk, and act consistently under pressure. Technical control removes that assumption by routing outputs to qualified reviewers, logging decisions, and preventing bypass. For AI governance, this is the difference between documented oversight and actual oversight. In security terms, it is the move from advisory control to enforced control.

Practical implication: Require technical approval checkpoints for AI outputs that can influence regulated or external-facing decisions.

Threat narrative

Attacker objective: The objective was not a criminal intrusion but the production of authoritative-looking content that bypassed review and later exposed the organisation to scrutiny.

1. Entry occurred when AI was used to draft official content without a control gate that required human review before publication or inclusion.
2. The governing failure was unverified AI-generated output, including fabricated sources, being accepted as authoritative content.
3. Impact followed when the municipality’s chat logs and report content became public, creating accountability, reputational, and operational exposure.

NHI Mgmt Group analysis

AI governance debt is now a board-level risk: organisations that treat inventory as governance but fail to enforce controls are accumulating unresolved risk inside everyday workflows. The Tromsø case shows that the real failure is not model use, but the absence of technical guardrails that make oversight executable. Boards should treat AI control maturity as evidence of operational accountability, not as a policy exercise.

Visibility without enforcement creates a false sense of control: inventories, classifications, and policies are necessary, but they do not stop unverified AI content from entering regulated processes. The governance model fails when human review remains optional or depends on user discipline. Practitioners should regard this as a control-design problem, not a training problem.

Governance and control are complementary layers: one discovers and classifies AI, the other blocks or routes AI actions before harm occurs. That combination is what turns AI oversight into a defensible operating model under the EU AI Act and similar accountability regimes. The practitioner conclusion is simple: if you cannot enforce the rule, you do not yet have the control.

AI interactions are becoming auditable records, not invisible traces: once chat logs and generated outputs can be disclosed, the organisation must assume its AI behaviour is discoverable. That shifts AI governance from informal guidance to evidence-backed supervision, and it raises the bar for human oversight in both public-sector and enterprise settings. Practitioners need audit trails that prove review happened, not just that it was expected.

From our research library:

• Only 37% of organisations had policies for managing AI or detecting shadow AI, according to IBM's 2025 Cost of a Data Breach Report.
• Read next: AI Governance Guide

What this signals

AI governance debt: when organisations accumulate policies, inventories, and disclaimers without technical enforcement, they create an assurance gap that grows with every new model, agent, and integration. The practical implication is that governance teams must shift from documentation-first programmes to control-first operating models that can withstand audit and disclosure.

The article’s deeper signal is that AI oversight is becoming part of ordinary security governance, not a niche compliance activity. Because AI systems now influence reports, workflows, and decisions, teams need review gates, logging, and change-triggered reassessment in the same way they manage other high-risk enterprise controls.

For practitioners

• Implement a governed AI inventory Track models, agents, tools, datasets, and third-party dependencies in one inventory so every AI use case has an owner, a risk rating, and a control status.
• Enforce human review checkpoints Block AI-generated outputs from progressing into reports, customer communications, or regulated workflows until a qualified reviewer explicitly approves them.
• Tie governance to change events Reassess AI systems when models are retrained, APIs change, plug-ins are added, or new data sources are introduced, because each change can alter risk.
• Record auditable AI decision trails Log prompts, outputs, approvals, and overrides so you can demonstrate actual oversight during audits, investigations, or disclosure requests.
• Separate acceptable, high-risk, and prohibited uses Define which AI use cases are allowed, which require approval, and which are off-limits, then make those boundaries visible to business owners and reviewers.

Key takeaways

• The central risk is not that organisations use AI, but that they allow AI output to enter records and workflows without enforceable human oversight.
• The Tromsø case and the EU AI Act together show that AI governance is moving from policy aspiration to auditable accountability.
• Practitioners should treat inventory, review checkpoints, and change-triggered reassessment as core controls, not optional governance extras.

Key terms

• AI Governance: AI governance is the set of controls used to discover, classify, approve, restrict, monitor, and revoke AI-enabled access. It connects identity, data, and policy so organisations can manage what AI can reach, what it can share, and when it should be stopped.
• Human-in-the-Loop Controls: Human-in-the-loop controls are decision gates that require a person to approve, review, or override an automated action before it proceeds. They are used when speed matters but the impact is high, such as account blocking, endpoint isolation, or access revocation, to keep automation accountable and bounded.
• Event-based Governance: A governance model that ties access decisions to identity, security, or operational events instead of fixed dates. It is stronger than periodic-only review because it validates permissions at the moment they change, not after the change has already aged.
• AI Inventory: An AI inventory is a governed record of all AI-related assets, enriched with owner, purpose, access, and risk context. It turns discovery into something security, compliance, and IAM teams can use to make approval, review, and revocation decisions.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, helps practitioners understand NHI governance, agentic AI identity, and machine identity security. It is relevant when your programme needs clearer control ownership across identity-driven risk.