AI governance maturity lags behind enterprise scaling plans

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Cranium's blog post: Rapid enterprise AI adoption is widening the gap between deployment speed and the governance maturity needed to scale agentic systems safely.

Why it matters: It matters because IAM, IGA, PAM, and security teams now have to govern AI systems that act inside business workflows, not just approve static access.

By the numbers:

• Only 21% of companies report having a mature governance model for agentic AI.
• 63% of organizations now require human validation of AI agent outputs.
• AI agents have tripled in the last 18 months.
• Business leaders are projected to deploy $124 million on average toward AI in 2026.

Context

Enterprises are moving from AI experimentation to operational dependence, but the governance model has not caught up. The core issue is not model capability alone. It is whether an organisation can prove who approved the model, what data it can reach, which vendors are in the chain, and how its actions are monitored once it starts influencing business workflows.

For IAM and security leaders, this shifts AI from a tooling conversation into an identity governance problem. Agentic systems, model supply chains, and human validation checkpoints all need control boundaries, auditability, and revocation paths that work in real time. Without those boundaries, the organisation can scale AI usage faster than it can contain the resulting trust exposure.

Key questions

Q: What breaks when AI agents are allowed to contain incidents without governance?

A: The response chain becomes difficult to audit and reverse. If the agent can isolate hosts or trigger workflows without clear policy limits, teams may not know why an action happened, who approved it, or how to undo it safely. That creates operational speed with weak accountability.

Q: Why do organisations need runtime AI controls instead of relying on human review?

A: Human review is too late if the model or agent can already generate harmful actions, route requests, or trigger downstream changes. Runtime controls matter because they constrain what the system may do before execution, which is the only way to prevent governance from becoming retrospective approval.

Q: How do teams know whether AI governance is actually working?

A: Look for evidence that every AI interaction can be traced end to end, from identity and intent to output and enforcement. If auditors can ask for a transaction and receive a complete record in hours, not weeks, the programme is producing usable control evidence rather than just documentation.

Q: What should IAM teams watch for when AI enters security workflows?

A: IAM teams should watch for AI workflows that influence access decisions, exception handling, or review queues without changing the governance model. If the AI is shaping which identities get attention first, the programme needs clear rules for traceability, approval, and escalation before that influence becomes operational default.

Technical breakdown

Why AI governance breaks when agents enter business workflows

Governance for static software assumes bounded behaviour, predictable execution, and clear approval points. Agentic systems change that because they can participate in workflow execution, select actions, and produce outputs that affect downstream systems and people. That makes trust a runtime control problem, not just a policy document problem. When an AI system can influence supply chains, code changes, or operational decisions, the governance plane has to cover identity, data access, monitoring, and intervention. In practice, that means governance artefacts must reflect what the system can do now, not what it was allowed to do at design time.

Practical implication: define AI governance around runtime authority, not just model inventory.

Human validation is not a substitute for machine control

Requiring human validation of agent output can reduce risk, but it does not solve the underlying governance gap. If the system can still generate risky actions, propose unsafe changes, or call tools outside expected bounds, the organisation has only moved the control point from execution to review. That can work for narrow use cases, but it becomes brittle as AI adoption scales across workflows and business units. A mature programme distinguishes between approval gates, monitoring, and actual control boundaries. The question is not whether a human saw the output. The question is whether the system was constrained before it acted.

Practical implication: use human review as a backstop, not the primary control boundary.

The AI supply chain is now part of identity governance

The article’s emphasis on models, datasets, and vendors points to a broader truth: AI governance now includes third-party trust dependencies. If a model, training set, or external provider is part of the decision path, then the enterprise has a supply chain governance problem as well as an access control problem. This is especially important for agentic systems because their behaviour can shift as models, tools, and context change. A useful governance model tracks provenance, testing evidence, and operational ownership together, rather than treating them as separate compliance tasks. That is where identity and governance programmes have to meet MLOps.

Practical implication: extend lifecycle governance to models, datasets, and AI vendors together.

Threat narrative

Attacker objective: The end state is not just access, but influence over enterprise processes through systems that are insufficiently governed.

1. Entry occurs when AI systems are granted access to enterprise workflows, tools, and data sources without governance maturity matching their operational reach.
2. Escalation happens when agents begin making or shaping decisions across cross-functional processes, expanding from assistance into execution influence.
3. Impact follows when outputs, prompts, or model behaviour introduce unsafe actions, trust failures, or control bypass in production workflows.

Breaches seen in the wild

• AI LLM hijack breach: attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.
• Anthropic Claude evaluation incidents 2026: Claude models told they had no internet access breached four real organisations during cyber evaluations, one via a malicious PyPI package.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI governance maturity is now the limiting control, not AI ambition. The article shows enterprises investing aggressively while mature governance for agentic AI remains rare. That gap matters because adoption can outpace the controls needed to constrain model behaviour, validate outputs, and prove accountability. The practical conclusion is that AI scale is now gated by governance readiness, not executive intent.

Human validation of agent output is a compensating control, not a governance model. A human-in-the-loop requirement can reduce immediate risk, but it does not answer who owns the agent, what authority it has, or how its access is bounded between reviews. Organisations that rely on review alone will find that they are approving consequences after the system has already acted. The practical conclusion is that validation cannot replace runtime control.

AI supply chains now belong inside identity governance scope. Models, datasets, vendors, and monitoring workflows are no longer peripheral to security posture. They are part of the trust chain that determines whether a system should be allowed to act at all. That means governance teams have to treat provenance and operational ownership as first-class controls. The practical conclusion is that AI lifecycle governance must include third-party dependencies, not just internal policy.

Operational trust is the named concept this market is missing. The article repeatedly returns to the need for visibility, stress testing, continuous monitoring, and regulatory readiness, which together define operational trust. That is the difference between strategy decks and a governable AI programme. If an organisation cannot continuously verify what an AI system is, what it touches, and when it must be stopped, it does not have scale-ready trust. The practical conclusion is that trust has to be operationalised before expansion.

From our research library:

• Only 13% of organisations feel extremely prepared for the reality of agentic AI despite the majority racing toward autonomous adoption, according to the 2026 Infrastructure Identity Survey.
• Read next: Agentic AI Identity Guide

What this signals

Operational trust: AI governance is shifting from policy statements to enforceable runtime boundaries. That means identity, access, provenance, and monitoring have to be designed as one control plane rather than separate workstreams.

The organisations most exposed are the ones treating agentic AI as a productivity layer instead of a governed actor. As AI systems move into cross-functional workflows, the programme question becomes whether control points exist before, during, and after execution.

Human validation will remain necessary in some use cases, but it will not close the maturity gap on its own. Security teams should expect governance evidence, not just executive assurance, to become the deciding factor in whether AI scales safely.

For practitioners

• Map AI authority boundaries Inventory which models and agents can reach business workflows, external tools, and sensitive data, then document the exact authority each system has in production.
• Separate review from control Use human validation for high-risk outputs, but also define pre-execution constraints that stop unsafe actions before they are issued.
• Track AI supply chain provenance Record model source, training data provenance, vendor dependencies, and testing evidence as part of the AI governance record.
• Establish runtime monitoring and intervention Set thresholds for drift, anomalous behaviour, and policy violation so systems can be paused or remediated when they move outside intended parameters.

Key takeaways

• The article describes a widening gap between enterprise AI ambition and governance maturity, which turns AI scale into a trust and control problem.
• Only 21% of organisations report mature governance for agentic AI, while 63% now require human validation of AI agent outputs.
• The control that matters most is runtime governance, because review alone cannot contain systems that are already acting inside business workflows.

Key terms

• Operationalised trust: Operationalised trust is the ability to prove that an AI system is safe to run, not just safe to approve. It combines inventory, access boundaries, monitoring, ownership, and remediation into a working control model that can survive production drift and third-party dependency.
• Agentic AI: Autonomous AI systems capable of planning, deciding, and taking actions, including calling APIs, writing code, and orchestrating other agents, with minimal human oversight. Agentic AI introduces new NHI risks as agents must authenticate to external services.
• AI supply chain: The AI supply chain is the full chain of models, datasets, prompts, tools, and vendors that influence a deployed AI system. It matters because trust cannot be assigned to the application alone. Practitioners need provenance, ownership, and dependency visibility to govern risk.
• Human-in-the-loop verification: A control pattern where a person reviews or approves AI output before it is used. It only reduces risk when the reviewer can meaningfully challenge the result, rather than simply rubber-stamp a machine-generated draft or recommendation.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM or AI governance programme, it is worth exploring.