AI governance for regulated industries still fails at production

At a glance

What this is: This is an analysis of why regulated industries struggle to move AI from pilot to production, with the central finding that governance evidence, not technical capability, is the main blocker.

Why it matters: It matters because IAM, security, and compliance teams now need controls that can prove accountability for AI interactions, tool use, and auditability across human and digital workforces.

By the numbers:

• Only 44% of AI proofs of concept had reached production as of early 2025.
• Gartner predicted 30% of generative AI projects would be abandoned after proof of concept by end of 2025.
• A survey of 3,235 business and IT leaders across 24 countries found that governance is central to successfully scaling AI.
• Visa’s AI tools blocked 85% more fraudulent transactions on Cyber Monday 2024 compared to the prior year.

👉 Read WitnessAI’s analysis of AI governance in regulated industries

Context

Regulated industries are finding that the hardest part of AI adoption is not model performance but proving control. In environments governed by sector rules, AI deployment only becomes viable when governance, accountability, and audit evidence are visible in production, not just in a pilot.

That is why AI governance now sits at the intersection of IAM, compliance, and operational resilience. The article’s core argument is that legacy file- and web-focused controls do not map cleanly to conversational AI, tool use, or runtime decision-making, especially when regulators expect evidence before rollout.

Key questions

Q: How should regulated industries move AI from pilot to production without losing control?

A: They should require auditable governance before scale, not after it. That means defining policy, traceability, human accountability, and runtime enforcement up front, then proving those controls in production conditions. If the organisation cannot show regulators what the system did, who approved it, and how risk was contained, the deployment is not ready for expansion.

Q: Why do traditional security controls fail for conversational AI in regulated environments?

A: Because they were built for static content and fixed destinations, not for interactions that change context, invoke tools, and trigger actions. Conversational AI turns prompts and outputs into active control points, so legacy web and file filters miss the decision path that creates regulatory exposure. The result is a control gap, not just a monitoring gap.

Q: How do organisations know if AI governance is strong enough for regulators?

A: They know it is working when they can produce repeatable evidence of enforcement, not just written policy. That evidence should show runtime controls, audit trails, human accountability, and framework-aligned measurement. If governance only exists in documentation and not in system behaviour, it will not satisfy oversight expectations.

Q: What is the difference between AI risk management and AI runtime defence?

A: AI risk management defines the governance structure, measures obligations, and assigns accountability. AI runtime defence applies those rules during live interactions by inspecting prompts, responses, and actions before they create harm. Regulated organisations need both, but runtime defence is what turns policy into enforceable behaviour.

Technical breakdown

Why legacy web and data controls fail for conversational AI

Conversational AI changes the security boundary because the interaction is not a simple request-response transaction. Prompts can carry instructions, context can change mid-session, and responses can trigger downstream actions, which means file-centric or web-centric controls miss the actual risk surface. Traditional DLP and web gateways were built to inspect static content and destination rules, not intent, tool invocation, or agent behaviour. In regulated environments, that gap matters because the control must stand up to audit, not just block obvious exfiltration attempts.

Practical implication: move from content-only inspection to policy enforcement that understands AI context, action paths, and runtime behaviour.

How regulated AI maps to NIST AI RMF and ISO/IEC 42001

The article treats NIST AI RMF and ISO/IEC 42001 as operational scaffolding for proving governance. NIST AI RMF organises the programme into Govern, Map, Measure, and Manage, while ISO/IEC 42001 adds a management-system structure that auditors can inspect. Together, they help teams translate broad regulatory obligations into repeatable evidence around oversight, testing, accountability, and continuous monitoring. This is especially useful where a single deployment can trigger multiple obligations at once, such as EU AI Act and DORA in financial services.

Practical implication: align AI control design and evidence collection to a framework map before production deployment expands.

What runtime defence means in regulated AI operations

Runtime defence means controlling AI at the point of interaction, not only at design time. In the article’s model, that includes bidirectional inspection of prompts and responses, intent-aware policy enforcement, identity attribution, and audit trails that connect actions to responsible people. The key point is that production AI often fails in the inference layer, where static reviews cannot see what the model did with context already in motion. That makes runtime controls part of governance, not an optional security add-on.

Practical implication: require runtime policy, traceability, and response inspection before allowing regulated workloads to move beyond limited trials.

Breaches seen in the wild

• DeepSeek breach — DeepSeek breach exposed 1M+ log lines and sensitive secret keys.
• Schneider Electric credentials breach — exposed credentials gave attackers access to Schneider Electric Jira, exfiltrating 40GB.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Governance evidence is now the production gate for regulated AI. The article shows that technical readiness has outpaced proof of control, which is why pilot programmes stall before broad rollout. Risk committees and compliance teams are no longer asking whether AI can work, but whether it can be shown to work inside the existing control environment. Practitioners should treat evidence generation as a deployment prerequisite, not a later validation step.

Legacy AI security assumptions break because conversational systems are policy-active, not passive. File and web controls assume discrete objects moving through fixed boundaries, but AI interactions can combine intent, context, tool access, and action execution in one flow. That makes static inspection incomplete for regulated workloads, especially where a single interaction can create compliance exposure. Practitioners should re-evaluate controls that only inspect artifacts rather than decisions.

NIST AI RMF and ISO/IEC 42001 are becoming the shared language of AI accountability. The article is pointing to a market pattern, not a single-product answer: regulated buyers need audit-ready control structures that regulators can understand. That pushes AI programmes toward documented governance, measurable enforcement, and repeatable operating models. Practitioners should align internal controls to framework language early so evidence scales with deployment.

Regulated AI is converging with identity governance, not replacing it. As AI systems gain access to tools, data, and workflows, the governance problem becomes who or what acted, under which authority, and with what traceability. This is where IAM, compliance, and AI security overlap. Practitioners should expect AI oversight to become a cross-functional identity control problem rather than a standalone model-risk exercise.

From our research:

• Organisations that describe themselves as confident in their AI deployment actually experience a 72% security incident rate, compared to 33% for those who remain cautious, according to the 2026 Infrastructure Identity Survey.
• Only 44% of organisations have implemented any policies to manage their AI agents, despite 92% agreeing that governing AI agents is critical to enterprise security.
• That gap between confidence and control is why OWASP NHI Top 10 is becoming a practical reference point for runtime AI governance.

What this signals

Policy evidence will matter more than model performance as AI moves into regulated workflows. The market signal is clear: organisations that can show governance, runtime enforcement, and traceability will move faster than those relying on pilot success alone. The practical challenge is not adding another AI control layer, but making existing oversight legible to compliance and audit teams.

With 70% of organisations already granting AI systems more access than human employees, per the 2026 Infrastructure Identity Survey, the governance gap is structural. That pattern tells identity teams that AI access is being granted faster than accountability models are being updated. Practitioners should expect pressure to prove that access, action, and review are linked before broader production approval.

Runtime inspection is emerging as the practical control plane for regulated AI. Teams that can enforce intent-aware policy and connect behaviour to accountable identities will have the clearest path to production. For broader control design, the Ultimate Guide to NHIs , Lifecycle Processes for Managing NHIs is a useful anchor for thinking about governance beyond the model itself.

For practitioners

• Build production evidence before expanding scope Require every regulated AI deployment to produce auditable proof of governance, policy enforcement, and human accountability before broad rollout. Make that evidence part of the go-live checklist, not a post-launch exercise.
• Replace content-only controls with context-aware policy Move from file and web inspection to intent-based enforcement that can evaluate prompts, tool calls, and response handling in the same session. Use policy that understands role, geography, and business context.
• Map controls to named frameworks early Anchor the programme to NIST AI RMF and ISO/IEC 42001 so compliance teams can trace governance, measurement, and management evidence back to recognised structures. That reduces friction when multiple regulators apply at once.
• Treat AI interactions as identity-bearing events Connect each AI action to a human owner or accountable process, then preserve immutable audit trails for prompts, responses, and downstream actions. This is the minimum basis for regulated oversight.

Key takeaways

• Regulated AI stalls when organisations cannot prove control in production, even when the underlying technology is ready.
• The scale of the governance gap is visible in the numbers, with production conversion lagging while compliance expectations keep tightening.
• Teams need runtime policy, traceability, and framework-aligned evidence if they want AI deployments to survive regulatory scrutiny.

Key terms

• Runtime Defence: Runtime defence is the practice of controlling AI behaviour while the system is actively processing prompts, responses, or tool actions. It focuses on live enforcement, not just pre-deployment review, because regulated risk often appears at the moment of inference and execution.
• Intent-Based Policy: Intent-based policy evaluates why an AI interaction is happening, not just what text it contains. In regulated environments this matters because the same data can be benign or risky depending on purpose, context, role, and downstream action, making intent a core control variable.
• AI Governance Evidence: AI governance evidence is the auditable proof that controls are working in practice, not only on paper. It includes policy logs, review records, traceability, and enforcement artefacts that show a regulator how AI decisions were constrained, attributed, and monitored.
• Bidirectional Inspection: Bidirectional inspection means examining both prompts sent to an AI system and the responses it produces. This is essential when outputs can trigger follow-on actions, leak sensitive data, or carry instructions that affect downstream systems, users, or automated workflows.

Deepen your knowledge

AI governance for regulated industries is a core topic in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building a control model for AI in a regulated environment, it is worth exploring.

This post draws on content published by WitnessAI: AI governance for regulated industries and the production gap. Read the original.