AI governance for regulated industries still fails at production

At a glance

What this is: This article argues that regulated industries can move AI into production only when governance evidence, compliance enforcement and runtime inspection are built into the operating model, not bolted on after pilots succeed.

Why it matters: For IAM, PAM, NHI and AI governance teams, the lesson is that production approval now depends on evidence of control across prompts, tools, identities and audit trails, not just model performance.

Context

AI governance for regulated industries is no longer a lab problem. In regulated sectors, the core issue is proving that AI systems can operate inside existing compliance obligations while still being observable, accountable and enforceable in production.

The article’s central point is that the pilot-to-production gap is driven by governance evidence, not model capability alone. That matters because regulated AI now sits at the intersection of risk, auditability and operational accountability, where legacy security patterns often fail to map cleanly to conversational systems and runtime decisions.

The article also treats regulated AI as a cross-functional identity and control problem. Humans, tools and autonomous actions all show up in the same workflow, so the question becomes whether the organisation can prove control over the full interaction path before rollout expands.

Key questions

Q: How can organisations prove AI governance to auditors and boards?

A: Organisations prove AI governance by producing evidence that the control operated, not just that a policy existed. That evidence should include inventory records, runtime logs, policy decisions, and escalation handling for both sanctioned and unsanctioned AI use. Framework alignment helps, but auditors and boards usually want demonstrable execution, not framework language alone.

Q: Why do AI pilots fail to reach production so often?

A: AI pilots fail when organisations design for experimentation but not for operational control. Production requires trustworthy data, access boundaries, monitoring, change control, and clear accountability. Without those controls, the pilot may work in isolation but collapse under real-world dependency, security, and compliance requirements.

Q: What breaks when legacy security controls are used for conversational AI?

A: Controls built for files, web traffic and static application sessions miss the interaction context that drives AI risk. They do not reliably interpret intent, tool use or the handoff from prompt to response to downstream action. That leaves regulated enterprises with controls that look familiar but do not actually govern the production flow.

Q: How should security teams choose between ISO 42001 and NIST AI RMF 1.0 for AI governance?

A: Choose ISO 42001 when you need a certifiable management system, external audit evidence, and a formal governance signal for customers or regulators. Choose NIST AI RMF 1.0 when you need a flexible, engineer-friendly risk playbook for day-to-day AI controls. Many mature organisations use both, with NIST guiding operations and ISO providing the umbrella for accountability and assurance.

Technical breakdown

Why conversational AI breaks legacy control assumptions

Conversational AI changes the control problem because the risk is not limited to static files, web sessions or traditional application transactions. A prompt can carry intent, data, instructions and policy-relevant context in one exchange, and the model can produce output that is immediately acted on by a user or downstream system. Legacy DLP, web filtering and file-centric inspection do not understand conversation state, tool invocation or policy at the point of inference. That is why regulated AI needs controls that inspect meaning, not only payloads.

Practical implication: place policy enforcement at the interaction layer, not only at storage, endpoint or network boundaries.

How runtime evidence becomes the real compliance barrier

The article makes a clear distinction between having an AI policy and being able to demonstrate that the policy is enforced in production. Regulators and risk committees are not satisfied by declarations; they need auditable evidence that the control operated when the system handled regulated data, made a decision or invoked a tool. In practice, that means policy logs, traceability, identity attribution and enforcement records must survive the full workflow. Without them, compliance remains theoretical even if the model itself is technically sound.

Practical implication: design AI controls so every significant interaction leaves an audit trail that can be produced during review.

Why intent-aware policy matters more than keyword filtering

The article argues that AI risk depends on purpose, not just content patterns. Two prompts can carry the same data and still represent very different risk if one is routine business use and the other is malicious or non-compliant. Keyword rules and regex logic were built for simpler detection problems, so they struggle with conversational context, mixed intent and rapidly changing tool use. Intent-aware policy is therefore the operational layer that lets teams distinguish acceptable business use from prohibited or high-risk use at runtime.

Practical implication: classify AI interaction intent before enforcing policy, rather than relying on static content matching alone.

Threat narrative

Attacker objective: The objective is to trigger unsafe or non-compliant AI-driven actions while avoiding the kind of evidence regulators would require to prove control.

1. Entry occurs when users, employees or agents send prompts and business content into AI systems that sit inside regulated workflows.
2. Access and abuse occur when the interaction includes sensitive data, hidden instructions or tool-enabled actions that are not adequately inspected before execution.
3. Impact follows when the model or connected toolchain produces an unauditable, non-compliant or harmful outcome that cannot be defended to regulators or risk committees.

Breaches seen in the wild

• Spain's first AI agent data breach 2026: Spain's AEPD logged its first breach notification attributed to an attacker's AI agent, which altered personal data and accessed invoices.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

The production problem in regulated AI is an evidence problem, not a model-quality problem. The article shows that organisations can reach a technically workable AI state while still failing the proof burden demanded by compliance, audit and risk review. That means the limiting factor is no longer whether the model can answer the question, but whether the organisation can demonstrate what happened, why it happened and under what policy. Practitioners should treat production readiness as an evidentiary control plane.

Legacy security controls were designed for file, web and endpoint boundaries, not conversational enforcement. The article’s strongest implication is that prompt-driven systems collapse the assumptions behind those controls because the risk lives in interaction context, not just in stored objects. A policy written for static data flows cannot on its own govern tool use, response content or downstream AI action. Practitioners should reframe AI security around runtime policy enforcement rather than retrofit thinking.

AI governance in regulated sectors now spans human identities, digital workflows and autonomous actions in one chain. That makes accountability a cross-domain problem, not a single-team problem. The article’s architecture point is that production control must connect human intent, tool authorisation and auditable execution in the same governance model. Practitioners should stop treating AI oversight as a separate programme and fold it into identity, security and compliance governance together.

Runtime inspection is becoming the named concept that separates pilot success from production control. The article repeatedly points to the need for policy enforcement at the moment of interaction, with visibility into prompts, responses and tool activity. That is the point where regulated AI either remains governable or becomes an unproven business risk. Practitioners should build around runtime inspection as a distinct governance layer, not an add-on to existing monitoring.

NIST AI RMF and ISO/IEC 42001 are functioning as evidence frameworks, not just governance references. The article shows how regulated organisations are using them to turn broad obligations into controls that can be seen, measured and explained. That matters because compliance teams need a repeatable way to translate policy into proof across jurisdictions. Practitioners should align AI governance artefacts to frameworks that regulators and auditors can actually evaluate.

What this signals

Runtime inspection is becoming the operational boundary for regulated AI. Teams should expect AI governance to shift from policy documents toward controls that inspect prompts, responses and tool use at the moment of execution. That change matters because production approval will increasingly depend on whether the organisation can explain and evidence decisions after the fact.

Regulated AI programmes should also expect identity and accountability to converge. Once humans, tools and agents are part of the same workflow, governance has to preserve traceability across that chain rather than treating each component separately. The practical test is whether a compliance team can reconstruct who or what caused the AI action without guesswork.

For practitioners

• Define production readiness as demonstrable control Require evidence that governance, security and compliance controls operate in live AI environments before wider rollout, not after the pilot phase closes.
• Map AI interactions to audit trails Capture prompts, responses, tool calls and identity attribution in a form that can satisfy risk committees and regulatory review.
• Move policy enforcement to runtime Apply intent-aware controls at the point of inference so the system can inspect meaning before an AI action is taken or propagated.
• Separate acceptable use from prohibited intent Classify the purpose of an interaction before decisioning, because identical content can carry very different business and regulatory risk.
• Unify governance across people and digital workers Treat human users, agentic workflows and AI tools as one accountability chain so evidence and ownership do not fragment across teams.

Key takeaways

• Regulated AI fails most often at the point where organisations must prove governance in live environments rather than merely describe it in policy.
• The article’s evidence points to a control gap between AI experimentation and production accountability, not a simple model-performance problem.
• Practitioners need runtime enforcement, audit trails and intent-aware policy if they want AI deployments to survive regulatory scrutiny.

Key terms

• Runtime Inspection: Real-time monitoring of AI prompts and responses while the interaction is happening. Unlike point-in-time review, runtime inspection can detect sensitive data, prompt injection, and harmful outputs during the session, which is essential when risk changes faster than governance cycles.
• Intent-Aware Policy: A policy model that evaluates what an actor is trying to accomplish, not just which technical action it requested. For agents, this matters because the same tool call can be legitimate or risky depending on prompt, context, and the sequence of actions that follows.
• Evidence of Control: Evidence of control is the artefact set that proves a security or governance control actually operated in production. For AI programmes, that usually includes logs, policy decisions, identity attribution and records of enforcement that can survive audit or regulatory scrutiny.
• Production readiness: The state in which code is fit to move through normal release processes without creating avoidable operational, security, or integration risk. For AI-generated code, readiness depends on contextual fit, reviewability, and testable alignment with established engineering standards.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.