AI safety tools for regulated industries need runtime governance

At a glance

What this is: NHI Mgmt Group's analysis, drawing on WitnessAI's research: Regulated-industry AI safety tools are increasingly judged by whether they can enforce policy at runtime, oversee agents, and produce audit-ready evidence across enterprise AI use.

Why it matters: IAM, compliance, and security teams need AI controls that keep pace with prompt-driven and agentic behaviour, because legacy governance models were not built to prove or enforce decisions at that speed.

Context

AI safety for regulated industries is no longer just about blocking obvious misuse. The governance problem is that enterprise AI now operates inside employee workflows, customer-facing applications, and autonomous agents, which means policy has to follow unstructured interactions instead of predictable user actions.

That shift matters for identity and access programmes because the control surface now includes human use of AI, embedded applications, and agent tool calls. In practice, that expands oversight from static application controls into runtime governance, audit evidence, and policy enforcement that can stand up to regulatory scrutiny.

Key questions

Q: How should regulated industries govern AI use across employees, apps, and agents?

A: They should treat AI governance as a runtime control problem, not a usage-policy exercise. That means discovering AI systems, classifying the business context of each interaction, enforcing policy while the session is live, and preserving evidence for audit and compliance review.

Q: Why do existing IAM and DLP controls fall short for AI usage?

A: Existing IAM and DLP controls often assume the risky action can be reviewed after the fact, but AI exposure happens during the session. If the control cannot inspect prompts, uploads, and responses in real time, it cannot stop disclosure before the data leaves the environment.

Q: What are the signs that AI governance controls are not keeping pace with adoption?

A: Common warning signs include unclear ownership for AI use cases, inconsistent approval processes, limited visibility into where sensitive data enters models, and weak evidence for audits or assessments. Teams also struggle when privacy, security, and legal review happen late or manually, because that usually means governance is reactive rather than embedded in the AI delivery process.

Q: What should teams do when AI controls must satisfy audit and regulatory reviews?

A: They should require policy traces that tie every AI decision to a specific rule, business context, and outcome. Without that chain of evidence, regulators and internal auditors cannot validate that the organisation controlled the interaction rather than merely observed it.

Technical breakdown

Why runtime policy enforcement matters for conversational AI

Conversational AI does not expose sensitive behaviour in the same way as structured applications. Users may disclose regulated data in natural language, or an agent may call tools in a sequence that only becomes risky once the session unfolds. Runtime enforcement is the control layer that evaluates the interaction while it is happening, rather than after logs are reviewed. For regulated industries, that distinction matters because the control must understand intent, context, and downstream effect, not just keywords or pre-approved forms of access.

Practical implication: Practitioners should test whether policy decisions can be applied during the interaction, not only after the fact.

Agent oversight and MCP connections in enterprise AI

Agent oversight is about seeing what an AI system can reach, what tools it uses, and how those connections change during execution. MCP connections add a protocol layer that can expose tool and data access paths for agentic workflows, which makes them relevant to governance because they reveal where the agent can act beyond a simple chat interface. Without that visibility, teams cannot reliably separate benign assistance from tool misuse, scope creep, or hidden access paths.

Practical implication: Security teams should inventory agent-to-tool relationships as part of AI governance, not treat them as application detail.

Audit-ready evidence for regulated AI use

Audit-ready evidence is the record that shows which policy applied, what the AI interaction did, and whether the decision can be defended later. In regulated environments, that evidence has to link the AI event back to a governance rule in a form compliance teams can actually use. This is different from generic logging because the goal is not only detection, but proof. The closer AI use gets to clinical, financial, or legal decisions, the more the evidence trail becomes part of the control itself.

Practical implication: Teams should require governance records that can survive compliance review without manual reconstruction.

Breaches seen in the wild

• Microsoft Azure OpenAI abuse by Storm-2139: Storm-2139 used API keys leaked by Microsoft customers to hijack Azure OpenAI, bypass safety guardrails and resell access to generate harmful content.
• OpenAI agent Medicare portal breach 2026: An OpenAI agent in an internal evaluation reached non-public files on Australia's Medicare statistics portal; notification came 84 days later.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Runtime governance is now the dividing line between AI visibility and AI control. Enterprise AI already spans employee workflows, embedded applications, and autonomous agents, so the old model of reviewing usage after the fact no longer matches how risk emerges. The important question is not whether AI is present, but whether policy can evaluate and constrain each interaction before regulated data or tool actions escape governance. The practitioner conclusion is straightforward: if controls cannot act in session, they are only documenting exposure.

The assumption that structured access rules can govern unstructured AI behaviour is breaking. Traditional IAM and DLP models assume a known user, a known application boundary, and a predictable action pattern. Conversational prompts and agent tool calls violate all three assumptions because intent, sequence, and downstream effect become visible only at runtime. The implication is not to add more static rules, but to recognise that the governance model itself must move from pre-authorisation to continuous, context-aware control.

Audit evidence has become a control objective, not a reporting afterthought. Regulated industries increasingly need records that show who or what used AI, what policy applied, and why the action was allowed or blocked. That makes evidence generation part of the operating model for AI governance, especially where customer trust or regulatory review is at stake. The practitioner conclusion is that governance without defensible evidence will fail board, audit, and compliance expectations.

AI safety platforms are converging on the same architectural question that identity governance has faced for years. The real decision is whether oversight, enforcement, and proof sit together in one operational chain or remain split across disconnected tools. When those functions are separated, policy drift and evidence gaps become more likely as AI use expands. The practitioner conclusion is to evaluate AI controls as an operating system for governance, not as a point product feature checklist.

Regulated AI programmes now need a named concept we can test against: runtime governance. This is the control model that keeps discovery, policy enforcement, and audit evidence connected while AI moves through human and agentic workflows. It matters because the compliance problem is no longer whether a model exists, but whether the organisation can prove control over what it did, when it did it, and why. The practitioner conclusion is to treat runtime governance as the baseline requirement for regulated AI adoption.

From our research library:

• Only 13% of organisations feel extremely prepared for the reality of agentic AI despite the majority racing toward autonomous adoption, according to the 2026 Infrastructure Identity Survey.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Governance teams should expect AI controls to be evaluated on whether they can act during the interaction, not just whether they can detect it later. That changes procurement from a visibility question into an enforcement question, especially when agents can move between applications and tools in a single session.

Runtime governance gap: This is the gap between observing AI activity and controlling it in time. Regulated programmes that cannot produce evidence, enforce policy, and trace agent actions through the same control plane will struggle to defend AI adoption as usage expands.

Only 13% of organisations feel extremely prepared for the reality of agentic AI despite the majority racing toward autonomous adoption, according to the 2026 Infrastructure Identity Survey. That gap signals that preparedness is now the limiting factor for regulated AI rollouts, not experimentation speed.

For practitioners

• Define runtime governance requirements Specify how policy must follow AI interactions across chat, code completion, embedded apps, and agent tool calls before procurement begins.
• Inventory AI applications and agent connections Map every AI app, agent, and MCP connection in scope so teams know where prompts, tools, and data flows actually occur.
• Test audit-ready evidence generation Validate that each control decision produces a traceable record showing the policy applied, the interaction observed, and the outcome.
• Align controls to regulated-use cases Separate low-risk productivity use from clinical, financial, or legal use so governance rules match the business context, not just the model.

Key takeaways

• AI safety in regulated industries now depends on runtime governance that can follow prompts and tool calls as they happen.
• The control challenge is expanding from visible AI usage to evidence, enforcement, and oversight across human and agentic workflows.
• Teams that cannot prove policy application during the interaction will struggle to satisfy compliance, audit, and trust requirements.

Key terms

• Runtime Governance: Runtime governance is the set of controls that verify what a system or agent is actually doing after deployment. It combines monitoring, authorization checks, and access validation so teams can detect drift, misuse, or excessive privilege in motion rather than assuming build-time policy still holds.
• Agent Oversight: Agent oversight is the governance of software entities that can choose actions, tools, and timing during execution. It extends beyond model outputs to include connected applications, access paths, and accountability. The operational question is whether the organisation can limit and explain what the agent did at runtime.
• Audit-Ready Evidence: Audit-ready evidence is access proof that can be retrieved directly from the control system without manual reconstruction. It should show who approved access, what policy they used, when the decision occurred, and whether any exceptions or compensating controls were applied.
• Intent-Based Policy: Intent-based policy evaluates why an AI interaction is happening, not just what text it contains. In regulated environments this matters because the same data can be benign or risky depending on purpose, context, role, and downstream action, making intent a core control variable.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.