AI explainability is not enough for governing autonomous systems

At a glance

What this is: This is an analysis of AI explainability and its governance limits, with a focus on how transparency, auditability, and control interact in regulated AI environments.

Why it matters: It matters because IAM, NHI, and autonomous-system governance all depend on knowing not just what an AI did, but who or what was allowed to act, when, and under which controls.

By the numbers:

• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%).

👉 Read WitnessAI's analysis of explainability in AI systems and governance

Context

AI explainability is the ability to understand how a model reached a result, but transparency alone does not equal governance. In enterprise settings, especially where AI systems influence decisions or act on behalf of users, the unresolved problem is control over access, intent, and accountability across the AI lifecycle.

For IAM and security teams, the core gap is that explainable outputs can still sit on top of weak entitlement design, poor lifecycle controls, or unclear operator ownership. That makes explainability useful for review, but insufficient as a stand-alone security control when AI activity needs to be governed like any other identity-bearing system.

Key questions

Q: How should security teams govern AI systems that are explainable but still powerful?

A: Security teams should treat explainability as evidence, not permission. A model can be understandable and still have excessive access, weak boundaries, or unclear ownership. Governance should define who can approve deployment, what systems the AI may reach, how actions are logged, and how access is revoked when behaviour or integrations change.

Q: Why is explainability not enough for AI risk management?

A: Explainability helps people understand outputs, but it does not restrict runtime behaviour. AI risk management also needs policy enforcement, lifecycle ownership, auditability, and revocation paths. Without those controls, the organisation may be able to explain a decision after the fact while still being unable to prevent unsafe access or action.

Q: What do organisations get wrong when they rely on post-hoc explanations?

A: They often assume that being able to explain a result means the system is controlled. In reality, post-hoc explanations are useful for analysis, but they do not prove least privilege, accountability, or secure delegation. The right test is whether the system can be governed before, during, and after execution.

Q: How do AI explainability and identity governance fit together?

A: Explainability tells you how a model reached an output, while identity governance tells you whether it should have been allowed to act at all. The two are complementary. Strong programmes link model behaviour to ownership, entitlements, approvals, logging, and deprovisioning so that transparency supports control instead of replacing it.

Technical breakdown

Explainable AI vs governable AI systems

Explainability and governance solve different problems. Explainability tells humans how a model produced an output, using methods such as feature attribution, counterfactuals, or post-hoc approximation. Governance asks who approved the model, what data and tools it can reach, when it can act, and how the organisation limits damage if behaviour drifts. In practice, a system can be highly explainable and still unsafe if it has excessive privileges, weak audit trails, or no clear ownership. That distinction matters more as AI moves from prediction into action, where decisions affect identity, data access, and downstream automation.

Practical implication: Treat explainability as an observability layer, not a substitute for access control, lifecycle management, or policy enforcement.

Why black box models create identity governance blind spots

Black box models are hard to inspect because their internal logic is distributed across many parameters rather than a simple rule path. That makes it difficult to predict how the system will behave under novel inputs, but the bigger governance problem is that the organisation may not know which entitlements, data sources, or tools the model can touch. Once an AI system is connected to enterprise systems, its risk profile looks less like a model issue and more like an identity issue. The control failure is not only explainability, but the absence of deterministic boundaries around what the system may access.

Practical implication: Inventory every tool, dataset, and privilege bound to the AI system before you rely on any explanation layer.

How explainability supports audit, but not accountability

Explainability can help investigators reconstruct why a system produced a specific recommendation or classification. It is useful for validation, compliance review, and post-incident analysis. But accountability requires more than being able to explain an output after the fact. It requires explicit ownership, enforceable boundaries, and a control model that survives changes in prompts, data, or model behaviour. In regulated or high-impact use cases, the organisation must be able to prove who authorised the system to act, how that authorisation is reviewed, and what records exist when the system behaves unexpectedly.

Practical implication: Pair explanation methods with named control owners, immutable logs, and reviewable authorisation records.

NHI Mgmt Group analysis

Explainability is a visibility control, not an identity control. The article correctly frames transparency as a way to understand model behaviour, but understanding behaviour does not govern access. For security programmes, the important distinction is that explainable outputs can still emerge from a system with unclear ownership, excessive privileges, or weak lifecycle controls. The practitioner conclusion is that explainability belongs beside governance controls, not in their place.

AI systems become identity problems once they are connected to enterprise tools. When a model can read data, trigger actions, or influence workflows, the risk surface shifts from model internals to access pathways. That means IAM, PAM, and lifecycle discipline become relevant even when the underlying issue is not malicious behaviour. The practitioner conclusion is to govern AI as a subject with access, not as a purely analytical service.

Runtime intent drift: explanations can describe what happened after the fact, but they cannot constrain what the system will decide to do next. This is the core failure mode for organisations that treat post-hoc explainability as a substitute for policy. The implication is that governance must account for action scope, approval boundaries, and revocation, not only interpretability. The practitioner conclusion is that visibility without enforceable policy leaves the control plane incomplete.

Explainability should be measured against auditability, not confidence alone. A model that seems understandable to business users can still be difficult to defend in incident response or compliance review if the organisation cannot tie outputs back to entitlements, prompts, data sources, and ownership. This is where AI governance converges with identity governance. The practitioner conclusion is to evaluate whether the organisation can reconstruct the full decision path, not just explain the prediction.

High-stakes AI needs lifecycle governance because explanation does not expire privileges. Even when a model is well documented, the surrounding access model can drift as integrations change and new workflows are added. That creates a governance gap between what was explained at design time and what is actually permitted at runtime. The practitioner conclusion is to review AI access, ownership, and deprovisioning with the same discipline used for other non-human identities.

From our research:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%), according to the AI Agents: The New Attack Surface report.
• Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation.
• That gap makes OWASP NHI Top 10 and NIST AI Risk Management Framework useful reference points for runtime governance and accountability.

What this signals

Runtime explainability will matter less than runtime control as AI adoption expands. The practical signal for security teams is that model transparency will not compensate for weak ownership, unclear delegation, or over-broad integration scope. When AI systems can act across business workflows, the programme question becomes whether every action path is bounded, reviewed, and revocable, not whether the output can be rationalised after the fact.

Explanation without entitlement discipline creates audit theatre. A system can produce detailed reasoning traces and still leave compliance teams unable to answer who authorised access, which tool was invoked, or whether the action should have been permitted. That is why identity programmes need to extend control thinking into AI-enabled workflows, especially where the model can initiate actions rather than simply recommend them.

With 52% of companies unable to track and audit the data their AI agents access, the governance gap is already operational, not theoretical, and it will widen unless AI access is treated as part of the identity estate. Link review processes, ownership records, and revocation triggers to the systems that actually execute work, not just to the models that explain it.

For practitioners

• Separate explanation from control design Use explainability methods to support review and debugging, but keep access policy, approval logic, and revocation decisions in separate governance layers.
• Map every AI entitlement and integration Document the data sources, APIs, tools, and workflow hooks the system can reach, then assign a control owner to each boundary.
• Require audit-ready decision records Preserve prompts, model outputs, policy decisions, and downstream actions so investigators can reconstruct what happened without relying on memory.
• Review AI lifecycle controls on a fixed cadence Reassess who owns the system, what it can access, and what must be revoked whenever the model, workflow, or integration set changes.

Key takeaways

• Explainability helps people understand AI decisions, but it does not by itself control access, delegation, or runtime scope.
• The evidence from AI agent deployments shows that behaviour already outpaces governance in many organisations, creating audit and compliance blind spots.
• Security teams should connect explainability to identity controls, ownership, and revocation so transparency supports enforcement instead of replacing it.

Key terms

• Explainable AI: Explainable AI is the set of methods used to make model decisions understandable to people. It can use feature attribution, interpretable models, or post-hoc analysis to show why a system produced a result. In governance terms, it improves review and validation, but it does not by itself enforce access or accountability.
• Post-hoc explanation: A post-hoc explanation is an explanation generated after a model makes a decision. It helps users inspect reasoning patterns in complex systems such as deep neural networks, but it is an approximation of behaviour, not the behaviour itself. For governance, it supports audit and debugging rather than control.
• Runtime governance: Runtime governance is the discipline of controlling what an AI system may do while it is operating. It covers authorisation, logging, policy enforcement, and revocation, so decisions stay within approved boundaries. In AI environments, runtime governance matters because transparency alone cannot stop unsafe actions.
• Identity-bound AI system: An identity-bound AI system is an AI service or agent that can access enterprise data, APIs, or workflows under an assigned identity or delegated authority. That identity creates accountability, but also introduces lifecycle, privilege, and revocation requirements that should be managed like other non-human identities.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.

This post draws on content published by WitnessAI: What is Explainability in AI? Read the original.