AI explainability and governance: what IAM teams should rethink

TL;DR: AI explainability helps stakeholders interpret model outputs, but it does not solve governance, accountability, or runtime control problems in high-stakes AI systems, according to WitnessAI. The real issue is that explainable decisions are still governed by opaque access paths, approval chains, and lifecycle assumptions that IAM and NHI programmes must now re-evaluate.

NHIMG editorial — based on content published by WitnessAI: What is Explainability in AI?

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%).

Questions worth separating out

Q: How should security teams govern AI systems that are explainable but still powerful?

A: Security teams should treat explainability as evidence, not permission.

Q: Why is explainability not enough for AI risk management?

A: Explainability helps people understand outputs, but it does not restrict runtime behaviour.

Q: What do organisations get wrong when they rely on post-hoc explanations?

A: They often assume that being able to explain a result means the system is controlled.

Practitioner guidance

• Separate explanation from control design Use explainability methods to support review and debugging, but keep access policy, approval logic, and revocation decisions in separate governance layers.
• Map every AI entitlement and integration Document the data sources, APIs, tools, and workflow hooks the system can reach, then assign a control owner to each boundary.
• Require audit-ready decision records Preserve prompts, model outputs, policy decisions, and downstream actions so investigators can reconstruct what happened without relying on memory.

What's in the full article

WitnessAI's full article covers the operational detail this post intentionally leaves for the source:

• Practical walkthroughs of explainability techniques such as SHAP, LIME, and counterfactual analysis.
• Examples of how different stakeholder groups interpret explanations in regulated environments.
• Discussion of how transparency supports compliance reviews, model monitoring, and audit readiness.
• Guidance on balancing interpretability and model performance in high-stakes use cases.

👉 Read WitnessAI's analysis of explainability in AI systems and governance →

AI explainability and governance: what IAM teams should rethink?

Explore further

View Full Forum →  |  NHI Foundation Course →