Notifications
Clear all
Topic starter
25/06/2026 12:24 am
TL;DR: AI explainability helps stakeholders interpret model outputs, but it does not solve governance, accountability, or runtime control problems in high-stakes AI systems, according to WitnessAI. The real issue is that explainable decisions are still governed by opaque access paths, approval chains, and lifecycle assumptions that IAM and NHI programmes must now re-evaluate.
NHIMG editorial — based on content published by WitnessAI: What is Explainability in AI?
By the numbers:
- 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%).
Questions worth separating out
Q: How should security teams govern AI systems that are explainable but still powerful?
A: Security teams should treat explainability as evidence, not permission.
Q: Why is explainability not enough for AI risk management?
A: Explainability helps people understand outputs, but it does not restrict runtime behaviour.
Q: What do organisations get wrong when they rely on post-hoc explanations?
A: They often assume that being able to explain a result means the system is controlled.
Practitioner guidance
- Separate explanation from control design Use explainability methods to support review and debugging, but keep access policy, approval logic, and revocation decisions in separate governance layers.
- Map every AI entitlement and integration Document the data sources, APIs, tools, and workflow hooks the system can reach, then assign a control owner to each boundary.
- Require audit-ready decision records Preserve prompts, model outputs, policy decisions, and downstream actions so investigators can reconstruct what happened without relying on memory.
What's in the full article
WitnessAI's full article covers the operational detail this post intentionally leaves for the source:
- Practical walkthroughs of explainability techniques such as SHAP, LIME, and counterfactual analysis.
- Examples of how different stakeholder groups interpret explanations in regulated environments.
- Discussion of how transparency supports compliance reviews, model monitoring, and audit readiness.
- Guidance on balancing interpretability and model performance in high-stakes use cases.
👉 Read WitnessAI's analysis of explainability in AI systems and governance →
AI explainability and governance: what IAM teams should rethink?
Explore further
25/06/2026 9:18 am
Explainability is a visibility control, not an identity control. The article correctly frames transparency as a way to understand model behaviour, but understanding behaviour does not govern access. For security programmes, the important distinction is that explainable outputs can still emerge from a system with unclear ownership, excessive privileges, or weak lifecycle controls. The practitioner conclusion is that explainability belongs beside governance controls, not in their place.
A few things that frame the scale:
- 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%), according to the AI Agents: The New Attack Surface report.
- Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation.
A question worth separating out:
Q: How do AI explainability and identity governance fit together?
A: Explainability tells you how a model reached an output, while identity governance tells you whether it should have been allowed to act at all. The two are complementary. Strong programmes link model behaviour to ownership, entitlements, approvals, logging, and deprovisioning so that transparency supports control instead of replacing it.
👉 Read our full editorial: AI explainability is not enough for governing autonomous systems
