Explainable AI governance: who should verify model decisions?

TL;DR: Bias in AI often emerges after deployment because black-box models hide the data patterns and decision logic that drive outcomes, according to Fiddler, which argues that explainability must be built into the AI lifecycle from design through production. Independent oversight matters because trust in model outputs depends on verifiable explanations, not vendor-generated assurances.

NHIMG editorial — based on content published by Fiddler: Explainable AI Goes Mainstream But Who Should Be Explaining?

Questions worth separating out

Q: How should organisations govern AI systems that can make consequential decisions?

A: Organisations should govern consequential AI systems with the same discipline used for high-risk identities: defined ownership, least privilege, logging, approval boundaries, and human override.

Q: Why do AI models create governance risk even without retraining?

A: Because behaviour can change at inference time when the model sees new context, examples, or instructions.

Q: How do you know if explainable AI is actually working?

A: It is working when analysts can resolve cases faster, false declines drop, customer complaints decrease and reviewers make more consistent decisions from the same evidence.

Practitioner guidance

• Define explanation requirements before model deployment Specify what evidence a model must provide for training data, feature influence, and prediction rationale before it can be released into production.
• Separate model building from independent review Assign a reviewer who did not develop the model to test whether explanations are consistent, auditable, and sufficient for the business decision being made.
• Track explanation quality across the lifecycle Monitor whether explanations remain stable as models are retrained, data changes, and business rules evolve, especially where AI affects identity or access decisions.

What's in the full article

Fiddler's full blog covers the operational detail this post intentionally leaves for the source:

• The article’s original examples of AI bias in healthcare, lending, and judicial decision contexts.
• The author’s reasoning on why third-party explainability matters for trust in model outputs.
• Fiddler’s perspective on the role of human-in-the-loop monitoring in ethical AI workflows.

👉 Read Fiddler’s blog on explainable AI and independent model oversight →

Explainable AI governance: who should verify model decisions?

Explore further

View Full Forum →  |  NHI Foundation Course →