Notifications
Clear all
Topic starter
06/06/2026 1:44 am
TL;DR: Explainable AI in finance is moving from post-hoc model explanations to runtime governance as credit, fraud, and AML decisions face stricter transparency demands, while generative and agentic AI fall outside traditional model risk frameworks, according to WitnessAI. The governance gap is no longer whether a model can explain itself, but whether institutions can prove controlled AI behaviour in production.
NHIMG editorial — based on content published by WitnessAI: explainable AI in finance and runtime governance for generative and agentic AI
By the numbers:
- The Bank of England and FCA’s 2024 survey found SHAP and feature importance are widely used among UK financial institutions.
- The most concrete near-term deadline is August 2, 2026, when the main provisions of the EU AI Act begin to apply.
Questions worth separating out
Q: How should financial institutions govern explainable AI in high-risk use cases?
A: They should match the explanation method to the decision and audience, then back it with audit trails and policy controls.
Q: Why do generative and agentic AI create problems for traditional model risk management?
A: Traditional model risk management assumes stable inputs, stable outputs, and a bounded decision path that can be validated before deployment.
Q: What do financial services teams get wrong about SHAP and LIME?
A: They often treat them as universal explanation tools, when they are better understood as partial methods for specific model types.
Practitioner guidance
- Map explanation method to decision audience Use local explanations for consumer-facing decisions, global explanations for model validation, and separate treatment for investigator workflows.
- Separate scorecard governance from generative AI governance Keep interpretable model controls for structured decisioning, but add runtime observability and policy enforcement for systems that retrieve data, draft content, or initiate actions.
- Inventory shadow AI and agentic workflows Maintain a live list of approved models, external AI tools, and autonomous workflows so compliance teams can see where regulated data or customer decisions are being exposed.
What's in the full article
WitnessAI's full article covers the operational detail this post intentionally leaves for the source:
- Step-by-step coverage of SHAP, LIME, counterfactuals, and inherently interpretable models for different financial use cases.
- The article’s practical comparison of explanation methods for credit underwriting, fraud detection, and AML decisioning.
- Its discussion of runtime governance architecture for AI activity, including observability, policy enforcement, and data protection.
- The regulatory discussion around current and forthcoming financial services expectations for high-risk AI use cases.
👉 Read WitnessAI's analysis of explainable AI in finance and runtime governance →
Explainable AI in finance: are your controls keeping up?
Explore further
06/06/2026 3:02 am
Explainability is no longer a sufficient governance primitive for financial AI. In credit, fraud, and AML, explanation methods still matter because they help reviewers understand model decisions. But once systems move into generative and agentic behaviour, the central governance question shifts from explanation to control, because the institution must evidence what the system did in production. Practitioners should treat explainability as necessary, but no longer as the whole governance model.
A few things that frame the scale:
- The average organisation believes more than 1 in 5 of their non-human identities are insufficiently secured, according to The 2024 ESG Report: Managing Non-Human Identities.
- 72% of organisations have experienced or suspect they have experienced a breach of non-human identities, including 46% confirmed and 26% suspected.
A question worth separating out:
Q: How do AI transparency requirements change when systems can act autonomously?
A: Transparency requirements move from explaining a single model output to proving control over a sequence of actions. If an AI system can query data, draft communications, and trigger transactions, institutions need visibility into each step, not just the final answer. That makes runtime policy enforcement and auditability central to accountability.
👉 Read our full editorial: Explainable AI in finance now requires runtime governance
