Notifications
Clear all
Topic starter
20/08/2026 5:40 pm
TL;DR: Accurate model explanations depend on counterfactuals that reflect the model’s actual behaviour, because correlated features can make observational explanations misleading and legally brittle, according to Fiddler. In practice, faithfulness matters more than interpretability alone, especially when automated decisions affect regulated identity, fraud, or access workflows.
NHIMG editorial — based on content published by Fiddler: Causality in Model Explanations and in the Real World
By the numbers:
- When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes, and as quickly as 9 minutes in some cases.
Questions worth separating out
Q: How should organisations validate whether an AI explanation is actually faithful?
A: Use counterfactual tests that change one input at a time and check whether the explanation tracks the model’s behaviour.
Q: Why do correlated features create problems in model governance?
A: Correlated features can make a model look more explainable than it really is, because attribution methods may split importance across inputs that move together.
Q: When should teams prefer intervention evidence over observational explanations?
A: Prefer intervention evidence whenever the model affects regulated or high-impact decisions, such as identity verification, fraud scoring, or access approval.
Practitioner guidance
- Define explanation fidelity tests Require every high-stakes model to prove that its explanation changes when a genuinely causal input changes and remains stable when a merely correlated input changes.
- Separate observational from intervention evidence Label explanations built from historical data as provisional unless they are validated through counterfactual analysis, experiments, or credible natural experiments.
- Document which inputs are ethically adjustable Maintain an approval list of variables that can be altered in testing, especially in identity, fraud, and access workflows where not all interventions are feasible.
What's in the full article
Fiddler's full blog covers the technical and legal nuance this post intentionally leaves at a higher level:
- Detailed walkthrough of how counterfactuals differ from observational attribution in model explanation workflows
- Discussion of Shapley values and why correlated features make exact attribution computationally and conceptually difficult
- Examples of randomized controlled trials and natural experiments as evidence sources for causal claims
- Further context on why faithful explanations matter for legally compliant automated decision-making
👉 Read Fiddler's analysis of causal explainability and faithful model explanations →
Causal explainability and model faithfulness: where teams still get it wrong?
Explore further
20/08/2026 6:24 pm
Faithful explainability is a governance control, not a documentation feature. When explanations diverge from the model’s actual decision logic, the organisation loses the ability to defend, audit, and correct automated outcomes. That matters in identity-heavy workflows where decisions affect trust, access, and risk treatment. Practitioners should treat explanation fidelity as part of control design, not as a communication layer.
A question worth separating out:
Q: What do security and identity teams get wrong about model explanations?
A: They often assume that a human-readable explanation is automatically trustworthy. In practice, a clear narrative can still be wrong if it is built from correlated data rather than causal signals. Teams should evaluate whether the explanation would still hold if the underlying feature relationships changed.
👉 Read our full editorial: Causal explainability needs faithful counterfactuals, not correlations
