Causal explainability needs faithful counterfactuals, not correlations

At a glance

What this is: This is a deep dive on causal explainability, showing that faithful counterfactuals are necessary when correlated features can make model explanations misleading.

Why it matters: It matters to IAM, fraud, and identity governance teams because automated decisions used in access, onboarding, or risk workflows can only be defended if the explanation reflects actual model behaviour, not a convenient proxy.

Context

Causal explainability is about determining whether a feature truly changes a model’s decision, rather than merely moving with other features that happen to be correlated. In regulated decisioning, that distinction matters because an explanation that feels plausible can still be wrong if it is built from observational patterns instead of interventions.

For identity and access workflows, the governance problem is not just interpretability but evidential faithfulness. When model outputs influence fraud triage, onboarding, or access decisions, teams need explanations that can stand up to challenge and do not smuggle correlation into policy decisions.

Key questions

Q: How should teams test whether a model explanation is faithful?

A: Use counterfactual interventions that change one feature at a time and verify whether the prediction changes in the way the explanation claims. If the explanation does not track the model’s actual response, it is descriptive at best and misleading at worst. Faithfulness should be validated against model behaviour, not inferred from correlation alone.

Q: Why do correlated features make model explanations unreliable?

A: Because correlations let an explanation assign importance to a feature that may only be moving with the real driver. A model can appear to rely on both inputs when it is actually using only one, so observational attribution can split credit incorrectly. That creates false confidence in the explanation and weakens governance evidence.

Q: When do explanations for automated decisions become legally risky?

A: They become risky when the explanation describes a plausible pattern rather than the actual logic used by the model. In regulated workflows, especially where automated decisions affect identity, fraud, or access, teams need meaningful information about the decision basis. If the explanation cannot survive intervention testing, it is not strong enough for audit or disclosure.

Q: What should identity and fraud teams do before relying on model explanations?

A: They should verify that the explanation method reflects causal influence, define which counterfactuals are realistic, and check for proxy features that can misstate the decision basis. That gives teams a defensible record for onboarding, access, or fraud workflows where model output affects real-world action.

Technical breakdown

Why correlated features distort model explanations

When features move together, a model explanation can assign importance to the wrong input even if the model itself does not rely on it. Observational methods infer contribution from historical co-occurrence, which breaks down when one feature is only a proxy for another. This is why a model can appear to use both income and zip code equally when only one feature is actually driving the prediction. The core technical issue is that explanation quality depends on the set of counterfactuals you allow, not just the scoring method. If the counterfactual set is unrealistic, the explanation becomes unstable or misleading.

Practical implication: Use intervention-based tests to verify which features actually change the model output before you treat an explanation as decision evidence.

Counterfactuals and interventions are the test of faithfulness

A faithful counterfactual is a plausible alternative input that isolates one change at a time so you can observe whether the prediction shifts. In the article’s framing, changing zip code while holding income constant shows whether zip code is causal to the model’s output, and changing income while holding zip code constant shows whether income is not. That is the key distinction between an explanation that is merely human-readable and one that is behaviourally correct. Shapley-style approaches can approximate this logic, but correlated features and unrealistic counterfactuals make the exact set of allowed interventions a central design problem.

Practical implication: Define and test counterfactuals around the model’s real inputs and operational constraints, not around convenient data correlations.

Why observational explanations can fail legal and operational scrutiny

Observational explanations are attractive because they are easier to compute and easier to read, but they can hide the real basis for a decision. If a regulated workflow uses model output to support credit, access, or fraud decisions, an explanation that overstates the role of a correlated feature can be materially misleading. The article also points to the legal requirement for meaningful information about automated decision-making, which raises the bar beyond surface-level interpretability. Faithfulness therefore becomes a governance requirement, not just a modelling preference.

Practical implication: Align explanation methods with regulatory disclosure and audit expectations so the explanation can be defended as evidence, not just presented as narrative.

NHI Mgmt Group analysis

Faithful explanation is a governance control, not a model-surface nicety. The article makes clear that interpretability alone is insufficient when correlated features can produce persuasive but false stories about why a model decided something. For teams governing identity, fraud, or access decisions, the real risk is policy built on explanations that cannot survive challenge. Practitioners should treat explanation faithfulness as part of decision governance.

Correlation-heavy explanations create a trust gap between model output and operational action. When a risk score feeds onboarding, step-up authentication, or fraud review, the organisation is acting on a decision that may have been justified by the wrong feature. That creates a mismatch between what the model did and what the control layer believes it did. Practitioners should make that mismatch visible before it becomes an audit issue.

Meaningful automated-decision disclosure depends on interventions, not proxy narratives. The article’s GDPR example matters because legal defensibility depends on explaining logic in a way that reflects actual causal influence. In identity-adjacent workflows, proxy explanations can be especially dangerous because they hide bias or redlining patterns behind plausible correlations. Practitioners should require evidence that explanation methods survive feature intervention.

Faithfulness reveals where model governance must move from explanation to validation. The stronger pattern here is that teams need to validate explanations against model behaviour, not just generate a readable output. That shifts governance toward testing counterfactual stability, feature dependence, and decision traceability. Practitioners should embed explanation validation into model review rather than treating it as a post-hoc report.

Counterfactual realism is the named concept that separates useful explanations from decorative ones. If the alternative inputs are not realistic, the resulting explanation may be mathematically neat but operationally wrong. The article shows that correlated inputs and unrealistic interventions are the core failure modes. Practitioners should define counterfactual realism as a review criterion for any model used in regulated decisions.

What this signals

Faithful counterfactuals are the practical difference between model transparency and model governance. In identity-adjacent workflows, especially fraud and access decisions, teams should assume that any explanation built from correlated observations is provisional until it is tested against interventions.

Counterfactual realism: the usable alternative input set is the control boundary that determines whether an explanation is evidence or ornament. When that boundary is undefined, teams invite proxy-driven decisions into regulated workflows.

For practitioners

• Validate explanations with feature interventions Test whether the model output changes when one feature is altered while others remain fixed, then compare that result to the explanation method’s attribution.
• Define realistic counterfactual boundaries Document which alternative inputs are operationally valid for the model, especially where correlated features can generate unrealistic or misleading scenarios.
• Separate proxy features from causal features Review whether any high-importance feature is acting as a proxy for another attribute that the model actually uses, then record that distinction in governance evidence.
• Align explanation output with regulatory disclosure Check that the explanation can support meaningful information about automated decision-making, including when a model output affects access, fraud, or onboarding decisions.

Key takeaways

• Model explanations can be persuasive without being correct if they rely on correlation rather than causal intervention.
• Faithfulness matters because regulated decision workflows need explanations that reflect what the model actually used, not what the data happened to suggest.
• The practical control is to test counterfactuals, define realistic alternatives, and validate that the explanation survives feature-by-feature intervention.

Key terms

• Counterfactual Analysis: Counterfactual analysis asks what would have happened if a different control, event, or decision had been in place. Security teams use it to test whether a proposed change would actually reduce risk, rather than assuming a best practice will work in every environment.
• Faithfulness: The degree to which an answer is supported by the retrieved context. In practice, it measures whether the model stayed grounded in supplied evidence rather than inventing details, which makes it a critical signal for high-trust AI workflows and audit-sensitive applications.
• Correlated Features: Inputs that tend to move together in the data, making them difficult to separate cleanly in attribution analysis. Correlation can cause explanations to assign importance to variables that are predictive but not truly causal in the decision process.
• Observational Explanation: An observational explanation infers importance from patterns in existing data rather than from controlled intervention. It can be useful for exploration, but it becomes unreliable when the goal is to identify causal influence or provide legally meaningful information about a model’s logic.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and machine identity security alongside the governance discipline behind modern identity programmes. It is designed for practitioners who need to connect access, risk, and control decisions across human and non-human identity estates.