Counterfactual and attribution explanations sharpen AI model governance

At a glance

What this is: This is a deep dive on two explainability methods in AI, showing that counterfactual and attribution explanations are complementary but solve different governance problems.

Why it matters: It matters because AI, IAM, and risk teams need explanations that support auditability, recourse, and decision accountability without mistaking one explanation style for complete model assurance.

👉 Read Fiddler's analysis of counterfactual and attribution explanations in AI

Context

AI explainability fails when teams treat one method as enough to justify a model decision. Counterfactual explanations focus on what would need to change for a different outcome, while attribution methods show which inputs influenced the current prediction. In high-stakes workflows such as lending, hiring, or healthcare, that distinction affects both governance and how reviewers interpret model behaviour.

For identity-adjacent programmes, the relevance is less about model transparency in the abstract and more about decision accountability. If AI systems are influencing identity verification, fraud scoring, access decisions, or case triage, teams need to know whether an explanation supports recourse, investigation, or regulatory review. The article’s starting point is typical for organisations trying to operationalise explainability without overclaiming what a single method can prove.

Key questions

Q: How should teams decide between counterfactual and attribution explanations?

A: Use counterfactual explanations when the business need is recourse, appeal handling, or showing what would need to change for a different outcome. Use attribution when the need is model inspection, case review, or understanding which inputs most influenced the current prediction. Most programmes need both, because they answer different governance questions and support different review workflows.

Q: Why do attribution explanations change when the baseline changes?

A: Attribution methods compare the observed input to a reference point, so the baseline is part of the explanation itself, not just an implementation detail. If the baseline shifts, the marginal contributions can shift as well. That is why teams should document baselines, test sensitivity, and avoid presenting attribution output as if it were an objective fact about causality.

Q: What breaks when counterfactual explanations ignore feasibility?

A: They can recommend changes that are mathematically valid but impossible, irrelevant, or policy-violating in the real world. That is common when the explanation ignores causal dependencies, immutable attributes, or downstream effects on related features. In practice, infeasible recourse weakens user trust and creates governance risk because the advice cannot be safely acted on.

Q: How do security and risk teams operationalise AI explanations in governance?

A: Treat explanations as a control input, not a control outcome. Counterfactuals can support appeals and remediation, while attributions can support monitoring and model review. Teams should require reproducibility, baseline documentation, and a clear mapping between explanation type and decision workflow before relying on outputs in production.

Technical breakdown

Counterfactual explanations and recourse

Counterfactual explanations ask what input change would produce a different prediction. They are useful when a user or reviewer needs a practical path to a more favourable outcome, but they depend on defining a valid distance metric, solving an optimisation problem, and ensuring the suggested change is feasible in the real world. That last constraint is often missed: a mathematically minimal change may be operationally impossible or may trigger other feature changes through causal dependencies.

Practical implication: teams should validate whether a counterfactual is actionable before using it in user-facing appeals or adverse-decision workflows.

Attribution methods and feature contribution

Attribution-based explanations estimate how much each feature contributed to the current prediction. Methods such as SHAP, LIME, and Integrated Gradients do this by comparing the observed input to counterfactual baselines, but the choice of baseline can change the result materially. Attribution is therefore an explanation of influence, not a proof of causality, and small perturbations can change the explanation even when the prediction remains stable.

Practical implication: treat attributions as one review signal, and test whether the explanation remains stable across baseline and perturbation choices.

Why the two methods do not substitute for each other

Counterfactual and attribution explanations answer different questions. Attribution shows why the model produced the current result, while counterfactuals show how to obtain a different one. That means a feature can be important for recourse without dominating attribution, and a highly attributed feature may be unusable for recourse because it is immutable or interacts with other inputs. Good governance uses both because they illuminate different parts of the model-data relationship.

Practical implication: define which explanation type supports appeals, which supports model review, and which supports regulatory documentation.

NHI Mgmt Group analysis

Explainability is a governance control, not a trust guarantee. Counterfactual and attribution methods improve visibility, but neither can certify that a model decision is fair, stable, or lawful. That matters for identity-adjacent AI use cases where the decision itself can affect access, onboarding, or fraud outcomes. Practitioners should treat explanation outputs as evidence to review, not as evidence that the model is safe to deploy.

Counterfactual recourse introduces a distinct accountability problem. If a model can tell a user what to change, the organisation must be able to defend whether that change is feasible, lawful, and consistent with policy. In identity verification and fraud workflows, recourse suggestions can become misleading if they ignore immutable attributes or hidden causal links. Teams need governance that separates actionable explanation from automated advice.

Attribution sensitivity creates a verification trust gap. When the same input yields materially different attributions depending on baselines or sampling choices, the explanation layer itself becomes a source of uncertainty. That does not make the methods useless, but it does mean reviewers need reproducibility checks and clear documentation of how explanations are generated. The practitioner conclusion is straightforward: if the explanation cannot be reproduced, it cannot be relied on for assurance.

Explanation design should match the decision context. In lending, hiring, access, or identity decisions, the question is not whether an explanation exists but whether it supports the right control objective. Counterfactuals are better for recourse and appeals, while attribution is better for model inspection and drift review. Teams should align explainability with the decision lifecycle instead of treating all explanations as interchangeable.

What this signals

Identity-adjacent AI programmes should expect explanation controls to be scrutinised alongside the decision itself. If a model influences onboarding, fraud triage, or access approval, explainability needs to support appealability and reviewer confidence, not just technical transparency.

Explanation trust gap: when baseline choice and feasibility checks are not documented, the explanation layer becomes another governance dependency. Teams should align explainability evidence with reviewable controls such as model monitoring, human oversight, and case audit trails.

Practitioners should also plan for explanation drift. As models, feature sets, and data distributions change, the explanation that was acceptable at launch can become unstable or misleading, so explanation testing needs to sit inside the wider model risk programme.

For practitioners

• Define the explanation purpose before deployment Separate user recourse, investigator review, and regulator-facing documentation into different explanation requirements so the same output is not forced to do all three jobs.
• Test counterfactual feasibility against policy Validate that any suggested change is operationally possible, legally acceptable, and consistent with the underlying causal structure of the data before exposing it to users.
• Stabilise attribution baselines Document the baseline dataset or reference point used for attribution and test whether the ranking of important features changes under alternative baselines.
• Link explanations to review workflows Use counterfactuals to support appeals or remediation paths, and use attributions to support model monitoring, case review, and exception analysis.

Key takeaways

• Counterfactual and attribution explanations solve different governance problems, so teams should not treat them as interchangeable.
• Explanation outputs can mislead when feasibility, causal dependency, or baseline choice are not controlled and documented.
• AI governance is stronger when explainability is tied to appeals, monitoring, and review workflows rather than used as standalone assurance.

Key terms

• Counterfactual Analysis: Counterfactual analysis asks what would have happened if a different control, event, or decision had been in place. Security teams use it to test whether a proposed change would actually reduce risk, rather than assuming a best practice will work in every environment.
• Attribution: Attribution is the ability to prove which identity performed a specific action, through which systems, and with what outcome. In agentic environments, attribution must span APIs, services, and workflow steps, because isolated logs are rarely enough to reconstruct decision chains or support audit and incident response.
• Baseline: A baseline is the approved configuration state that security, compliance, and operations teams use as the reference point for control. In practice, it only works when it stays current, is owned, and is tied to monitoring so changes can be detected before they become exposure.
• Recourse: Recourse is a path a person or system can take to change an undesirable decision outcome. In explainable AI, good recourse must be actionable, feasible, and consistent with policy, otherwise it becomes a theoretical suggestion rather than a usable governance control.

What's in the full article

Fiddler's full blog post covers the detailed mechanics this analysis intentionally leaves for the source:

• Worked examples of counterfactual and attribution outputs across specific model types and feature sets
• Detailed discussion of optimisation, baselines, and sensitivity issues that shape explanation quality
• The original comparisons the authors use to show why the two explanation styles diverge in practice

👉 Fiddler's full post covers the conceptual comparisons and examples that support implementation decisions.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, human identity, secrets management, and workload identity. It helps practitioners connect identity controls to wider security and governance programmes.