Shapley values expose the trade-off between fairness and compute

At a glance

What this is: This is an explainer on Shapley values for ML interpretability, with the key finding that feature attribution is principled but computationally expensive and highly sensitive to how “missing” inputs are modelled.

Why it matters: For IAM, identity verification, fraud, and AI governance teams, the lesson is that explanation quality depends on the assumptions behind the attribution method, not just on whether the output looks intuitive.

👉 Read Fiddler's explainer on what Shapley values are and how they support ML explainability

Context

Shapley values solve a fairness problem in machine learning explainability by assigning each input feature a contribution to the final prediction. In practice, that matters because explanations are only as trustworthy as the method used to generate them, especially when those explanations support regulated decisions, model oversight, or identity-related risk reviews.

The identity angle is indirect but real: when model outputs influence fraud screening, lending, account recovery, or identity verification workflows, the explanation layer becomes part of governance. Teams need to understand not only what the model predicted, but also whether the attribution method can withstand challenge under audit, dispute, or model risk review.

Key questions

Q: How should teams decide whether Shapley values are suitable for a model explanation workflow?

A: Use Shapley values when you need a principled, axiomatic attribution method and can define feature absence in a defensible way. They are most useful when explanation quality matters more than speed, but they become weaker when the counterfactual assumption is unstable or when the model has many correlated inputs.

Q: Why do Shapley explanations become unreliable in some tabular models?

A: Tabular models often require synthetic assumptions about what it means for a feature to be absent. If that baseline or sampling rule is poorly chosen, the resulting attribution reflects the assumption as much as the model, which can make the explanation misleading even when the prediction itself is accurate.

Q: What should practitioners measure before trusting a Shapley attribution result?

A: They should measure stability across samples, sensitivity to the missing-feature baseline, and the variance introduced by approximation. A single attribution score is not enough if the explanation is produced through sampling, because the output may shift materially between runs or input assumptions.

Q: How do governance teams evaluate model explanations for regulated decisions?

A: They should review whether the explanation method is traceable, repeatable, and aligned to the data type and decision context. For regulated workflows, the explanation process should be documented well enough that an auditor or reviewer can see how the attribution was generated and where its limits begin.

Technical breakdown

How Shapley values attribute prediction credit

Shapley values come from cooperative game theory, where each participant’s contribution is estimated across all possible orderings of participation. In ML, the “game” is the prediction and the “players” are features. The Shapley value for a feature is its average marginal contribution across permutations, which gives the method its fairness properties: efficiency, dummy, symmetry, and linearity. That is why it is attractive for explainability, especially when organisations need a defensible way to describe why a model produced a specific score or classification.

Practical implication: use Shapley-based attribution when you need a principled explanation layer, not when you need a cheap heuristic.

Why feature absence is the hardest modelling decision

Shapley values require comparing model outputs with and without a feature, but “absent” is straightforward only in some data types. In text, a word can be removed directly. In tabular models, absence is usually approximated by a baseline value, such as a mean or median, or by sampling from a distribution and averaging predictions. That choice is not neutral. It changes the counterfactual world the model is being asked to evaluate, which means the resulting attribution can shift materially depending on the method used to simulate missingness.

Practical implication: document the missing-feature assumption explicitly, because attribution outputs inherit that design choice.

Why computation and uncertainty limit practical use

The exact Shapley calculation grows factorially with the number of features because it considers every ordering, then is often reduced to an exponential number of model evaluations. Most real implementations therefore sample permutations or subsets, which introduces variance into the attribution result. That makes confidence intervals or other uncertainty measures important, especially when explanations are used in regulated or high-stakes settings. A point estimate without uncertainty can look precise while masking instability in the underlying explanation.

Practical implication: treat Shapley outputs as estimates with variance, not as immutable ground truth.

NHI Mgmt Group analysis

Shapley values remain one of the clearest examples of explainability done with formal discipline, but they are not a governance shortcut. The method is valuable because it ties attribution to explicit axioms rather than intuition. Yet the quality of the explanation still depends on the missing-feature model, which means governance teams should treat it as a structured estimate rather than a definitive truth statement.

The real control gap is not explanation availability, but explanation ambiguity. A model can produce a neat attribution chart while still relying on questionable assumptions about absent inputs, especially in tabular and identity-related decisioning. That is why explainability has to be assessed alongside model risk, not separated from it.

Feature attribution in regulated workflows needs uncertainty-aware oversight. If a method requires sampling to stay tractable, then the explanation layer itself becomes stochastic and should be reviewed like any other risk signal. For lending, fraud, and identity verification programmes, the practitioner conclusion is simple: confidence in the model must include confidence in the explanation method.

Shapley-based explainability is strongest when organisations can define the counterfactual boundary cleanly. Text and some categorical workflows fit that pattern better than complex tabular systems with correlated features and synthetic baselines. The more ambiguous the absent-feature assumption, the weaker the governance value of the attribution. Practitioners should align explainability methods to the data structure, not force one method across every model.

What this signals

Counterfactual clarity is becoming a governance requirement, not an academic preference. As more organisations use model explanations in customer-facing decisions, the missing-feature assumption becomes part of the control environment. If teams cannot explain how absence is simulated, they cannot reliably explain what the attribution means.

Explainability methods will increasingly be judged on operational repeatability as much as mathematical elegance. That means model risk functions need to test whether an attribution method can survive sampling variance, correlated inputs, and changing data distributions without producing unstable narratives.

For practitioners

• Define the counterfactual rule for missing features Document exactly how your team represents feature absence in each model class, including baseline substitution or sampling logic for tabular data. This becomes part of the model’s governance record and should be reviewed alongside validation results.
• Record uncertainty around attribution outputs Require confidence intervals, sampling variance, or stability checks for any Shapley-based explanation used in a decision workflow. If attribution changes materially across runs, the explanation should not be treated as audit-grade evidence.
• Match the explainability method to the data structure Use direct feature removal only where absence is meaningful and observable, such as text inputs. For correlated tabular data, assess whether the chosen distribution for absent features distorts the explanation.
• Separate explanation review from model approval Add a specific governance checkpoint for explanation methodology, not just prediction accuracy. This helps reviewers test whether the attribution approach is defensible under challenge, especially in regulated identity or lending decisions.

Key takeaways

• Shapley values provide a formal and fair framework for attributing model predictions, but the explanation is only as good as the counterfactual assumption behind it.
• In practical deployments, the biggest constraint is compute, because exact attribution scales poorly and approximate methods introduce uncertainty.
• Teams using explainability in regulated or identity-related workflows should govern the explanation method itself, not just the model it explains.

Key terms

• Shapley Value: A Shapley value is a way of fairly assigning credit to each participant in a cooperative outcome. In machine learning, it measures how much each feature contributes to a prediction by averaging that feature’s marginal effect across many possible feature combinations.
• Counterfactual Feature Absence: Counterfactual feature absence is the method used to estimate what a model would predict if one input were missing. In practice, teams simulate absence by removing text, substituting a baseline, or sampling from a distribution, and each choice can change the explanation materially.
• Attribution Variance: Attribution variance is the degree to which an explanation output changes across repeated runs or different approximation samples. It matters because sampled Shapley methods produce estimates, not exact values, and high variance can make a model explanation look more certain than it really is.

What's in the full article

Fiddler's full blog covers the implementation detail this post intentionally leaves for the source:

• Worked examples of how Shapley marginal contributions are computed across permutations
• Further detail on baseline substitution versus sampling-based feature absence for tabular models
• Discussion of computational shortcuts and why approximation introduces uncertainty
• The preprint guidance referenced by the author on choosing distributions for absent features

👉 Fiddler's full post covers the axioms, algorithm steps, and missing-feature challenges in more detail

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and identity lifecycle controls. It is designed for practitioners who need to connect governance decisions to broader security programmes.