Shapley values expose the trade-off between fairness and compute

At a glance

What this is: Fiddler’s explainer shows Shapley values as a principled way to attribute model predictions to features, while highlighting the practical trade-off between fairness, feature-absence assumptions, and compute cost.

Why it matters: This matters because teams using explainability for lending, adverse action, or model governance need attribution methods that are defensible, but also tractable and honest about uncertainty.

Context

Shapley values sit at the intersection of explainability, model governance, and decision accountability. In plain terms, they try to assign each feature a fair share of credit or blame for a model output, which makes them attractive when teams need to explain decisions to users, reviewers, or regulators. The hard part is that the method requires a way to imagine a feature as absent, and that assumption is not always straightforward in real models.

For machine learning programmes, the governance issue is not whether an attribution method is mathematically elegant, but whether its assumptions remain stable across data types and use cases. In lending and other high-stakes workflows, feature attribution has to support defensible explanations without hiding approximation choices or sampling uncertainty. That is why Shapley values are useful, but also easy to overstate if teams treat them as exact truth.

The article frames a common ML explainability trade-off that is typical in practice: stronger theoretical fairness often comes with heavier computational and modelling overhead.

Key questions

Q: How should teams govern Shapley-based explanations in high-stakes ML decisions?

A: Treat the explanation method as part of model governance, not a post-processing step. Teams should define how feature absence is simulated, who approves the baseline or sampling strategy, and what evidence is needed before the output can support lending decisions, adverse action notices, or other regulated use cases.

Q: Why can Shapley values produce misleading explanations if the baseline choice is weak?

A: Because the attribution depends on the counterfactual world created when a feature is treated as absent. If the baseline or sampling distribution does not reflect the model’s real operating context, the resulting contribution scores can look precise while reflecting an artificial scenario rather than the actual decision logic.

Q: What are the signs that sampled attribution scores are too uncertain to trust?

A: Watch for wide confidence intervals, unstable rankings across repeated runs, and large shifts in attribution when the sampling design changes. Those are signs that the explanation is more sensitive to approximation choices than to the model’s actual behaviour, which weakens its use in governance or audit.

Q: Should organisations use Shapley values or simpler feature importance methods for explainability?

A: Use Shapley values when the decision is high stakes and you need a fair, axiomatic attribution method that can stand up to scrutiny. Use simpler methods when the business question is directional insight rather than a defensible explanation for human review, and the extra compute cost would not change the decision.

Technical breakdown

How Shapley values allocate model attribution

Shapley values come from cooperative game theory, where each participant contributes to a shared outcome. In machine learning, the model prediction is treated like the total payoff and each feature is treated like a player. The method estimates the average marginal contribution of each feature across all possible orderings, which is why it is often described as fair and axiomatically unique. The key appeal is that the result is grounded in formal properties such as efficiency, dummy, symmetry, and linearity. That makes it attractive when explainability needs to be more than a heuristic ranking of feature importance.

Practical implication: Use Shapley values when you need explanation logic that can be defended against formal fairness criteria, not just intuitive feature ranking.

Why feature absence is the hardest modelling choice

Shapley-style attribution depends on comparing the model output with and without a feature, but many real-world features cannot simply be deleted. For tabular data, teams often replace the missing feature with a baseline value such as a mean or median, or they sample values from a distribution and average the predicted output. That choice is not a technical footnote. It changes the counterfactual world the model is being asked to evaluate, which can shift the attribution materially. In practice, the method is only as credible as the missingness assumption behind it, especially when the model input is highly correlated or context-dependent.

Practical implication: Document the missingness strategy explicitly, because the attribution is only as reliable as the counterfactual used to simulate absence.

Why compute cost and sampling uncertainty matter

The brute-force form of Shapley calculation scales poorly because it considers many orderings of features. Even after algebraic shortcuts, the problem can still require 2^n model invocations, which becomes expensive quickly as feature count grows. Most operational implementations therefore rely on sampling to make the calculation tractable. That reduces cost, but it introduces variance. Once sampling is part of the method, attribution should be treated as an estimate with uncertainty, not a fixed score. Confidence intervals or similar uncertainty bounds are therefore part of responsible use, especially in regulated decision workflows.

Practical implication: Treat sampled attributions as estimates with uncertainty bounds, not as exact explanations ready for audit without qualification.

NHI Mgmt Group analysis

Shapley values are best understood as governance tools, not just explanation tools. Their value is that they force teams to specify how model contributions are defined and how counterfactuals are constructed. That makes them useful for lending and other high-stakes decisions, but only if the organisation is willing to govern the assumptions behind the attribution, not merely publish the output. The practitioner takeaway is to treat explainability as a controlled method, not a decorative layer.

The real risk is not that Shapley values are unfair in theory, but that teams misread approximations as ground truth. Baseline choice, sampling strategy, and feature dependence all shape the result. When those choices are undocumented, the explanation can look precise while remaining fragile. The practitioner conclusion is that model governance must cover the attribution method itself, not only the underlying model.

Feature attribution in regulated ML creates an accountability boundary that many teams still under-specify. If an adverse action notice or internal review relies on Shapley values, the organisation needs a defensible story for why the chosen counterfactual reflects the real decision context. That is especially important when the explanation is expected to support human review. The practitioner conclusion is that explainability evidence should be treated as part of the control environment.

Shapley values expose explainability debt when teams scale model governance faster than computational design. A method that is elegant in principle can become operationally brittle if sampling, uncertainty, and compute constraints are not built into the workflow from the start. That makes this a programme design issue, not just a data science choice. The practitioner conclusion is to align explanation depth with the decision risk that the model actually carries.

What this signals

Explainability governance needs to move from score generation to assumption management. Shapley values make that shift visible because the attribution only exists after the team chooses a counterfactual for feature absence. For practitioners, the practical question is whether those assumptions are versioned, reviewable, and aligned to the model’s actual decision context.

Shapley-based attribution becomes fragile when organisations scale model use faster than their control design. Compute constraints, sampling variance, and correlated features all create room for inconsistent explanations across teams or environments. The signal for programme owners is that explainability controls should be tested like any other governance control, not trusted because the method is mathematically elegant.

For practitioners

• Define the counterfactual missingness policy Specify how the organisation simulates feature absence for each model class, including baseline replacement, sampling distribution, and any feature dependency handling. Keep the policy versioned so explanations remain reproducible across model changes.
• Require uncertainty bounds for sampled attributions When sampling is used to approximate Shapley values, publish confidence intervals or an equivalent uncertainty measure alongside the attribution scores. This prevents stakeholders from treating an estimate as a deterministic explanation.
• Document explanation assumptions in model governance Record the feature absence method, sampling design, and limitations in the model card or governance pack so reviewers can assess whether the explanation is fit for lending, review, or adverse action use.
• Match attribution depth to decision risk Use the full Shapley approach where decisions are high impact and explanation quality matters, but avoid overcommitting the method to low-risk use cases where simpler feature attribution is sufficient.

Key takeaways

• Shapley values give machine learning teams a defensible way to attribute prediction outcomes, but the method only works if the organisation can justify how it simulates missing features.
• The practical limits are computational cost and approximation uncertainty, which means sampled results need to be treated as estimates rather than exact explanations.
• For regulated use cases, the control question is not whether the model can be explained, but whether the explanation method is documented, reproducible, and fit for the decision at hand.

Key terms

• Shapley Value: A Shapley value is a way of fairly assigning credit to each participant in a cooperative outcome. In machine learning, it measures how much each feature contributes to a prediction by averaging that feature’s marginal effect across many possible feature combinations.
• Counterfactual Analysis: Counterfactual analysis asks what would have happened if a different control, event, or decision had been in place. Security teams use it to test whether a proposed change would actually reduce risk, rather than assuming a best practice will work in every environment.
• Marginal Contribution: Marginal contribution is the amount a feature adds to a model output when it is introduced into a partial input set. Shapley methods average that contribution across many feature orderings to estimate each feature’s share of the final prediction.
• Sampling Uncertainty: Sampling uncertainty is the variation introduced when an expensive calculation is approximated using sampled trials instead of exhaustive evaluation. In Shapley-style explainability, it affects how stable and trustworthy the resulting attribution scores are and should be reported alongside the estimates.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore nhimg.org for resources that connect identity governance to the broader security disciplines your programme depends on.