Why do Shapley explanations become unreliable in some tabular models?

Why This Matters for Security Teams

Shapley explanations are appealing because they look mathematically fair, but in tabular settings the explanation can become fragile when feature absence is defined through a synthetic baseline rather than a real-world state. That matters because security, fraud, and identity teams often use explanations to justify triage, model governance, or analyst review. If the baseline is unrealistic, the attribution can suggest a feature is influential for the wrong reason, which weakens trust in both the model and the operating process.

For practitioners, the key issue is not whether the prediction is correct, but whether the explanation is stable under the way the data was generated and preprocessed. Correlated features, heavy imputation, and encoded categories can all distort shapley value. Guidance from NIST SP 800-53 Rev 5 Security and Privacy Controls reinforces the broader control expectation: model decisions and supporting evidence should be governed, traceable, and reviewable, not treated as opaque outputs. In practice, many security teams encounter explanation failures only after analysts notice that the model’s reasoning changes with preprocessing choices rather than through intentional explainability validation.

How It Works in Practice

Shapley methods estimate each feature’s contribution by comparing model output across many feature coalitions. That works best when missingness has a defensible meaning. In tabular models, however, “feature absent” is usually simulated by replacing a value with a baseline, averaging over background rows, or sampling from a reference distribution. Each choice creates a different counterfactual world, so the attribution is only as reliable as the rule used to construct that world.

The practical failure modes usually appear in four places:

• When features are strongly correlated, credit can be split in ways that do not match operational reality.
• When one-hot encoding or feature engineering changes the data shape, the explanation may reflect the encoding artifact rather than the original business variable.
• When missing values are common, imputation can blur whether the model is reacting to the feature or the missingness pattern itself.
• When the background dataset is too narrow, the explanation can overfit to a single population slice and lose generality.

Good practice is to validate explanations against known cases, compare multiple baselines, and document the reference distribution used for the attribution run. The NIST AI Risk Management Framework is useful here because it treats transparency, validity, and monitoring as governance requirements rather than optional reporting. Teams that need a threat-centric lens can also compare their model risk assumptions to MITRE ATLAS, especially where adversarial manipulation or data poisoning could shift feature relationships. These controls tend to break down when the production population differs materially from the reference data because the explanation is then anchored to a counterfactual that no longer resembles actual use.

Common Variations and Edge Cases

Tighter explanation controls often increase development and review overhead, requiring organisations to balance interpretability against modelling speed and predictive lift. That tradeoff becomes most visible in high-dimensional tabular systems, where exact Shapley computation is expensive and approximate methods are commonly used. Current guidance suggests treating approximate explanations as decision support, not as authoritative proof of causality.

There are a few edge cases worth calling out. In regulated workflows, explanations may need to support auditability, but the model can still be statistically valid even if the attribution is not business-intuitive. In highly correlated data, feature grouping or conditional Shapley approaches can help, yet there is no universal standard for the “correct” grouping strategy. In fraud, KYC, or access-risk scoring, the model may also rely on proxy signals that are acceptable for prediction but problematic for explanation, especially if those proxies map to sensitive attributes or operational artefacts.

For teams using tabular models in identity or security workflows, the safest stance is to separate three questions: does the model predict well, is the explanation stable, and is the explanation meaningful to a reviewer? When those answers diverge, the explanation should be flagged as informative but not relied on for policy decisions. The NIST SP 800-63 Digital Identity Guidelines are a useful reminder that assurance depends on evidence quality, not just a confident output. OWASP guidance on secure design thinking also aligns with this approach: validate the control boundary, then validate the explanation layer separately.

Related resources from NHI Mgmt Group

• When does regex-based secret detection become too unreliable for production use?
• Why does SoD become harder in customised ERP role models?
• Why do fine-grained authorization models become hard to govern at scale?
• Why do SCIM integrations become unreliable at enterprise scale?