What do financial services teams get wrong about SHAP and LIME?

Why Financial Services Teams Misread SHAP and LIME

SHAP and LIME are often used as if they provide a full explanation of why a model made a decision, but in practice they are local approximation tools. That matters in financial services because model risk, conduct risk, and auditability are not solved by a chart that looks understandable. The more automated the workflow becomes, the more teams need to separate “useful evidence” from “governance-grade assurance.” NIST’s NIST SP 800-63 Digital Identity Guidelines are identity-focused, but the same principle applies here: assurance has to match the trust decision, not the convenience of the output.

The common mistake is assuming an explanation method can validate fairness, stability, leakage, or causal behavior. It cannot. SHAP and LIME can help identify feature influence on structured models, but they do not prove why the model is right, how it will behave after drift, or whether a downstream workflow is safe under operational pressure. In financial services, that gap becomes acute when a model is embedded in underwriting, AML triage, or fraud operations, where small errors can compound into customer harm or regulatory scrutiny. NHI Management Group has seen this pattern before in identity-driven compromises, including the Zacks Investment Research breach, where control gaps were operational, not just analytical. In practice, many security teams discover the limits of explainability only after a production issue, rather than through deliberate model governance.

How SHAP and LIME Should Be Used in Practice

The right way to use SHAP and LIME is as part of a broader validation stack, not as the stack itself. For tabular models, they can help analysts inspect whether the expected variables are driving outcomes and whether obvious anomalies exist. For example, if a lending model appears to overweight a proxy variable, local explanations can trigger review. That is useful. It is not sufficient. Current guidance suggests pairing explainability with model documentation, drift monitoring, human review thresholds, and controls that map to NIST SP 800-63 Digital Identity Guidelines only where identity assurance is part of the workflow.

Practitioners should think in terms of control layers:

• Use SHAP or LIME to detect suspicious feature reliance, not to certify correctness.
• Test stability across samples, time windows, and input perturbations.
• Separate explanation for analysts from evidence required for audit and compliance.
• Document where the method is mathematically valid and where it is only heuristic.

This matters even more when the model sits beside NHI-controlled services, because secrets, service accounts, and automated approvals can turn a small model defect into a broader operational failure. NHI Management Group data shows that 97% of NHIs carry excessive privileges, and that over-privileged automation expands the blast radius when model outputs drive actions. The Zacks Investment Research breach is a reminder that identity and access failures often become security incidents before the analytics layer is fully understood. These controls tend to break down when explanation tools are treated as a proxy for governance in high-volume, fast-changing decision pipelines.

Where the Edge Cases and Exceptions Bite Hardest

Tighter explainability review often increases model-operations overhead, so financial institutions have to balance speed against confidence. That tradeoff is especially sharp in regulated environments where a model is updated frequently or used across multiple lines of business. There is no universal standard for treating SHAP or LIME output as admissible evidence, so current guidance is to define the decision context up front and avoid overclaiming what the method can prove. The NIST SP 800-63 Digital Identity Guidelines and NIST AI governance practices both point toward fit-for-purpose assurance rather than one-size-fits-all proof.

Edge cases include ensemble models, highly correlated features, non-stationary data, and LLM-adjacent workflows where outputs are mediated by prompts, tools, and policy layers. In those settings, local feature attribution can be misleading because the system is not making a single static prediction. The answer is not to abandon SHAP or LIME, but to position them correctly: useful for diagnostics, weak for causal claims, and insufficient for autonomous or semi-autonomous processes without additional controls. That is why NHI Management Group recommends pairing them with identity-bound execution controls, especially where service accounts, API keys, or delegated access determine what an automated workflow can actually do. When model behaviour, identity, and access decisions are coupled, explanation artifacts alone cannot prevent misuse; they only help reconstruct it after the fact.

Related resources from NHI Mgmt Group

• What do security teams get wrong about coarse consent for AI agents?
• What do security teams get wrong about WebSocket-based local services?
• How should teams think about AI agent privileges?
• What do security teams get wrong about session tokens and MFA?