What breaks when Integrated Gradients is applied to non-differentiable models?

Why This Matters for Security Teams

integrated gradients is often treated as a general-purpose explanation method, but its assumptions matter. When a model cannot expose stable gradients, the method loses the mathematical basis that makes the attribution useful. That creates a governance problem as much as a technical one: an explanation layer may look rigorous while actually producing artifacts that are undefined, noisy, or misleading for the model class in use. For security teams, that matters when explainability is tied to model assurance, audit evidence, model risk review, or incident investigation.

In practice, the risk is not just that an explanation is unavailable. It is that teams may continue using a method outside its valid operating range and then rely on the output for decisions about safety, bias, abuse detection, or control effectiveness. Current guidance from NIST SP 800-53 Rev 5 Security and Privacy Controls reinforces the broader expectation that controls must be suitable for the system they govern, not just technically present. In practice, many security teams discover this mismatch only after a review, model challenge, or production incident has already exposed the explanation gap rather than through intentional validation.

How It Works in Practice

Integrated Gradients attributes a model output by accumulating gradients along a path from a baseline input to the real input. That works when the model is differentiable enough for gradients to exist and be informative. For non-differentiable models, such as tree ensembles, rule engines, and other discrete decision systems, the gradient is either unavailable or unhelpful because small input changes do not map to smooth output changes. The result is that the path integral loses meaning, and the attribution no longer reflects the model’s actual decision process.

Operationally, teams usually respond in one of three ways: they switch to an explanation method designed for that model family, they wrap the model in a differentiable surrogate for local analysis, or they accept a different assurance question entirely and focus on feature importance, rule tracing, or case-level reasoning. For AI governance, the question is not whether an explanation exists, but whether it is faithful enough for the decision being defended. This aligns with the broader assurance logic in the NIST AI Risk Management Framework, which emphasises valid measurement, traceability, and context-aware evaluation.

• Use Integrated Gradients only when the target model exposes meaningful gradients.
• For tree-based or rule-based systems, prefer native explanation methods or model-specific attribution tools.
• Validate whether the explanation is local, global, or merely approximate before presenting it to reviewers.
• Document the model class, explanation method, and known limitations in governance evidence.

Where this guidance breaks down is in hybrid pipelines that combine non-differentiable preprocessing, retrieval logic, and a differentiable downstream model, because attribution can become fragmented across components and no single method explains the full path.

Common Variations and Edge Cases

Tighter explainability requirements often increase operational overhead, requiring organisations to balance faithful attribution against deployment speed and model complexity. The common edge case is a mixed architecture where only part of the system is differentiable. In those environments, current guidance suggests treating the explanation problem as component-specific rather than assuming one method can cover the whole pipeline. That is especially important in AI security workflows where retrieval, post-processing, and guardrail logic can influence the final answer as much as the core model.

There is no universal standard for this yet, but best practice is evolving toward model-specific explainability and stronger provenance controls. For adversarial testing and failure analysis, the MITRE ATLAS framework is useful when the concern is how the model can be manipulated, while OWASP Top 10 for Large Language Model Applications helps teams think through explanation misuse in agentic or LLM-adjacent systems. The main edge case is governance pressure: teams sometimes demand a single explanation technique across all models for standardisation, but that can reduce fidelity and increase false confidence.

For that reason, explanation strategy should follow model architecture, not organisational convenience. If the model is non-differentiable, the safer assumption is that Integrated Gradients is not the right control unless a specialised adaptation has been validated for that exact environment.

Related resources from NHI Mgmt Group

• What breaks when access review models are applied to agentic AI?
• Why do non-human identities break legacy governance models?
• What breaks when organisations cannot see their non-human identities?
• What breaks when non-human identities are left out of governance?