What breaks when counterfactual explanations ignore feasibility?

They can recommend changes that are mathematically valid but impossible, irrelevant, or policy-violating in the real world. That is common when the explanation ignores causal dependencies, immutable attributes, or downstream effects on related features. In practice, infeasible recourse weakens user trust and creates governance risk because the advice cannot be safely acted on.

Why This Matters for Security Teams

Feasibility is the difference between a useful counterfactual and a compliance-shaped suggestion that cannot be acted on. When an explanation tells a person to change an immutable attribute, violate policy, or reverse a dependency that the model never represented, the result is not just poor UX. It can create audit noise, mislead decision-makers, and hide model weakness behind polished language. For security and risk teams, that is a governance problem as much as a model-quality problem.

Current guidance suggests counterfactual explanations should be evaluated for actionability, consistency with domain rules, and alignment with the system’s actual decision logic. If those checks are absent, teams may wrongly assume the model is transparent when it is only producing plausible-sounding advice. That matters in fraud, identity, lending, access decisions, and any workflow where a user may rely on the explanation to take corrective action. It also intersects with broader AI security concerns because explanation layers can be manipulated to appear reasonable while masking brittle or biased behaviour, a pattern that overlaps with threat analysis in the MITRE ATLAS adversarial AI threat matrix.

In practice, many security teams encounter infeasible recourse only after users attempt to follow it and the organisation has already absorbed the operational and trust damage.

How It Works in Practice

A counterfactual explanation is meant to show the smallest change that would alter a model output. That sounds simple, but feasibility depends on constraints outside the model itself. A person cannot change age, a transaction cannot be retroactively unlogged, and an access decision may depend on upstream identity signals that the user cannot directly control. Good implementations therefore test candidate counterfactuals against causal and policy constraints before they are shown.

In practice, this means explanation pipelines should validate more than mathematical distance. They need to check whether a proposed change is:

• causally possible, meaning it does not break feature dependencies;
• policy-compliant, meaning it does not instruct users to alter protected or immutable attributes;
• operationally relevant, meaning it maps to actions the person or system can actually take;
• stable, meaning a small change in one feature does not create contradictory advice elsewhere.

For AI systems in sensitive environments, that validation should sit alongside governance controls rather than as an afterthought. NIST’s AI risk guidance frames this as part of managing validity, reliability, and harmful output risk, while the control discipline in NIST SP 800-53 Rev 5 Security and Privacy Controls helps translate the requirement into review, logging, and change-management expectations. If the model supports high-impact decisions, the explanation layer should also be tested against realistic attacker and user behaviour, not only benchmark metrics. Public advisories such as CISA cyber threat advisories are useful for tracking how misleading system behaviour can be abused in real operations.

Teams should also look for lifecycle controls: dataset review, feature-constraint documentation, explanation QA, and human review for any actionability claims. These controls tend to break down when explanations are generated on top of sparse, heavily engineered, or policy-constrained decision systems because the recourse engine cannot distinguish between what is optimizable and what is merely correlated.

Common Variations and Edge Cases

Tighter feasibility checks often increase implementation complexity and review overhead, requiring organisations to balance user clarity against model and policy constraints.

There is no universal standard for feasibility scoring yet, so implementations vary. Some teams use causal graphs, some use hard business rules, and others rely on post-generation filtering. Best practice is evolving, and each approach has tradeoffs. Hard filters reduce unsafe suggestions but may leave users with few or no actionable counterfactuals. Softer approaches can preserve usefulness, but they can also allow borderline advice that looks plausible while still being impossible to execute.

Edge cases are common in identity, fraud, and access contexts. A model might suggest that a user should “increase account age,” “provide a different national identifier,” or “move to a lower-risk device profile” when those are not realistic or lawful actions. In agentic or automated workflows, infeasible recourse is even riskier because the system may try to act on its own explanation without understanding downstream dependencies. That is where explanation quality becomes a control issue, not just a documentation issue. The same caution applies when analysing threats that exploit AI-mediated decisions, a topic covered in the Anthropic — first AI-orchestrated cyber espionage campaign report.

The practical rule is simple: if the counterfactual cannot be acted on by the relevant person, system, or process, it should be treated as a model output defect. In high-stakes environments, that defect should be logged, reviewed, and remediated rather than presented as guidance.

Related resources from NHI Mgmt Group

• What breaks when automation teams ignore access governance for AI workflows?
• What breaks when organisations ignore session security after MFA?
• What breaks when AI teams only validate models and ignore the data plane?
• What breaks when access reviews ignore inherited permissions on file shares?