How should organisations govern AI systems that are explainable but still unfair?

Why explainability does not settle fairness

Explainability helps people understand how a system arrived at a result, but it does not prove the result is acceptable, lawful, or equitable. A model can be transparent about the factors it used and still produce systematically worse outcomes for some groups if the training data, labels, thresholds, or business objective are biased. Governance should therefore separate “can we explain it?” from “should we trust it to decide?”

That distinction matters most when the model affects access, opportunity, eligibility, pricing, prioritisation, or any other decision with human impact. In those settings, an explanation is only useful if it supports review of whether the model is using appropriate signals and whether the outcome is defensible in context. The governance question is not just interpretability, but whether the explanation reveals a decision process that is sound, proportionate, and challengeable.

What organisations need to review together

Fair governance tests the data, the use case, and the decision process as one system. Data review looks for skewed representation, proxy variables, stale labels, and historic bias that may be preserved even in a fully explainable model. Use case review asks whether the model is being applied in a setting where the stakes, populations, or legal duties require tighter controls or human review. Decision-process review checks whether the model’s output is advisory, decisive, or merely one input into a broader assessment.

This combined review should also ask whether the explanation is actually understandable to the people who must rely on it. A technically precise rationale may still be too opaque for reviewers, appeal handlers, or impacted individuals. For that reason, governance should require not just model-level interpretability, but decision-level traceability, clear ownership, and a documented path for escalation when an output appears inconsistent, discriminatory, or out of policy. NHIMG’s AI Impact Assessment Template is useful here because it ties purpose, affected people, bias review, and sign-off into one review artifact.

How to make fairness governable in practice

Fairness becomes governable when organisations define what acceptable performance looks like before deployment, then keep checking it after deployment. That means setting review criteria, testing for disparate impact where relevant, and requiring human accountability for decisions that materially affect people. It also means preserving the right to challenge outputs, correct bad inputs, and pause use when the model drifts or produces patterns that the business cannot justify.

In mature governance, explainability is one evidence source among several, not the final arbiter. Teams should compare model explanations with observed outcomes, exception handling, and complaint patterns. Where the explanation points to a feature or rule that creates unacceptable inequity, the remedy may be retraining, feature removal, threshold changes, or narrowing the use case rather than simply adding more explanation. NHIMG’s AI Governance Guide is a useful companion for the operating model around committees, approvals, oversight, and decommissioning, and the ISO/IEC 42001 Guide helps frame the management-system discipline behind those controls.

Risk and Threat Considerations

Explainable but unfair systems create a governance risk because they can look defensible while still producing harmful or discriminatory outcomes. The failure mode is subtle: reviewers may trust the explanation, miss the bias in the data or objective, and allow the model to scale the same mistake across many decisions.

Failure mechanism: The model’s rationale is visible, but the underlying training set, proxies, thresholds, or policy assumptions still encode bias, so the system remains unfair even when it is easy to interpret.

Impact: Organisations can amplify unequal treatment, expose themselves to complaints or regulatory scrutiny, and lose the ability to defend decisions that materially affect individuals.

Practitioner Guidance

What to prioritise: Put fairness review on the same approval path as explainability review. If a model is intended for high-impact decisions, require documented evidence that the data, objective, and decision workflow were assessed together before production use.

What to verify: Confirm that there is a named owner for appeals, overrides, and remediation, and that reviewers can see which inputs drove the output, which assumptions were accepted, and when the model must be suspended. If those elements are missing, the organisation has transparency without effective governance.

Practitioner takeaway: The useful governance test is not whether the model can be explained, but whether the explanation is strong enough to support accountability, challenge, and correction when the outcome is still unfair.

Related resources from NHI Mgmt Group

• How should security teams govern AI systems that are explainable but still powerful?
• How should organisations govern explainable AI systems in regulated workflows?
• How should organisations govern AI systems that can still produce plausible but wrong outputs?
• How should security teams govern API keys used for generative AI access?