Explainable and Fair AI: Bias Testing and Explanations

An AI system that cannot show who it treats worse, or explain why it reached a decision, is a legal, operational and security liability long before it becomes a headline. Bias enters through data, through design choices and through the people who use the output, and it can do all of that without anyone intending harm. Regulators now expect evidence: bias examination of training data under the EU AI Act, explanations for people affected by significant automated decisions under GDPR Article 22 and AI Act Article 86, and published bias audits for hiring tools in New York City. This guide covers how to test for bias, which fairness metrics to choose and why they conflict, how to build explanations that people can actually use, and how to monitor for bias drift after go-live. It is for AI engineers, model risk, privacy and security practitioners asked to sign off AI systems.

Key takeaways

Bias has three sources, not one. NIST groups it into systemic, computational and statistical, and human-cognitive bias, and only the middle group is visible in a confusion matrix.
Fairness metrics conflict by design. Demographic parity, equalised odds and equal opportunity measure different things, so pick the metric from the harm you are trying to prevent and document why.
Test by group, at deployment conditions. Stratified evaluation across groups and intersections, on data that looks like production, is the core of any bias test; an aggregate accuracy figure hides the problem.
An explanation is a product for a reader. The ICO separates rationale, responsibility, data, fairness, safety and performance, and impact explanations; a SHAP plot satisfies none of them on its own.
Bias drifts after launch. Population change, feedback loops and model updates move group outcomes, so selection rates and error rates per group belong in production monitoring, with thresholds and owners.

Where bias comes from

NIST AI 100-1 (the AI Risk Management Framework) lists "fair, with harmful bias managed" as one of seven trustworthiness characteristics and warns that mitigating bias does not by itself make a system fair: a model balanced across demographic groups can still be inaccessible to disabled users. NIST SP 1270 expands the three categories with examples, which make a useful checklist when you scope a test.

Category (NIST SP 1270)	Examples	Where it shows up in practice	What tests or reviews catch it
Systemic	Historical, institutional, societal bias	Training labels that encode past decisions (who was hired, who was approved for credit)	Label review with domain experts; questioning whether the target variable is the right one
Computational and statistical	Sampling and selection, representation, measurement, temporal, survivorship bias	Under-represented groups, proxy features (postcode for ethnicity), stale data, data only from successful cases	Group representation analysis, proxy detection, per-group error rates, time-split validation
Human-cognitive	Automation complacency, confirmation bias, anchoring	Reviewers rubber-stamping model output, or over-riding it selectively for some groups	Override rate analysis by group, reviewer sampling, UI design reviews
Use-related	Deployment bias, emergent bias, feedback loops	A model used outside its intended population, or whose decisions shape its next training set	Intended-use checks at intake, monitoring of input distributions, separation of decision data from training data

SP 1270 also notes that the most accurate model is not necessarily the least harmful one, so selecting on accuracy alone is a bias decision, and that zero bias risk is not achievable: measure, reduce to an agreed tolerance, document and monitor.

Bias in generative AI and LLM judges

Representational harm: text and image generators can reproduce stereotypes and homogenise outputs. NIST AI 600-1 lists "harmful bias or homogenization" as one of its twelve generative AI risks.
Allocation through LLMs: an LLM that screens CVs, triages claims or summarises case notes makes or shapes allocative decisions, and should be tested with the same group-level methods as a classifier, using counterfactual prompts (identical inputs that differ only in name, gender marker or dialect).
Judge bias: when one model grades another (LLM-as-judge), the judge can prefer longer answers, the first option shown, or its own style. Calibrate judges against a human-labelled sample, randomise option order and report agreement rates before trusting judge scores in a fairness evaluation.
Action bias in agents: agents that choose which customer, ticket or transaction to act on create the same allocation risk at machine speed. Log the selection criteria with each action so outcomes can be analysed by group later. See the AI Agent Audit Trails for Regulators Guide.

Choosing fairness metrics

There is no single mathematical definition of fairness, and NIST SP 1270 notes that some observational definitions can increase discrimination in some cases. Choose the metric from the harm: is the concern that a group is selected less often (allocation), or that the model makes more mistakes for a group (quality of service)? The table uses the definitions in the open-source Fairlearn documentation, which match the wider literature.

Metric	What it requires	Use when	Watch out for
Demographic parity (selection rate parity)	Equal selection rates across groups, regardless of true label	Allocation decisions where historical labels are suspect (hiring shortlists, outreach)	Ignores whether people actually qualify; can conflict with accuracy where base rates genuinely differ
Selection rate ratio (impact ratio)	Lowest group selection rate divided by highest; 1.0 is parity	Employment screening and audits expressed as ratios, such as NYC Local Law 144 bias audits	The US four-fifths (0.8) rule of thumb is a screening signal, not a legal safe harbour; small groups make ratios unstable
Equalised odds	Equal true positive and false positive rates across groups	Decisions where both error types harm people (fraud flags, risk scores)	Depends on labels being trustworthy; hard to satisfy together with calibration
Equal opportunity	Equal true positive rates only	Benefit decisions where missing a qualified person is the main harm	Ignores false positive disparities, so the cost of wrong flags can still fall on one group
Calibration by group	A score of 0.7 means the same probability for every group	Scores that humans read and act on (credit, clinical risk)	Can coexist with large error rate gaps when base rates differ
Counterfactual consistency	Output unchanged when only a protected attribute or proxy changes	LLM and generative systems where labels do not exist	Only tests the attributes you think to vary; names and dialects are imperfect stand-ins

Report differences and ratios: Fairlearn's demographic_parity_difference() and demographic_parity_ratio(), and equalized_odds_difference(), give comparable numbers; MetricFrame breaks any metric down by group and by intersections.
Check intersections: aggregate results can mislead (Simpson's paradox), and a model can be fair for women and for older applicants but not for older women. Testing many small subgroups creates a multiple comparisons problem, so report confidence intervals and minimum group sizes.
Write the tolerance down first: agree the metric, the threshold (for example, impact ratio of 0.8 or better, or a TPR gap under 5 percentage points) and the escalation route before you see the results.

How to run a bias test

Frame the decision. Record the intended purpose, the affected population, the decision the output feeds and the harm of each error type. Link it to the use case's risk tier; see the AI Use Case Risk Tiering Guide.
Get group data lawfully. Decide which groups to test (sex, ethnicity, age, disability and relevant local categories) and how you will obtain the attribute: collected with consent, inferred under a documented method, or held by a separate trusted party. Under the EU AI Act this processing is constrained (see below), and under GDPR it is special category data in most cases.
Profile the data. Compare group representation in training, validation and test sets with the deployment population; search for proxy features by testing how well each feature predicts the protected attribute; check label quality per group with domain experts.
Evaluate stratified performance. Run every chosen metric per group and per intersection, on a held-out set that matches deployment conditions, as NIST SP 1270 recommends. Include confidence intervals.
Run counterfactual tests. Paired prompts that change only names, pronouns or dialect for LLMs; perturbed proxy features for tabular models.
Mitigate and re-test. Options sit at three points: pre-processing (reweighting, resampling, removing or transforming proxies), in-processing (fairness constraints during training) and post-processing (group-aware thresholds, where lawful). Re-run the full test and record the accuracy cost of each option.
Decide and document. A named owner accepts residual disparity against the agreed tolerance, or the system does not ship. Record the result in the model card and the AI impact assessment. See the AI Impact Assessment Template.
Pilot with rollback. Release to a limited population or in shadow mode (model scores logged, humans decide), compare group outcomes with the baseline process, and keep the previous model or manual process ready to restore if thresholds are breached.

Explaining decisions people can use

NIST distinguishes transparency (what happened), explainability (how the system reached the output) and interpretability (why it matters to the user in context). The ICO and Alan Turing Institute guidance "Explaining decisions made with AI" turns this into six explanation types and separates process-based explanations (good governance was followed) from outcome-based ones (why this decision for this person). The ICO notes the guidance is under review after the Data (Use and Access) Act, so check for updates.

ICO explanation type	What the person learns	Evidence you need to hold
Rationale	The reasons for the decision in plain language	Per-decision feature attributions or reason codes, mapped to plain-language reasons
Responsibility	Who built and runs the system and how to get a human review	RACI, contact route, human review procedure
Data	Which data was used and how	Data sheet, input specification, data lineage for the decision
Fairness	What was done to avoid bias and whether they were treated equitably	Bias test results, chosen metric and tolerance, monitoring results
Safety and performance	How accurate, reliable, secure and robust the system is	Accuracy by group, robustness and security test reports
Impact	How the system affects individuals and society, and how that is monitored	Impact assessment, complaint and appeal statistics

Choosing an explanation technique

Interpretable by design: scorecards, generalised additive models and shallow trees give exact reasons. Prefer them for high-impact decisions where performance loss is small; the explanation is the model.
Post-hoc attribution: SHAP and LIME estimate feature contributions for complex models. They are approximations, can be unstable for correlated features and can be manipulated, so validate them against known cases and never present them as the model's literal reasoning.
Counterfactual explanations: "you would have been approved if your declared income were higher by X" is often the most useful outcome explanation for a person; restrict counterfactuals to features the person can actually change.
LLM-generated rationales: a model's self-description of its reasoning is not a faithful explanation. Use retrieval citations and logged tool calls as evidence instead, and test rationales for consistency.
Documentation explanations: model cards, data sheets and instructions for use carry the process-based explanation and the deployer-facing information required by AI Act Article 13.

What the law requires

Requirements differ by jurisdiction and change often; confirm current status before relying on any line below.

Instrument	Bias and fairness duty	Explanation duty	Status note
EU AI Act, Regulation (EU) 2024/1689, Art. 10	Providers of high-risk systems examine data sets for possible biases affecting health, safety or fundamental rights or leading to prohibited discrimination (10(2)(f)), take measures to detect, prevent and mitigate them (10(2)(g)), and use data that is relevant, sufficiently representative and with appropriate statistical properties for the groups concerned (10(3))	Not in Art. 10	The Digital Omnibus on AI received final Council approval on 29 June 2026; it moves the legal basis for processing special category data for bias detection from Art. 10(5) to a new Art. 4a and extends it to other providers and deployers. It does not create a general duty to run bias correction. Check the Official Journal for the final text
EU AI Act, Art. 13	Instructions for use state performance for specific persons or groups where appropriate (13(3)(b)(v))	Systems must be transparent enough for deployers to interpret output (13(1)); instructions include accuracy metrics, input data specifications, information to interpret output, and human oversight measures	Under the Omnibus, Annex III high-risk obligations apply from 2 December 2027 and Annex I product systems from 2 August 2028
EU AI Act, Art. 86	None directly	Affected persons can request clear and meaningful explanations of the AI system's role and the main elements of a deployer's decision, for Annex III systems (except point 2) with legal or similarly significant adverse effects	Applies only where other Union law does not already give the same right
GDPR Art. 22 (EU)	Fairness principle in Art. 5(1)(a)	Right not to be subject to solely automated decisions with legal or similarly significant effects, unless necessary for a contract, authorised by law or based on explicit consent; then human intervention, the right to express a view and to contest	UK rules on automated decisions were changed by the Data (Use and Access) Act 2025; ICO guidance is under review
NYC Local Law 144 of 2021	Automated employment decision tools need a bias audit within one year of use, with a public summary of results	Notice to candidates and employees, 10 business days before use	Enforced by DCWP since 5 July 2023
Colorado	SB 24-205 has been replaced by SB 26-189 (automated decision-making technology in consequential decisions)	See the dedicated guide	See the Colorado AI Act Guide for current obligations and dates

For the wider EU obligations see the EU AI Act High-Risk AI Systems Guide, and for data protection interaction see the AI and GDPR Guide.

Decision guidance

Interpretable model or complex model plus explainer: if the decision has legal or similarly significant effects and an interpretable model is within a few points of accuracy, choose the interpretable model. Use a complex model only where the gain is material and you can validate the explainer.
Collect protected attributes or not: not collecting them does not remove bias, it removes your ability to measure it. Where collection is not possible, use a documented inference method or a trusted third party, and record the error rate of the inference.
Group thresholds: post-processing with different thresholds per group can close gaps fast but may be unlawful direct differential treatment in some jurisdictions and sectors. Take legal advice before using it; prefer data and feature fixes.
Human in the loop: a reviewer only counts as meaningful oversight if they have the authority, time and information to disagree. Measure override rates; a rate near zero suggests automation complacency.
Small versus large organisations: a small team buying a vendor tool should demand the vendor's bias audit, group performance figures and instructions for use, then test on its own population sample. Large organisations with in-house models need a standing fairness test harness in the ML pipeline and an independent second line review, as in the AI Model Risk Management Guide (SR 11-7). Contract terms belong in the AI Procurement Checklist.

Detection and measurement after go-live

Bias drift is the change in group outcomes over time, caused by population shift, upstream data changes, model retraining or feedback loops where the model's decisions become the next training labels. Log, per decision: model version, inputs (or a hash), output, final human decision, override reason and, where lawfully held, group attributes in a separate store. See the Cloud Logging and Audit Trail Guide.

Selection rate ratio by group and intersection: weekly or per batch, alert below the agreed tolerance.
Error rate gaps: TPR and FPR differences per group once outcomes are known, with confidence intervals.
Input drift by group: population stability index or similar on key features, split by group.
Human override rate by group: both very low and uneven rates are warning signs.
Explanation and appeal metrics: explanation requests answered within the service level, and the share of appeals that overturn the decision, by group.
Re-test cadence: percentage of production models with a bias test less than 12 months old, and after every material model change.

Framework mapping

Framework	Where explainability and fairness sit
NIST AI RMF (AI 100-1)	Trustworthiness characteristics "explainable and interpretable" and "fair, with harmful bias managed"; GOVERN sets tolerance and roles, MAP frames context and affected groups, MEASURE runs the tests, MANAGE responds and monitors. See the NIST AI RMF Guide
NIST SP 1270	Bias categories, TEVV recommendations and socio-technical approach used in this guide
NIST CSF 2.0	GV.OC-03 legal and regulatory requirements, including privacy and civil liberties, are understood and managed
NIST SP 800-53 Rev. 5	RA-3 Risk Assessment, SA-11 Developer Testing and Evaluation, CA-7 Continuous Monitoring, PT-3 Personally Identifiable Information Processing Purposes (for group attribute data)
ISO/IEC TR 24027:2021	Measurement techniques and methods for assessing bias across all AI lifecycle phases
EU AI Act	Art. 10 data governance, Art. 13 transparency to deployers, Art. 86 right to explanation

Common failure patterns

Testing aggregate accuracy only: a model with 94% accuracy overall can have double the false positive rate for one group. Without stratified results the problem is invisible until complaints arrive.
"We removed the protected attribute": postcode, first name, school, employment gaps and device type act as proxies. Removing the attribute also removes the ability to measure the effect.
Choosing the metric after seeing the results: picking whichever fairness metric looks best is easy to spot in an audit and undermines the whole test. Fix the metric and tolerance in the test plan.
Presenting attribution plots as explanations: a SHAP chart sent to a rejected applicant is neither a rationale nor a route to human review. Build plain-language reason codes and a contact path.
Testing once, at launch: retraining, a new customer segment or a changed upstream form can move group outcomes within weeks. Treat bias metrics as production telemetry.
Weak protection of the fairness data: group attributes collected for testing are some of the most sensitive data you hold. Keep them separated, access-controlled and deleted when no longer needed, consistent with the conditions the AI Act sets for this processing.

Practitioner checklist

Record intended purpose, affected groups and the harm of each error type for every in-scope AI use case.
Agree fairness metrics, thresholds and escalation owners before testing, and document the reason for each choice.
Establish a lawful, access-controlled way to hold or infer group attributes for testing.
Run stratified and intersectional evaluation with confidence intervals on deployment-like data.
Add counterfactual prompt tests for LLM-based decisions and calibrate any LLM judge against human labels.
Provide per-decision reason codes, a human review route and the six ICO explanation types for significant decisions.
Pilot in shadow mode with a rollback path to the previous model or process.
Monitor selection rate ratios, error rate gaps, drift and override rates by group in production.
Re-test after every material model or data change, and at least annually.
Confirm the current status of the AI Act Omnibus, UK automated decision rules, NYC LL144 and Colorado law for your use cases.

Standards and references

Related NHI Mgmt Group resources: AI Risk Assessment Guide · AI Governance Guide · EU AI Act Compliance Guide · AI Red Teaming Guide. This guide summarises the EU AI Act, GDPR, NYC Local Law 144 and Colorado law as they relate to AI bias and explanation and is not legal advice.

Explainable and Fair AI Guide: Testing for Bias and Explaining Decisions