Statistical parity checks whether different groups receive positive outcomes at similar rates. Equal opportunity focuses on whether qualified individuals across groups have similar true positive rates. Equalised odds is stricter because it requires similarity in both true positive and false positive rates. Together, these metrics reveal different ways a model can be unfair.
What each fairness metric is actually testing
These three metrics answer different fairness questions, so they are not interchangeable. Statistical parity asks whether groups get positive predictions at similar overall rates. equal opportunity narrows the lens to qualified cases and asks whether true positive rates are similar across groups. Equalised odds is stricter still, because it asks for parity in both true positive and false positive rates.
The practical difference is the population each metric conditions on. Statistical parity ignores whether the model is actually correct, which makes it easy to satisfy while still producing unfair decisions. Equal opportunity focuses on benefit allocation to people who should have received the positive outcome. Equalised odds adds a penalty for unequal error burden, which matters when false positives create real harm.
For teams testing a model, the right metric depends on what “fair” means in the decision context. Hiring, lending, fraud, admissions, and access decisions can tolerate different error trade-offs, but they should not be judged with a metric that hides the most important failure mode. That is why fairness testing usually compares several metrics rather than treating one number as definitive.
Where the metrics disagree in practice
It is common for a model to satisfy one fairness criterion while failing another. A classifier can have statistical parity but still be badly calibrated or inaccurate for one group. It can also achieve equal opportunity while producing more false positives in another group, which means it is still imposing a heavier burden on that group. Equalised odds catches that broader imbalance, but it may be harder to satisfy without changing thresholds or accepting some utility loss.
The disagreement matters because each metric encodes a different policy choice. Statistical parity focuses on outcome balance. Equal opportunity reflects concern about missed beneficial decisions among qualified individuals. Equalised odds treats false positives and false negatives as equally important fairness harms, which is useful when both kinds of error affect access, trust, or downstream treatment. In many real systems, you need to be explicit about which harm you are trying to control.
- OWASP Web Security Testing Guide can help teams structure repeatable testing around decision endpoints and validation points.
- NIST Cybersecurity Framework 2.0 provides a broader governance lens for documenting controls, monitoring, and response around model-driven systems.
When fairness metrics diverge, the gap is often not a mathematical problem alone, but a thresholding and policy problem. Changing the decision threshold can improve one metric while worsening another, so teams should test on the operating point that will actually be deployed rather than on an abstract score alone. If the deployment setting changes, the fairness result can change with it.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST CSF 2.0 and NIST AI RMF set the technical controls, while ISO/IEC 42001:2023 define the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | GV.OC-01 — Organizational Context | Fairness testing supports governance choices for high-impact model use. |
| Recommendation — Define the model's decision context and acceptable fairness trade-offs before deployment. | ||
| NIST AI RMF | MAP 2.3 — Measure and Monitor | Comparing fairness metrics is a monitoring activity for model behavior. |
| Recommendation — Measure fairness across multiple metrics and monitor how thresholds change the result. | ||
| ISO/IEC 42001:2023 | A.5 — AI risk treatment | Selecting and documenting fairness criteria is part of AI risk treatment. |
| Recommendation — Document which fairness metric governs each use case and why it was chosen. | ||
Practitioner Guidance
What to prioritise: Start by identifying which harm matters most in the decision context, then choose the metric that tests that harm directly. If the main concern is disparate access, statistical parity may be a useful screen; if the concern is missed benefit for qualified people, equal opportunity is the better lens; if both types of classification error matter, test equalised odds as well.
What to verify: Check that the “qualified” label is defensible and stable, because equal opportunity depends on the quality of that ground truth. Also verify the threshold used in production, since fairness findings at one threshold do not automatically transfer to another. For regulated or high-impact use cases, keep the confusion matrix by group so you can see which error type is driving the disparity.
Common mistake: Treating statistical parity as proof of fairness is the fastest way to miss harmful error asymmetry. A model can look balanced on outcomes while still denying qualified individuals a fair chance or burdening one group with more false positives.
Practitioner takeaway: Fairness testing is about choosing the right harm model, not picking the best-sounding metric, and the most useful check is the one that matches the decision consequence you are actually trying to control.
Related resources from NHI Mgmt Group
- What is the difference between equality of outcomes and equality of opportunity in AI fairness testing?
- What is the difference between model testing and cloud AI posture management?
- What is the difference between prompt injection testing and model adversarial testing?
- What is the difference between safe AI pentesting and uncontrolled model-assisted testing?
