What is the difference between statistical parity, equal opportunity, and equalised odds in model fairness testing?

Why these fairness metrics answer different questions

Statistical parity, equal opportunity, and equalised odds all try to compare model behaviour across groups, but they are not interchangeable. The right metric depends on whether you care most about overall selection rates, outcomes for qualified people, or both types of classification error. That choice matters because fairness constraints can improve one kind of equity while worsening another.

Statistical parity is the broadest test: it asks whether groups receive positive predictions at similar rates, regardless of who is truly qualified. Equal opportunity narrows the focus to true positives, so it asks whether qualified people in each group are equally likely to be correctly approved or identified. Equalised odds adds the false positive side as well, which makes it a stricter and usually harder standard to satisfy.

In practice, these definitions reflect different fairness philosophies. Parity is often easier to explain to non-technical stakeholders, but it can be insensitive to ground truth. Equal opportunity is more aligned with avoiding missed benefit for qualified individuals. Equalised odds is more demanding because it tries to balance both benefit delivery and false alarm burden across groups.

How the metrics differ in practice

The differences become clearer when you think about confusion-matrix rates rather than abstract labels. Statistical parity compares positive prediction rates, so it can look acceptable even if the model is less accurate for one group. Equal opportunity compares true positive rates only, so it does not say anything about whether one group also suffers more false positives. Equalised odds requires both true positive rates and false positive rates to be similar, which reduces the chance that one group is helped or harmed in a systematically different way.

That stricter requirement has a trade-off: equalised odds can be harder to achieve without reducing model utility, especially when base rates differ between groups. In some settings, satisfying all three metrics at once is mathematically impossible unless the model is trivial or the underlying distributions are unusually favourable. That is why fairness testing usually involves selecting the metric that best matches the decision context, not treating all three as universal goals.

For example, if the main concern is access to an opportunity, equal opportunity may be the most defensible starting point. If the system creates both approvals and denials with real consequences, equalised odds is often the more complete diagnostic. If the organisation only measures selection rates, it may miss unequal error patterns hidden behind a superficially balanced outcome rate.

How to choose the right fairness test for the use case

The right metric should follow the decision being automated, the harm you are trying to prevent, and the point at which error matters. A screening model that recommends candidates, flags fraud, or prioritises cases may need different fairness criteria depending on whether the main harm is exclusion, incorrect inclusion, or uneven error burden. The metric should be tied to the cost of a false positive versus a false negative, not chosen only because it is popular in fairness discussions.

Teams should also test more than one metric before deciding that a model is "fair". A model can satisfy statistical parity while still being unevenly accurate, or satisfy equal opportunity while still generating more false alarms for one group. Checking all three helps reveal which kind of fairness the model actually achieves and which type of unfairness may remain hidden.

Where a model cannot meet the strictest metric without unacceptable performance loss, the practical answer is usually to document the trade-off, monitor subgroup error rates, and decide which harm the organisation is willing to reduce first. Fairness testing is therefore not just a scorecard exercise, it is a governance decision about which errors are acceptable and for whom.

Practitioner Guidance

Decision rule: If the business harm is mainly about access to a positive outcome, start with equal opportunity; if both approvals and denials carry meaningful harm, evaluate equalised odds; if you only need a coarse balance check, use statistical parity as a screening metric, not a final verdict.

What to verify: Compare the metrics on the same evaluation set, the same threshold, and the same protected groups. A fairness result is not trustworthy if the threshold, sample size, or group definition changes between runs.

Common mistake: Treating one fairness metric as a universal proxy for fairness. A model can look balanced on one metric and still produce materially different error patterns that matter more in the real decision process.

Practitioner takeaway: Choose the fairness metric that matches the harm you are trying to control, then use the other metrics to surface the trade-offs you may otherwise miss.

Related resources from NHI Mgmt Group

• Guide: Explainable and Fair AI Guide
• Model Fairness
• What is the difference between equality of outcomes and equality of opportunity in AI fairness testing?
• Should organisations compare demographic parity with equal opportunity in fair lending governance?
• What is the difference between model testing and cloud AI posture management?