What is the difference between using the mean and the median to assess disparate impact in regression models?

The mean compares average scores between groups, while the median-based approach compares the share of candidates scoring above the dataset midpoint. The mean is easier to distort with outliers or score shifts. The median is more robust, but it still assumes that splitting the data in half is a meaningful proxy for fairness in the specific use case.

Why This Matters for Security Teams

Mean and median are not interchangeable lenses for disparate impact. The mean answers whether one group’s average model score is lower or higher, which is useful when the decision process is sensitive to overall score shifts. The median-based approach asks a different question, whether one group is passing the midpoint threshold at the same rate as another. That makes the choice of statistic a governance decision, not just a mathematical preference.

For security, compliance, and fairness review, the risk is that teams treat one summary as if it captures all materially relevant bias. A mean can hide a small number of extreme scores, while a median can hide important distribution changes above and below the threshold. If the model is used for ranking, prioritisation, or triage, these differences can materially affect who advances and who is filtered out. The right comparison depends on how the model output is actually consumed.

In practice, many review failures happen when teams pick the statistic that is easiest to defend after the fact rather than the one that matches the decision boundary in production.

How It Works in Practice

The mean-based approach compares average scores between groups, so it is most informative when the model output is intended to behave like a continuous signal. If one group’s scores are pulled down by a few very low values, the mean will reflect that shift immediately. That is useful when the organisation wants to understand whether the model is systematically compressing or lifting a group’s scores across the full range.

The median-based approach is more threshold-oriented. It effectively asks whether one group is concentrated above or below the middle of the score distribution. In practice, that can be easier to interpret for pass-fail or shortlist decisions, because it aligns more closely with a cutoff-based workflow. It is also less sensitive to outliers, which is helpful when a few unusual cases would otherwise dominate the result.

Use the mean when the distribution itself matters, especially for ranking, scoring, or prioritisation systems.
Use the median when the operational question is closer to, “who lands above the midpoint or threshold?”
Check both when you need to understand whether bias comes from broad score shifts or from concentration around a decision boundary.

For practitioners, the key is to match the statistic to the downstream decision rule. If the production decision uses a cutoff, the median-style question may be more relevant; if the output drives a continuous ranking, the mean may be more revealing. These controls tend to break down when the score distribution is highly skewed or when the cutoff is arbitrary relative to the business outcome.

Common Variations and Edge Cases

Tighter fairness measurement often increases interpretive complexity, requiring organisations to balance simplicity against fidelity to the actual decision process. That trade-off matters because different model families and deployment settings can make one statistic look cleaner without making it more truthful.

One common edge case is a dataset with strong clustering near the midpoint. In that setting, a median-based comparison may appear stable even when many candidates are only marginally above or below the threshold, which can make small model changes look larger or smaller than they are. Another edge case is severe class imbalance or score compression, where the median can become a weak proxy for decision quality because half the population may sit in a narrow band.

There is also a practical distinction between fairness in scoring and fairness in selection. If the organisation later applies a separate business rule, such as a top-k cutoff or a manual review queue, neither the mean nor the median alone fully describes disparate impact. The best practice is evolving toward evaluating the statistic that corresponds to the actual operational use case, rather than treating a single summary measure as universally valid.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

NIST CSF 2.0 and NIST AI RMF set the technical controls, while ISO/IEC 42001:2023 define the regulatory obligations.

Framework	Control / Reference	Relevance
NIST CSF 2.0	GV.OV — Oversight	Fairness metric choice is a governance and oversight decision for model use.
Recommendation — Define the fairness metric that matches the production decision rule and review it as part of governance.
NIST AI RMF	MAP — Map Context	The metric must map to the model's actual decision context and intended use.
	MEASURE — Measure and Analyze	Assessing disparate impact requires measurement that reflects the model's observed behavior.
Recommendation — Map the score statistic to the real decision boundary before using it in fairness assessments. Measure both distribution shift and threshold effects so the fairness result matches deployment behavior.
ISO/IEC 42001:2023	A.6.1 — AI risk assessment	Choosing a disparate-impact measure is part of AI risk evaluation and control selection.
Recommendation — Document why the selected fairness metric fits the use case and validate it during AI risk assessment.

Practitioner Guidance

What to prioritise: Start by identifying the real decision rule in production, then choose the statistic that best mirrors that rule. If the model score is consumed as a continuous ranking signal, mean-based review usually carries more diagnostic value; if a midpoint or pass-fail threshold drives the outcome, the median-style lens is more decision-relevant.

What to verify: Confirm whether the score distribution is skewed, clipped, or highly clustered near the cutoff. If it is, inspect both the overall distribution and the decision boundary, because a single summary metric can miss the mechanism that is actually producing disparate impact.

Decision rule: Treat a large gap in one statistic and a small gap in the other as a signal to investigate where the disparity sits in the distribution, not as a reason to pick the friendlier number. If the statistic does not track the downstream action, it should not be used as the primary fairness argument.

Practitioner takeaway: The useful question is not which statistic is “better” in the abstract, but which one best reflects the way the model changes real outcomes for each group.