What is the difference between selection rate and scoring rate in a bias audit?

How the Two Rates Separate Binary Outcomes from Continuous Scores

selection rate and scoring rate answer different measurement questions. Selection rate applies when the model or decision process ends in a discrete outcome, such as hired or not hired, approved or denied, selected or not selected. The comparison is based on how often each subgroup receives the positive outcome, so it reflects representation at the decision boundary rather than the shape of the underlying score distribution.

Scoring rate applies when the system emits a continuous value and the audit needs a fair comparison across groups before a cutoff is applied. In the audit method described here, the sample median is used to convert scores into pass or fail, which makes subgroup comparison possible even when the model itself does not produce a binary decision. That distinction matters because the measurement design affects both the impact ratio and the meaning of any disparity you find.

When the outcome is already binary, selection rate is the more direct measure because it reflects the actual decision the system made. When the system is scoring people or entities on a continuum, scoring rate helps avoid pretending the score is inherently categorical before the audit defines a threshold. That keeps the comparison tied to the evaluation method rather than to an arbitrary assumption about where the score should be cut.

Why the Audit Method Changes What You Can Compare

In bias audit, the metric choice is not cosmetic. A binary outcome supports a straightforward positive-outcome rate by subgroup, while a score-based system requires a rule for converting values into comparable outcomes. Using the sample median is one practical way to create that comparison set, but it also means the audit is evaluating relative standing within the sample rather than an externally fixed business threshold.

That has two practical consequences. First, the same system can look different depending on whether the audit is measuring selection behavior or score distribution. Second, the interpretation of the result depends on whether the underlying process is truly a decision system or only a ranking system. If the score is later used operationally as a pass or fail gate, the conversion rule becomes part of the audit design and should be treated as such.

For practitioners, the most important check is to align the metric with the decision architecture. If downstream users act on a yes or no result, selection rate is the cleaner measure. If the system ranks candidates or risk levels before a human or policy threshold is applied, scoring rate is the better starting point because it exposes whether one subgroup is consistently concentrated above or below the reference point.

Risk and Threat Considerations

Bias audits can miss material disparity if the wrong rate is used for the model output type. A binary system measured as though it were continuous can blur the actual decision pattern, while a scored system measured without a defensible threshold can hide subgroup differences behind an arbitrary cutoff.

Failure mechanism: The audit misclassifies the output structure, then compares groups using a metric that does not match how the system is actually used. That can distort impact ratio analysis, produce misleading parity conclusions, and leave harmful subgroup effects unchallenged.

Impact: Teams may approve a system that appears fair under the wrong measurement method, or reject a system for a disparity that is an artifact of the audit design rather than the operational decision process. In regulated or high-stakes settings, that creates governance, compliance, and reputational exposure.

Practitioner Guidance

What to verify: Confirm whether the system’s real output is binary or continuous before choosing the rate. If the decision is derived from scores, document the thresholding rule and make sure the subgroup comparison uses the same rule consistently across the audit.

Decision rule: If the outcome is the final decision, use selection rate; if the outcome is a score that later becomes a decision, use scoring rate to understand subgroup positioning before the cutoff is applied. Do not mix the two in the same comparison without explaining the change in measurement basis.

Practitioner takeaway: The key judgment is not which rate sounds more rigorous, but which one matches the system’s actual decision form, because a fair audit can still be misleading if the measurement layer does not reflect how the model is used.

Related resources from NHI Mgmt Group

• What is the difference between direct political bias scoring and cross model judging?
• What is the difference between New York City Local Law 144 and the newer state-level AEDT bias audit approaches?
• What is the difference between New York City bias audit requirements and California’s employment AI rules?
• What is the difference between an automated employment decision tool and a bias audit under Local Law 144?