What breaks when teams rely only on percent agreement for judge evaluation?

Why This Matters for Security Teams

Percent agreement is easy to report and easy to misunderstand. In judge evaluation, it can create a false sense of quality when the label distribution is uneven, which is common in moderation, fraud review, security triage, and policy enforcement. Teams may think their judge is reliable because it matches the majority label often, while still failing on the cases that drive risk, escalation, or user harm. That is why evaluation should be tied to decision quality, not just raw agreement.

This matters operationally because a judge that looks strong on paper can still miss the minority class, such as abusive content, malicious prompts, or false claims that need escalation. The NIST Cybersecurity Framework 2.0 stresses outcome-oriented risk management, and that mindset applies here: the metric has to reflect the failure mode that matters. Current guidance suggests pairing simple agreement with measures that account for chance and class imbalance, such as kappa or alpha, plus precision and recall.

In practice, many teams discover the weakness of percent agreement only after a real incident reveals that the judge had been rubber-stamping the dominant class all along, rather than detecting the cases that actually required intervention.

How It Works in Practice

Percent agreement is the share of cases where two annotators or a judge and gold label choose the same answer. The formula is simple, which is part of the problem. It treats all matches as equally informative and all mismatches as equally bad, even though some errors are much more costly than others. If 90 percent of your data belongs to one label, a judge can appear highly consistent by repeatedly selecting that label, even when it is weak on the rare but important class.

In practice, stronger evaluation uses several views at once:

• Chance-adjusted agreement, such as kappa or alpha, to reduce the effect of label prevalence.
• Precision and recall for each class, especially the minority or high-risk class.
• Confusion matrices to show where errors cluster.
• Sampling checks on difficult slices, such as edge cases, adversarial inputs, or low-confidence examples.

That approach is more aligned with NIST AI Risk Management Framework principles, which emphasize validity, reliability, and contextual performance rather than a single simplistic metric. It also fits the logic in MITRE ATLAS when judge behavior is part of an adversarial or abuse-facing pipeline, because attackers and edge cases often concentrate in the labels percent agreement hides. If the judge is used in an agentic workflow, teams should also consider whether the evaluation set reflects the actions the system can actually take, not just the labels it can predict.

For governance, the practical question is not whether agreement is high, but whether the judge is dependable on the outcomes that matter most to the business and the threat model. These controls tend to break down when the dataset is heavily imbalanced and the evaluation set does not include enough minority-class examples to reveal the failure pattern.

Common Variations and Edge Cases

Tighter evaluation often increases review time and annotation cost, so organisations have to balance speed against confidence. That tradeoff becomes more pronounced when labels are subjective, multi-label, or policy-dependent, because there is no universal standard for perfect agreement in those cases.

One common edge case is when the judge is used for ranking or routing rather than final classification. Percent agreement may look acceptable, but the system can still fail if the wrong items are consistently prioritised. Another is evolving policy, where label definitions change over time and the historical agreement score no longer reflects current expectations. In those situations, current guidance suggests tracking agreement by slice, version, and reviewer cohort rather than relying on one blended number.

For AI governance teams, the relevant question is whether the judge is stable under ambiguity, not whether it matches a majority label. If the use case includes security filtering, fraud review, or human escalation, pairing agreement with calibration checks and error analysis is more defensible than treating percent agreement as a standalone quality gate. The NIST Cybersecurity Framework 2.0 is useful here as a reminder to validate controls against actual risk outcomes, not only summary scores. In multilingual or rapidly changing policy environments, this guidance can also degrade because label ambiguity makes agreement scores less comparable across time and reviewers.

Related resources from NHI Mgmt Group

• What breaks when teams rely on periodic access certification alone?
• What breaks when teams rely on system state restore for identity servers?
• What breaks when teams rely on visibility without enforcement for AI agents?
• What breaks when teams rely on single-turn filters to stop AI abuse?