Percent Agreement

What Percent Agreement Measures

Percent agreement is the simplest way to describe annotator alignment: it counts how often two evaluators assign the same label. Because it is intuitive and fast to compute, it is often used as an initial sanity check, but it treats all matches alike and does not separate true agreement from agreement that may happen by chance.

Why Percent Agreement Can Be Misleading

The main limitation is that percent agreement rises whenever labels are easy to guess or one class dominates the dataset, even if the raters are not especially consistent. That makes it a weak stand-alone measure for comparing judge models, human annotators, or mixed human and model labeling workflows where chance-corrected reliability is more informative.

In practice, this means a high percentage can still hide poor reliability if the label distribution is skewed or if one evaluator tends to choose the most common class. Percent agreement is therefore best treated as a descriptive snapshot, not as proof that the labeling process is robust.

Where Percent Agreement Is Useful

Despite its limitations, percent agreement is useful when you need a quick, transparent read on whether two labelers are broadly aligned before investing in more rigorous analysis. It is also a helpful first-pass metric during annotation setup, pilot review, or debugging of labeling instructions, because it is easy for non-specialists to understand.

It works best as a companion measure. If the goal is to evaluate annotation quality or model-to-human consistency, it should usually be paired with a chance-corrected statistic so that simple label frequency does not overstate reliability.

Percent Agreement Versus Chance-Corrected Reliability

Percent agreement and chance-corrected metrics answer different questions. Agreement tells you how often labels match; corrected metrics ask whether that level of matching is better than what random alignment would produce under the same label distribution.

That distinction matters whenever the labels are imbalanced, the task is subjective, or the evaluation is being used to compare humans with machine-generated annotations. In those settings, percent agreement can still be reported for readability, but it should not be the final measure of record.

How to Interpret It in Annotation Workflows

Percent agreement is most helpful when you use it as an early signal, then interpret it alongside disagreement patterns, class balance, and the review process that produced the labels. A low score can reveal unclear instructions or ambiguous examples, while a high score may simply reflect an easy task or a dominant class.

For that reason, the metric is best read as a workflow health check rather than a statistical conclusion. It shows whether two evaluators often converged, but not whether the annotation scheme itself is reliable enough for downstream use.

Practitioner Guidance

Why practitioners should care: Percent agreement is still useful because it gives a fast, communicable first view of alignment, especially during pilot annotation or model evaluation. The mistake is to treat it as a reliability endpoint when it is really only a surface-level summary.

Common misunderstanding: Many teams assume that a higher percentage automatically means better annotation quality. In reality, the metric can look strong even when the task is easy, the labels are imbalanced, or the apparent agreement is partly accidental.

Practitioner takeaway: Use percent agreement for readability and quick checks, then rely on a chance-corrected metric before making quality or model-comparison decisions.

Related resources from NHI Mgmt Group

• What breaks when teams rely only on percent agreement for judge evaluation?
• What is the difference between percent agreement and Cohen’s Kappa in LLM evaluation?
• What breaks when eSignature evidence is separated from the agreement?
• How should organisations govern digital agreement workflows in regulated environments?