What is the difference between percent agreement and Cohen’s Kappa in LLM evaluation?

Why the two metrics answer different evaluation questions

Percent agreement is the simplest way to report alignment: it tells you how often two raters picked the same label. That makes it easy to read, but it can overstate quality when one label is common and agreement would occur even with weak judgment. In llm evaluation, that matters because class imbalance and ambiguous outputs can make raw agreement look better than the judges really are.

Cohen’s Kappa asks a harder question: how much agreement remains after accounting for the agreement you would expect by chance alone? That makes it a better fit when you want to compare evaluators, prompts, or rubric versions on a more defensible scale. For practitioners, the key difference is that percent agreement measures surface alignment, while Kappa tries to measure alignment beyond chance.

When the task has skewed label distributions, the gap between the two can be large. A model or rubric that mostly produces the majority label may produce high percent agreement without demonstrating strong discriminative ability. Kappa is designed to expose that problem, which is why it is usually the more credible summary for inter rater reliability in LLM judging tasks.

What each metric can hide in LLM judge workflows

Percent agreement is useful for a quick operational check, but it is blind to agreement structure. It does not tell you whether two judges agree because they are both good, because the rubric is tight, or because one label dominates the dataset. It also does not show whether disagreement clusters around borderline cases, which is often the most important signal in LLM evaluation.

Cohen’s Kappa is more informative, but it is not a magic score. It still depends on the label set, prevalence, and how the raters use the rubric. A low Kappa can appear even when raw agreement looks decent if one category dominates, and a very high Kappa can still hide systematic bias if both raters share the same blind spot. The metric improves interpretation, but it does not replace rubric design or adjudication.

If you are evaluating LLM outputs that affect security, moderation, or quality gates, use the metric that matches the decision you are trying to support. Percent agreement is fine for a fast sanity check. Kappa is the better choice when you need a reliability claim that will stand up to scrutiny, especially for comparing human judges against one another or against an automated judge.

Choosing the right reliability signal for an LLM evaluation protocol

What to verify: Check the label distribution before trusting percent agreement, because a skewed task can make simple match rates misleading. If one category dominates, treat high agreement as a preliminary signal only.

Decision rule: Use percent agreement when you need a lightweight operational readout, and use Cohen’s Kappa when the result will influence a benchmark, a judge comparison, or a publication claim about evaluator consistency.

What practitioners underestimate: Kappa is only as useful as the rubric and sampling strategy behind it. If the labels are underspecified or the examples are too easy, both metrics can look acceptable while still missing the hard boundary cases that matter most in practice.

Practitioner takeaway: For LLM evaluation, percent agreement tells you whether raters matched, but Cohen’s Kappa tells you whether that match is meaningful enough to trust for comparison and decision making.

Related resources from NHI Mgmt Group

• What is the difference between tracing and evaluation in LLM operations?
• What is the difference between tracing and LLM-as-judge evaluation in audio AI systems?
• What is the difference between a straightforward question set and an ambiguity-heavy question set in LLM evaluation?
• What is the difference between span-level and trace-level evaluation in LLM applications?