Cohen’s Kappa

A statistical measure of inter rater agreement that adjusts for the amount of agreement expected by chance. It is more informative than raw percent agreement when two evaluators may sometimes align randomly. In model evaluation, Kappa helps show whether a judge is truly consistent with human annotations.

What Cohen’s Kappa Measures, and Why It Exists

Cohen’s Kappa is a chance-adjusted agreement statistic. It is used when two raters assign categorical labels, but raw percent agreement would overstate quality because some matches occur randomly. That makes Kappa especially useful when you want to know whether consistency is real, not just frequent.

In practice, Kappa answers a different question from accuracy: it asks whether the observed agreement exceeds what you would expect if both raters were guessing according to the same label distribution. A high score generally means the labeling rule is being applied consistently enough to be meaningful.

That distinction matters in evaluation work. A model review can look strong on raw agreement while still being weak on judgment quality if one class dominates. Kappa corrects for that imbalance, which is why it is often chosen for annotation studies, classifier validation, and human review workflows.

How to Interpret Cohen’s Kappa

Kappa values are usually read on a scale from poor to strong agreement, but the exact interpretation depends on the label set, class balance, and task context. A moderate score may be acceptable in difficult annotation tasks, while the same score could be inadequate in a high-stakes review process.

Because Kappa is sensitive to prevalence, it can behave unexpectedly when one category is rare or when both raters heavily favor the same label. In those cases, raw agreement and Kappa may tell different stories, so practitioners should read the statistic alongside the class distribution and confusion pattern.

The measure is also limited to two raters and categorical labels in its classic form. If the task involves ordinal ratings, multiple annotators, or continuous outputs, another agreement metric may be more appropriate, or Kappa may need a specialized variant.

Where Cohen’s Kappa Is Used

Cohen’s Kappa is common in machine learning evaluation, annotation quality checks, and research settings where labels come from humans rather than from an objective sensor. It helps show whether a classifier is aligning with human judgment in a way that is stable and defensible, not merely matching the majority class.

It is especially useful when a team needs to compare a system against a human reference set. In that setting, Kappa can reveal whether a model is learning the structure of the annotation task or only exploiting label skew. For background on broader identity and evaluation contexts, NHI Mgmt Group’s Ultimate Guide to NHIs is a useful reference point for how identity-related systems are measured and governed in practice.

In security-adjacent workflows, Kappa can help when teams are validating reviewer consistency for alerts, policy labels, or access-review decisions. The statistic does not replace operational controls, but it does give a clearer signal when consistency itself is the thing being measured.

Common Pitfalls and Practical Limits

The most common mistake is treating Kappa as a universal quality score. It is not. A low value may reflect label imbalance, unclear task definitions, or genuinely inconsistent raters, while a high value can still hide poor coverage if the class set is trivial or the task is too easy.

Another pitfall is using Kappa without first establishing a shared labeling rule. If the rubric is ambiguous, the statistic will surface disagreement, but it will not explain why. That is why Kappa works best after the annotation scheme has been tightened and raters have been calibrated.

For imbalanced data, the statistic should be read carefully alongside raw counts. When one category dominates, a model or reviewer can appear reasonably consistent by chance alone, so Kappa is most valuable when it is paired with the confusion matrix and the underlying task definition.

Risk and Threat Considerations

When Kappa is used to validate human or model judgments in security, compliance, or moderation workflows, weak agreement can hide a real control problem. If reviewers are not consistently applying the same rule, the process can drift into false confidence even when raw agreement looks acceptable.

Failure mechanism: class imbalance, vague labeling guidance, or over-reliance on percent agreement can make a review process appear stable while actually producing inconsistent decisions.

Impact: inconsistent labeling can degrade detection quality, undermine auditability, and cause bad downstream decisions in moderation, triage, or model evaluation.

Practitioner Guidance

What to watch for: use Kappa when the quality question is agreement beyond chance, not just whether two raters matched often. If the task is skewed, make sure you also inspect the label distribution and confusion matrix, because Kappa can be informative but still easy to misread in imbalanced settings.

Practitioner takeaway: the statistic is most useful when the labeling rule is already well-defined and you want evidence that the rule is being applied consistently.