Blog

Krippendorff’s alpha: measuring inter-annotator agreement in AI data

Appen
Published August 2025 · Updated October 2026

Krippendorff’s alpha measures how well annotators agree beyond chance across any number of raters, data types, and missing labels, with thresholds to act on.

An audit of 10 widely used machine learning test sets found an average label error rate of at least 3.3%, which was enough to reorder model rankings once the labels were corrected (Northcutt et al., 2021). Krippendorff's alpha is a reliability score that tells you how much of your annotators' agreement goes beyond what chance alone would produce. This article explains how alpha works, how to interpret it, and how to apply it to AI annotation and evaluation projects.

What is Krippendorff's alpha?

Krippendorff's alpha (α) is a chance-corrected reliability coefficient. It compares the disagreement observed among annotators with the disagreement you would expect if the same labels were paired at random. It works with any number of raters, nominal-to-ratio data, and missing labels. A value of 1 means perfect agreement, and 0 means agreement no better than chance (Krippendorff, 2018).

Krippendorff developed the coefficient for content analysis. Hayes and Krippendorff (2007) later proposed it as a standard reliability measure for coding data because it generalizes several earlier coefficients. In AI work, that generality is practical: one metric can cover intent labels, toxicity tiers and 1-10 quality scores in the same project. To see where alpha fits within a broader quality program, see our overview of AI data quality →.

Why inter-rater reliability matters for AI data

Inter-rater reliability (IRR), also called inter-annotator agreement, is the measurable core of any human in the loop workflow. If annotators can't reproduce each other's judgments, a model trained or evaluated on their labels learns their inconsistency. Low agreement usually comes from one of these failure modes:

  • Ambiguous guidelines: Category boundaries such as "somewhat" versus "highly" toxic are loosely defined, so each annotator draws the line in a different place.
  • Uneven calibration: One annotator or pool applies the rubric more strictly, which shifts label distributions without anyone noticing.
  • Inflated raw agreement: When one class dominates, percent agreement looks high even if annotators agree on rare classes no better than chance.
  • Noisy evaluation signals: Large language model (LLM) preference and rubric scores become unreliable benchmarks when raters disagree on what "good" means.
  • Audit exposure: The EU AI Act requires training, validation and testing data for high-risk systems to be, to the best extent possible, free of errors and complete (European Parliament and Council, 2024).

Reliability is also a prerequisite for rubric-based LLM evaluation, where scores are only as stable as the raters applying the rubric.

How Krippendorff's alpha works

Alpha is defined as one minus the ratio of observed disagreement to expected disagreement (Krippendorff, 2011):

α = 1 − Dₒ / Dₑ

  • Observed disagreement (Dₒ): The average distance between pairs of values that different annotators assigned to the same item.
  • Expected disagreement (Dₑ): The average distance between all pairable values in the dataset, regardless of item. This is the disagreement you would see if labels were paired at random.
  • Distance metric (δ²): The function that scores how severe each mismatch is. It's set by the measurement level of your labels.

Only pairable values count. An item with a missing rating still contributes the pairs it has, and an item with a single rating is excluded.

Measurement level Example AI labels How mismatches are scored
Nominal Intent classes, entity types Every mismatch counts equally
Ordinal Toxicity tiers, Likert ratings Penalty grows with the ranked values lying between the two labels
Interval 1–10 quality scores Squared difference between values
Ratio Bounding-box area, durations Squared difference relative to the values' sum

Worked example: toxicity ratings split across annotators

Three annotators rate six LLM responses as not toxic (N), somewhat toxic (S) or highly toxic (H).

Response Annotator A Annotator B Annotator C
1 N N N
2 S S H
3 H H H
4 N S N
5 S S S
6 H S H
  • Percent pairwise agreement: 66.7%. 12 of the 18 annotator pairs match.
  • Nominal alpha: 0.52. Every mismatch is penalized as heavily as N versus H.
  • Ordinal alpha: 0.73. All six disagreements fall between adjacent tiers, and the ordinal metric accounts for that.

Ordinal is the right setting here because the scale is ranked. Even so, 0.73 falls in the tentative band, and responses 2 and 6 show why: annotators split on the somewhat/highly boundary. The fix is adding boundary examples to the guideline, not adding more annotators. For a related approach to ambiguous safety labels, see AI safety evaluation with tricategorical reasoning.

Krippendorff's alpha vs Cohen's kappa and Fleiss' kappa

Dimension Percent agreement Cohen's kappa Fleiss' kappa Krippendorff's alpha
Number of raters Any Exactly two (Cohen, 1960) Two or more, fixed number per item (Fleiss, 1971) Any, can vary per item
Chance correction None Yes Yes Yes
Measurement levels Nominal Nominal; ordinal via weighted kappa (Cohen, 1968) Nominal Nominal, ordinal, interval, ratio
Missing ratings Tolerated pairwise Not supported Not supported Supported
Disagreement severity Not modeled Weighted variant only Not modeled Set by distance metric
Typical fit Quick sanity checks Two-annotator nominal tasks Crowd tasks with equal judgments per item Mixed designs with gaps and scaled labels

Artstein and Poesio (2008) review these coefficients for computational linguistics and explain when weighted coefficients such as alpha are the better choice. To apply agreement measurement to scoring frameworks, see LLM evaluation rubrics.

How to calculate and interpret Krippendorff's alpha

  1. Set your threshold before labeling: Fix the minimum acceptable alpha in the project plan so nobody can rationalize results afterward. Krippendorff (2004) recommends three bands:
    1. Reliable (α ≥ 0.800): Use the data for training and reported conclusions.
    2. Tentative (0.667 ≤ α < 0.800): Use the data only for exploratory or provisional findings.
    3. Unreliable (α < 0.667): Don't use the data. Revise guidelines and relabel.
  2. Specify the measurement level: Match the distance metric to the label scale using the table above. Treating an ordinal scale as nominal understated agreement by 21 points in the worked example.
  3. Build the reliability matrix: Put annotators in rows, items in columns, and leave blanks for missing ratings. Confirm that every item you want measured has at least two ratings and that rare categories appear often enough to estimate.
  4. Compute with a tested implementation: Use the Python krippendorff package, R's irr::kripp.alpha, or the KALPHA macro for SPSS and SAS (Hayes and Krippendorff, 2007) rather than hand-built code.
  5. Locate the disagreement: Recompute alpha with each annotator removed in turn, and again per category. A jump when one annotator is dropped points to a calibration problem. A low per-category score points to a guideline boundary.
  6. Fix the cause and re-measure: Revise guidelines with boundary examples, recalibrate annotators, and recompute on a fresh sample. For subjective tasks, persistent disagreement may be signal rather than error, so keep the label distribution instead of forcing consensus (Aroyo and Welty, 2015).

To see how these controls run during live annotation, read about ADAP quality flow and test questions →.

How Appen supports inter-annotator agreement measurement

Alpha diagnoses a reliability problem. Fixing it depends on how the labeling program is designed. Appen builds agreement into that design so research teams receive data they can defend in a paper, a model card or an audit.

  • Golden sets for calibration: Subject matter experts adjudicate a reference set before production. This gives annotators a fixed target and gives you a baseline for measuring drift.
  • Test questions in ADAP: The AI Data Platform (ADAP) mixes hidden test questions into live work and removes contributors whose accuracy falls below the project threshold.
  • Redundant judgments by design: Several independent judgments per item produce the complete reliability matrix that alpha needs, instead of spot-checked samples.
  • Targeted contributor recruiting: Contributors are recruited by region, device and channel across 500+ locales, so disagreement reflects the task rather than a language or context mismatch.
  • Off-the-shelf datasets: Pre-built datasets in the Appen data catalog come with documented collection and annotation specifications.

To set reliability targets for your next annotation or evaluation project, plan your annotation quality program with our team.

Frequently asked questions about Krippendorff's alpha

What is a good Krippendorff's alpha score?

Treat 0.800 as the minimum for data you will train on or report, and 0.667 as the minimum for exploratory work. Judge the lower end of a bootstrapped 95% confidence interval rather than the point estimate. An alpha of 0.82 with a lower bound of 0.74 hasn't cleared 0.800. Stricter minimums make sense for safety or medical labels, where one systematic error is costly.

Can Krippendorff's alpha be negative?

Yes. A negative alpha means annotators disagree more than chance would predict, which points to a systematic cause rather than random noise. Common causes are an inverted scale (one group reading 1 as best), swapped label definitions, or a guideline that two annotator pools read differently. The lower bound depends on the data and sample size, so investigate any value below 0 before relabeling (Krippendorff, 2011).

How many raters and items does Krippendorff's alpha need?

Alpha works with two or more raters, and raters don't need to label every item. Items with only one rating are simply excluded. The limiting factor is usually the number of items, not raters. Each category needs enough pairable values for a stable estimate, so rare classes need larger samples. Krippendorff (2018) gives minimum sample-size guidance based on the number of categories and the confidence required.

Can Krippendorff's alpha be used for multi-label annotation?

Not with the standard nominal setting, which assumes one value per item per rater. For set-valued labels, such as an image tagged "person, bicycle," use a set-based distance such as measuring agreement on set-valued items (MASI). MASI gives partial credit for overlapping sets (Passonneau, 2006). You can also compute a binary alpha per label, which shows which tags drive disagreement.

Can Krippendorff's alpha measure LLM-as-a-judge agreement with humans?

Yes. Treat the model as one more rater and compute alpha twice: once for humans only and once with the judge included. If adding the judge lowers alpha, the judge diverges from human consensus. If alpha holds steady, the judge behaves like another calibrated annotator. Run the comparison for each rubric criterion, because agreement can differ sharply between criteria. Explore LLM-as-a-judge rubric design and AI research services.

References

Tags:
Data Annotation
Human Evaluation
Training Data
Talk to our team