Inter-Rater Reliability Beyond Cohen’s Kappa 2026: ICC, Krippendorff’s Alpha & Bland-Altman

Cohen’s kappa is the measure most methodology textbooks reach for first when two raters classify the same set of units — and for good reason. Our guide on inter-rater reliability and Cohen’s kappa covers that foundation in full. But kappa has real limits: it is restricted to two raters, to categorical data, and to complete datasets with no missing values. The moment your design involves a continuous outcome, three or more coders, ordinal scales, or incomplete rating matrices, you need a different tool — and choosing the wrong one is a reportable methodological flaw that peer reviewers and thesis examiners will flag.

This guide maps the statistical landscape beyond Cohen’s kappa for 2026. It covers intraclass correlation coefficients (ICC) and the ten model variants described by Koo and Li (2016) in the Journal of Chiropractic Medicine; Krippendorff’s alpha for any measurement level; Bland-Altman limits of agreement for continuous clinical measurements; Fleiss’ kappa for panels of raters; and weighted kappa for ordered categories. A decision table at the end lets you select the right statistic in under sixty seconds.

Why Cohen’s Kappa Is Not Enough

Cohen’s kappa (κ) was introduced in 1960 as a chance-corrected measure of agreement between two raters on a nominal scale. Its formula corrects observed agreement for the level of agreement expected by chance alone, yielding values from −1 (perfect disagreement) through 0 (chance level) to +1 (perfect agreement). This design decision embeds two structural constraints that limit applicability.

First, kappa treats all disagreements equally. A rater who scores an item “1” when the correct score is “2” is penalised identically to a rater who scores it “5”. For ordinal scales — pain ratings, Likert scales, diagnostic severity grades — this is theoretically inappropriate. Second, the formula is defined only for two raters. A study with three independent coders cannot produce a single kappa; researchers sometimes average pairwise kappas, but this inflates apparent reliability and ignores rater interdependence.

Beyond those structural limits, kappa requires complete data. Every unit must be rated by every rater with no missing values. In content analysis of large corpora — where coders may rate different random subsamples — kappa simply cannot be computed. Krippendorff’s alpha handles exactly this scenario. And when the outcome is a continuous measurement (blood pressure, reaction time, rating on a 100-point visual analogue scale), kappa is wholly inappropriate; ICC and Bland-Altman are the appropriate alternatives.

The relationship between inter-rater reliability and broader construct, internal, and external validity is direct: unreliable rating is a source of random measurement error that attenuates observed effect sizes and inflates the probability of Type II errors. Reporting the wrong reliability index can obscure this problem from reviewers entirely.

Intraclass Correlation Coefficient: Choosing the Right Model

The ICC is a family of statistics, not a single coefficient. Koo and Li (2016) formalised ten distinct ICC forms, organised along three dimensions: the model (one-way random, two-way random, two-way mixed), the type (single rater vs. average of raters), and the definition (absolute agreement vs. consistency). Selecting the wrong form and reporting it simply as “ICC” is a methodological error.

The Three Model Dimensions

Model 1 — One-way random effects. Each subject is rated by a different set of raters drawn at random from a larger population of raters. This is appropriate when the raters in your study are not the same raters you would deploy in practice. Because rater effects are not distinguished from error, Model 1 tends to produce lower ICC values.

Model 2 — Two-way random effects. The same raters assess all subjects, and those raters are considered a random sample of all possible raters. This is the most defensible model when you want to generalise conclusions beyond your specific raters. Choose absolute agreement when you require that raters produce identical scores; choose consistency when systematic offset between raters does not matter (e.g., when scores will be adjusted to a common mean).

Model 3 — Two-way mixed effects. The same raters assess all subjects, but those raters are the only raters of interest — they are fixed, not a sample. Model 3 is appropriate when the study reports reliability only for those specific clinicians, examiners, or coders. Critically, Model 3 consistency and Model 2 consistency yield identical numerical values; the distinction lies in how you generalise the conclusion.

Single vs. Average Measures

Each model can report ICC for a single rater or for the average of k raters. Average measures ICC is analogous to using Spearman-Brown prophecy to project reliability after aggregating raters. If your final outcome is the mean of three raters’ scores, report ICC(average); if each subject receives one rater’s score in practice, report ICC(single). Reporting average-measures ICC when only one rater will be used in production inflates reported reliability substantially.

Step-by-step tutorial on calculating ICC for inter-rater reliability in SPSS. Source: Dr. Todd Grande (2M+ subscribers) on YouTube.

ICC Interpretation Benchmarks

Koo and Li (2016) recommend interpreting ICC estimates using the lower bound of the 95% confidence interval, not the point estimate alone. This matters because ICC point estimates can be unstable with small samples, and a confident-looking value of 0.82 may have a confidence interval spanning 0.55 to 0.94. The benchmark thresholds based on their framework are as follows: values below 0.50 indicate poor reliability; 0.50 to 0.74 indicate moderate reliability; 0.75 to 0.89 indicate good reliability; and values of 0.90 or above indicate excellent reliability.

Always report: the ICC model form in full notation, point estimate, 95% CI, F statistic, and degrees of freedom. Software such as SPSS (Analyze → Scale → Reliability Analysis) and R’s irr or psych packages produce all of these. Reporting only a point estimate without the CI is insufficient for most journals in clinical and social sciences.

Krippendorff’s Alpha: The Universal Agreement Coefficient

Krippendorff’s alpha (α) was developed within communication research and content analysis as a measure that generalises across four dimensions that constrain most other reliability indices: it accommodates any number of coders, any number of categories, any level of measurement (nominal, ordinal, interval, ratio), and incomplete data matrices where different coders rate different subsets of units. These properties make it the preferred choice in large-scale content analysis, qualitative coding studies, and annotation projects in computational linguistics.

The coefficient is defined as:

α = 1 − (D_o / D_e)

where D_o is the observed disagreement and D_e is the disagreement expected by chance. The metric of disagreement is adapted to the level of measurement: nominal data uses a simple difference function; ordinal data uses a rank-based function; interval and ratio data use squared differences weighted appropriately. This adaptation means that alpha for ordinal data penalises large disagreements more than small ones — a property that categorical kappa entirely lacks.

Krippendorff’s own interpretation guidelines, refined in his 2004 and 2019 work on Content Analysis, suggest requiring α ≥ .800 for conclusions with confidence, and treating α ≥ .667 as the floor for tentative conclusions. Values below .667 indicate that reliability is insufficient to support coding-based inferences. These thresholds are widely adopted in communication, political science, and media studies research, though some fields apply stricter standards.

A practical advantage worth emphasising: when two coders rate entirely non-overlapping subsets of a corpus (a common design to manage workload), pairwise kappa cannot be computed at all. Krippendorff’s alpha remains computable as long as some units share ratings across coders. This is one reason it is increasingly used in thematic analysis research involving multiple coding passes by different team members.

Computing Krippendorff’s Alpha

In R, the kripp.alpha() function in the irr package accepts a matrix of ratings (coders in rows, units in columns) and a method argument specifying the measurement level: "nominal", "ordinal", "interval", or "ratio". Missing values are passed as NA and handled automatically. Hayes and Krippendorff’s KALPHA macro for SPSS and SAS is also widely used and freely available.

Bland-Altman Limits of Agreement for Continuous Measurements

When two methods — or two raters — produce continuous numerical measurements of the same physical quantity, neither ICC nor kappa is the right primary analysis. The question is not “do raters rank subjects in the same order?” but “are the two methods interchangeable for clinical use?” This requires quantifying the magnitude of individual-level disagreement, and that is precisely what Bland and Altman’s 1986 Lancet paper — cited over 47,000 times as of 2021 — formalises.

The Bland-Altman method plots the difference between the two measurements (Method A − Method B) on the y-axis against the mean of the two measurements on the x-axis. The resulting plot reveals three things simultaneously: systematic bias (the mean difference d̄), the spread of random disagreement, and whether disagreement is constant across the measurement range or proportional to the magnitude of the quantity measured.

The limits of agreement are defined as d̄ ± 1.96 × SD(d), where SD(d) is the standard deviation of the differences. These limits contain approximately 95% of individual pairwise differences. Crucially, the decision about whether those limits are acceptable is a clinical or substantive judgment, not a statistical one: if measuring diastolic blood pressure, a limit of agreement of ±5 mmHg may be acceptable where ±20 mmHg is not, regardless of any p-value.

A key assumption of the standard Bland-Altman analysis is that the limits of agreement are constant across the measurement range. If the plot shows a funnel shape — differences growing with the magnitude of the mean — a log transformation or regression-based analysis is required. This is a common reporting gap in published papers, as noted in the reporting standards review by Gerke (2020) in Statistics in Medicine.

Fleiss’ Kappa and Weighted Kappa

Fleiss’ Kappa for Three or More Raters

Fleiss’ kappa extends Cohen’s kappa to panels of raters. Unlike Cohen’s κ, it does not require the same raters to assess all units — it accommodates designs where k raters are randomly assigned to each unit, as long as the number of raters per unit is constant. The statistic quantifies overall agreement across the panel beyond chance. Interpretation thresholds are broadly the same as Cohen’s kappa: values above 0.60 are conventionally considered substantial, and values above 0.80 are near-perfect.

A limitation of Fleiss’ kappa is that it, like Cohen’s kappa, treats all disagreements equally, making it unsuitable for ordinal outcomes. For an ordinal variable rated by three or more raters, Kendall’s W (coefficient of concordance) or Krippendorff’s alpha with method = “ordinal” are more appropriate. This is an underappreciated distinction: researchers in medical and educational contexts frequently apply Fleiss’ kappa to ordinal ratings — a practice that produces misleading inflated estimates when near-miss disagreements dominate.

Weighted Kappa for Ordered Categories

Weighted kappa (κ_w) extends Cohen’s κ to two-rater ordinal data by assigning penalty weights to disagreements. Two weighting schemes are in common use. Linear weights penalise disagreements proportionally to their distance on the ordinal scale (a two-category error is twice as bad as a one-category error). Quadratic weights penalise large disagreements disproportionately. With quadratic weights, weighted kappa is algebraically equivalent to the intraclass correlation coefficient under a two-way random model — a useful fact that unifies the two frameworks conceptually.

Choose linear weights when the scale is truly evenly spaced; choose quadratic weights when large misclassifications are substantially more costly than small ones. Always specify the weighting scheme in your methods section: “weighted kappa (quadratic weights)” rather than simply “weighted kappa.”

Decision Table: Which Statistic to Use

Reporting Requirements and Common Mistakes

Robust inter-rater reliability reporting requires more than stating a single coefficient. Reviewers in high-impact journals increasingly follow checklists that require the following elements, regardless of which statistic is chosen.

• Specify the statistic completely. “ICC” alone is ambiguous; write “ICC(2,1) absolute agreement, two-way random effects model” or “Krippendorff’s alpha, ordinal metric.”
• Report point estimate and 95% CI. For ICC, cite both. For Krippendorff’s alpha, bootstrapped CIs are available through Hayes’ KALPHA macro.
• State the number of raters and rated units. Sample size affects the precision of the ICC estimate; small samples produce wide confidence intervals that reviewers will penalise if unreported.
• Justify the threshold used. If you report ICC = 0.76 and claim “good reliability,” cite Koo and Li (2016). If you report α = 0.79 and note it falls short of Krippendorff’s .80 standard, explain the impact on interpretation.
• For Bland-Altman, provide the clinically acceptable difference a priori. Without a pre-specified criterion, the limits of agreement analysis cannot formally support a conclusion about interchangeability.

A common mistake in theses and systematic reviews is conflating test-retest reliability (same rater, same instrument, different time points) with inter-rater reliability. These are distinct psychometric properties requiring separate analyses. In systematic review contexts — particularly dual-independent screening — the funnel plot and publication bias assessment is also a phase where inter-rater reliability should be reported for study inclusion decisions, using the methods above.

The relationship between inter-rater reliability and the reproducibility crisis in research is direct: inadequate or miscommunicated reliability statistics contribute to findings that cannot be replicated, because the measurement instrument (the rating process) itself is a source of uncontrolled variance. Transparent reliability reporting is therefore not a box-ticking exercise — it is a prerequisite for reproducible science. For broader statistics guidance in your dissertation, see our treatment of effect size and confidence intervals, which addresses the complementary problem of over-reliance on p-values.

Frequently Asked Questions