Inter-Rater Reliability and Cohen’s Kappa: A Complete Methods Guide for Researchers (2026)
When two or more researchers code the same dataset — whether classifying patient symptoms, rating interview transcripts, or labelling images for machine learning — the credibility of the entire study rests on a single question: do the raters agree? Inter-rater reliability (IRR), and its most widely deployed coefficient, Cohen’s kappa (κ), provide the quantitative answer. Without a rigorous IRR assessment, any subsequent analysis — thematic, statistical, or inferential — is built on an unverifiable foundation. Yet IRR remains one of the most frequently misapplied concepts in empirical research, with many researchers still relying on raw percentage agreement and misreading kappa benchmarks published half a century ago.
This guide offers a comprehensive treatment of inter-rater reliability Cohen’s kappa suitable for PhD candidates, postdoctoral researchers, and journal reviewers. It covers the mathematical derivation of κ from first principles, a fully worked numerical example, the authoritative Landis & Koch (1977) interpretation scale with its documented limitations, extensions to weighted kappa and Fleiss’ kappa for multiple raters, Krippendorff’s alpha for continuous and ordinal data, and step-by-step reporting templates for APA 7. Software syntax for R and SPSS is included throughout.
IRR is one component of the broader methodological reliability challenge in empirical science. The systemic consequences of poor measurement reliability — including inflated false-positive rates and replication failure — are documented extensively in our analysis of the reproducibility crisis in research.
Cohen’s kappa (κ) measures the agreement between two raters on categorical data, corrected for chance. Values are interpreted using the Landis & Koch (1977) scale: ≤0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, 0.81–1.00 almost perfect. Most peer-reviewed journals require κ ≥ 0.61 for minimal acceptance; health-sciences standards often require κ ≥ 0.75 (McHugh, 2012). Use weighted kappa for ordinal categories, Fleiss’ kappa for three or more raters, and Krippendorff’s alpha when data types are mixed or data are missing.
1. What Is Inter-Rater Reliability and Why Does It Matter?
Video: Estimating Inter-Rater Reliability with Cohen’s Kappa in SPSS — Dr. Todd Grande (1.6M subscribers)
Inter-rater reliability refers to the degree of consistency or agreement among independent judges who rate, classify, or score the same set of observations. It is a fundamental index of measurement quality in any study where human judgement is involved in data generation — a condition that applies to an enormous range of research contexts: clinical diagnosis, content analysis, systematic review screening, behavioural observation, qualitative coding, radiological assessment, and annotation pipelines for artificial intelligence.
From a validity perspective, low IRR is not merely a procedural inconvenience. If two trained coders disagree substantially on category assignments, at least one — and possibly both — are introducing systematic or random error into the dataset. Downstream statistical conclusions drawn from such data carry an inflated false-positive risk. High IRR does not guarantee construct validity, but it is a necessary precondition: a measurement instrument cannot be valid if it is not first reliable.
IRR assessment is distinct from but related to test-retest reliability (consistency of a single rater across time) and internal consistency (homogeneity of items within a scale). When writing a dissertation methodology chapter, researchers must distinguish these forms of reliability and justify the appropriate coefficient for their study design.
In qualitative research methods that involve systematic coding — such as grounded theory, framework analysis, or content analysis — IRR provides the bridge between interpretive richness and methodological rigour. Reviewers at high-impact journals routinely reject manuscripts that fail to report IRR with adequate detail.
2. The Pitfalls of Percentage Agreement
Before Cohen’s kappa was introduced, researchers commonly reported percentage agreement (PA): the proportion of observations on which two raters agree, expressed as a percentage. PA is intuitive and easy to compute, but it has a critical flaw: it does not account for the level of agreement that would be expected by chance alone.
Consider a study in which two raters independently classify 100 patient records as either “high risk” or “low risk,” and 90% of records actually belong to the “low risk” category. Even if both raters categorise every record randomly (i.e., with zero diagnostic skill), they will agree on approximately 82% of cases simply because both will label most records “low risk” by chance (0.9 × 0.9 + 0.1 × 0.1 = 0.82). A reported PA of 82% would be entirely misleading.
This inflation problem is particularly severe when category distributions are skewed — a phenomenon known as the base-rate problem or prevalence effect. McHugh (2012) demonstrated that in healthcare studies, accepting PA alone without a chance-corrected coefficient means that “40% to 50% of the data being analysed are erroneous” in studies with moderate IRR levels.
Cohen (1960) proposed kappa precisely to solve this problem by subtracting the agreement expected by chance from the observed agreement and expressing the result as a proportion of the maximum improvement over chance possible.
3. Cohen’s Kappa: Formula and Derivation
Cohen’s kappa is defined as:
Where:
- Po = the observed proportion of agreement (i.e., the proportion of items on which the two raters agree)
- Pe = the proportion of agreement expected by chance, based on the marginal distributions of each rater’s ratings
- (1 − Pe) = the maximum agreement possible above chance
The expected agreement Pe is computed from the marginal totals of the contingency matrix:
where ni+ is the row marginal total for category i, n+i is the column marginal total for category i, and N is the total number of observations.
Kappa ranges from −1 to +1. A value of +1 indicates perfect agreement; 0 indicates agreement no better than chance; negative values indicate systematic disagreement (raters actively avoid each other’s categories). In practice, negative kappa values are rare and typically signal a methodological problem such as label ambiguity or training inconsistency.
4. Worked Calculation: Step-by-Step Example
Consider a content-analysis study in which two coders independently classify 120 news articles into three categories: Positive (P), Neutral (N), and Negative (G). Their ratings produce the following contingency matrix (rows = Rater A, columns = Rater B):
| Rater A Rater B | P | N | G | Row total |
|---|---|---|---|---|
| P | 30 | 5 | 2 | 37 |
| N | 4 | 42 | 3 | 49 |
| G | 2 | 3 | 29 | 34 |
| Column total | 36 | 50 | 34 | 120 |
Step 1: Compute Po
Agreements lie on the diagonal: 30 + 42 + 29 = 101.
Po = 101 / 120 = 0.842
Step 2: Compute Pe
- For category P: (37/120) × (36/120) = 0.308 × 0.300 = 0.0925
- For category N: (49/120) × (50/120) = 0.408 × 0.417 = 0.1701
- For category G: (34/120) × (34/120) = 0.283 × 0.283 = 0.0803
- Pe = 0.0925 + 0.1701 + 0.0803 = 0.343
Step 3: Compute κ
κ = (0.842 − 0.343) / (1 − 0.343) = 0.499 / 0.657 = 0.760
Using the Landis & Koch (1977) scale, κ = 0.760 falls in the substantial agreement range (0.61–0.80). This indicates that the two coders apply the three-category scheme consistently, with agreement substantially exceeding chance — a level acceptable for publication in most social science journals.
5. Interpreting Kappa: Landis & Koch (1977) and Beyond
The most widely cited benchmarks for kappa interpretation were proposed by Landis & Koch (1977) in their paper on the measurement of observer agreement for categorical data. Their scale has become the de facto standard across disciplines, appearing in tens of thousands of published studies.
| κ Range | Landis & Koch (1977) | McHugh (2012) — Healthcare Focus |
|---|---|---|
| < 0.00 | No agreement | No agreement |
| 0.01–0.20 | Slight | None to slight |
| 0.21–0.40 | Fair | Minimal |
| 0.41–0.60 | Moderate | Weak |
| 0.61–0.74 | Substantial | Moderate |
| 0.75–0.84 | Substantial / Almost perfect | Good |
| 0.85–0.94 | Almost perfect | Very good |
| 0.95–1.00 | Almost perfect | Excellent |
It is essential to acknowledge an important limitation of the Landis & Koch scale that is often overlooked in methods sections: as documented in the original 1977 paper, the thresholds were based on the authors’ personal opinion rather than on empirical evidence. Subsequent methodologists have noted that these benchmarks may be inappropriately applied across contexts with very different error tolerances. In clinical diagnosis or systematic review screening, κ values in the “moderate” range by Landis & Koch standards may represent unacceptably high rates of miscoding. McHugh (2012) explicitly advises that kappa below 0.60 implies that “40% to 50% of the data being analysed are erroneous,” rendering statistical analysis unreliable.
Researchers should therefore cite the benchmark scale they are using, justify its application to their study context, and consider reporting confidence intervals around κ rather than treating a point estimate as sufficient.
6. Weighted Kappa for Ordinal Data
Standard Cohen’s kappa treats all disagreements as equivalent: a one-category misclassification counts the same as a maximum disagreement. When the categories are ordinal — for example, pain severity ratings from 1 (none) to 5 (severe) — this is inappropriate. A coder rating a patient as “3 — moderate” when the other coder assigns “4 — moderately severe” is making a much smaller error than one who assigns “1 — none.”
Cohen (1968) introduced weighted kappa (κw) to address this. Weights are assigned to each cell of the contingency matrix based on the magnitude of disagreement. Two weighting schemes are standard:
- Linear weights: wij = 1 − (|i − j| / (k − 1)), where k is the number of categories. Disagreements one step apart receive a partial credit proportional to their proximity.
- Quadratic weights: wij = 1 − ((i − j)² / (k − 1)²). Larger disagreements are penalised more heavily. This is equivalent to the intraclass correlation coefficient (ICC) for two raters on ordinal data, making it preferable when the intervals between categories are assumed to be equal.
For most rating scales used in psychology, education, and health research, quadratic weighting is preferred because it reflects the disproportionate methodological cost of large disagreements. When using qualitative data coding schemes that involve ordered severity levels or frequency anchors, weighted kappa should be the default reporting choice.
7. Fleiss’ Kappa for More Than Two Raters
Cohen’s original kappa is defined for exactly two raters. In practice, many content analysis and systematic review workflows involve three or more independent coders — a design that increases reliability estimates but requires an extended formula.
Fleiss (1971) generalised kappa to handle multiple raters with a different subset of raters per item (i.e., not all raters need rate all items). The formula is:
Where P̄o and P̄e are mean observed and expected agreement values computed across all items and all rater pairs. Fleiss’ kappa is available in R via the irr package (kappam.fleiss()) and in SPSS via the Reliability procedure with multiple raters defined.
A key limitation of Fleiss’ kappa is that it produces a single overall reliability estimate and does not provide category-specific or rater-specific information. When diagnostic screening or systematic review title-and-abstract screening involves three coders, Fleiss’ kappa is the appropriate statistic; however, researchers are also advised to report pairwise Cohen’s kappa values between each rater pair to identify which dyads are driving any agreement shortfall.
For grounded theory studies using multiple coders to develop and apply a coding scheme, Fleiss’ kappa provides a defensible single IRR value suitable for reporting in the methodology chapter.
8. Krippendorff’s Alpha: The Most Flexible IRR Coefficient
Krippendorff’s alpha (α) was developed in the context of communication and content analysis research (Krippendorff, 1970, 2004) and has become the preferred coefficient in disciplines where data do not fit the assumptions of Cohen’s kappa. Its principal advantages are:
- Data type flexibility: Unlike kappa, alpha adapts to nominal, ordinal, interval, and ratio scales through a corresponding distance metric applied to the disagreement matrix.
- Missing data handling: Alpha can be computed when not all raters assess all items — a common situation in annotation projects and content analysis studies with large item pools.
- Scalability to many raters: Alpha extends naturally beyond two raters without requiring the averaging approximation used by Fleiss’ kappa.
The general formula is:
Where Do is the observed disagreement and De is the disagreement expected by chance, both computed using the appropriate distance function for the data type (e.g., squared difference for interval data, absolute difference for ordinal data).
Krippendorff (2004) recommends α ≥ 0.80 as the threshold for conclusions that will be treated as reliable, with α between 0.667 and 0.80 acceptable only for tentative conclusions. In machine learning annotation tasks where multiple workers label large corpora, Krippendorff’s alpha has become the standard reporting metric.
In R, alpha is computed via kripp.alpha() in the irr package. In Python, the nltk package provides AnnotationTask with alpha support.
9. How to Report Inter-Rater Reliability in APA 7
APA 7 (American Psychological Association, 2020) does not prescribe a single mandatory format for IRR statistics, but professional convention and reviewer expectations require the following elements:
- State the coefficient used and justify the choice: Name the statistic (Cohen’s kappa, weighted kappa, Fleiss’ kappa, or Krippendorff’s alpha), identify the data type, and cite the originating methodological paper.
- Report the value to two decimal places: APA recommends rounding to two decimal places for correlation-type coefficients. Do not use a leading zero (κ = .76, not κ = 0.76) — kappa cannot exceed 1.
- Include a 95% confidence interval: Confidence intervals are now expected by most high-impact journals. Bootstrap CIs are acceptable when parametric assumptions are uncertain.
- Cite the interpretation scale used: Reference Landis & Koch (1977) or McHugh (2012) explicitly; do not leave the interpretation implicit.
- Report the percentage of items double-coded: Specify the proportion of the corpus independently rated by both coders (ideally 20–30% minimum for large datasets).
- Describe the adjudication procedure: Explain how disagreements were resolved — by consensus, third-rater arbitration, or exclusion.
Example APA 7 reporting statement (for the worked example in Section 4):
For thematic analysis studies that use structured coding frameworks, this reporting template should appear in the Data Analysis subsection of the Methodology chapter, immediately after the description of the coding procedure.
10. Software Implementation: R and SPSS
Cohen’s Kappa in R
The irr package (Gamer et al., 2019) is the standard library for IRR in R and covers Cohen’s kappa, weighted kappa, Fleiss’ kappa, and Krippendorff’s alpha.
# Install and load the irr package
install.packages("irr")
library(irr)
# Two raters, n = 120 items — ratings as factor vectors
rater_A <- c(rep("P",37), rep("N",49), rep("G",34))
rater_B <- c(rep("P",30), rep("P",5), rep("P",2),
rep("N",4), rep("N",42), rep("N",3),
rep("G",2), rep("G",3), rep("G",29))
ratings <- cbind(rater_A, rater_B)
# Cohen's unweighted kappa
kappa2(ratings)
# Weighted kappa (quadratic weights — default)
kappa2(ratings, weight = "squared")
# Fleiss' kappa for 3 raters
# kappam.fleiss(ratings_matrix_3_raters)
# Krippendorff's alpha (nominal data)
kripp.alpha(t(ratings), method = "nominal")Cohen’s Kappa in SPSS
In SPSS, navigate to Analyze → Descriptive Statistics → Crosstabs. Place Rater A in the row box and Rater B in the column box. Under Statistics, tick Kappa. The output table will display Cohen’s κ, its standard error, and an asymptotic significance test. For weighted kappa, use the Analyze → Scale → Reliability Analysis pathway with the Intraclass correlation coefficient option, selecting the two-way mixed model with absolute agreement — this is algebraically equivalent to quadratic weighted kappa for two raters.
Researchers should note that SPSS does not natively compute Krippendorff’s alpha or Fleiss’ kappa; these require the R interface or dedicated macros (e.g., Hayes & Krippendorff, 2007, available at afhayes.com).
Frequently Asked Questions
What is a good value for Cohen’s kappa?
Using the Landis & Koch (1977) scale, κ ≥ 0.61 (substantial agreement) is the generally accepted minimum for publication in social and behavioural sciences. Health sciences and clinical research typically require κ ≥ 0.75 (McHugh, 2012). For systematic review screening, the Cochrane Handbook recommends κ ≥ 0.61 before proceeding without arbitration. Values above 0.80 are considered strong agreement across virtually all disciplines.
What is the difference between Cohen’s kappa and weighted kappa?
Cohen’s unweighted kappa treats all disagreements as equal — a one-step error is penalised identically to a maximum-distance error. Weighted kappa assigns partial credit to near-agreement, making it appropriate for ordinal scales (e.g., Likert scales, severity ratings). Quadratic weighting is most common in health research; linear weighting is used when all adjacent-category errors are considered equally problematic. If your categories are purely nominal with no natural ordering, use unweighted kappa.
When should I use Krippendorff’s alpha instead of kappa?
Krippendorff’s alpha is preferable when (a) you have more than two raters and not all raters code all items, (b) your data are interval or ratio rather than nominal or ordinal, (c) you need a single unified reliability statistic across a large annotation workforce, or (d) you are working in content analysis, communication studies, or computational linguistics where alpha is the disciplinary norm. If you have exactly two raters and complete nominal data, Cohen’s kappa and Krippendorff’s alpha will produce very similar values.
What percentage of data should be double-coded for IRR?
There is no universal rule, but common practice in social science is to independently double-code a random 20–30% sample of the corpus for large datasets (n > 500), or to double-code the entire corpus for smaller studies. Systematic review methodology (e.g., Cochrane, PRISMA 2020) requires independent dual screening at the title-and-abstract stage for all records. The sample used for IRR calculation should be explicitly described in the methods section, including how items were selected and whether the sample was representative of the full dataset’s topic distribution.
Why is my kappa low even though percentage agreement is high?
This is the prevalence effect: when one category dominates the distribution (e.g., 90% of items are “negative”), both raters will agree by chance on most items simply by defaulting to the majority category. Kappa correctly identifies that this high surface agreement contains little diagnostic information. Solutions include computing PABAK (prevalence-adjusted, bias-adjusted kappa), stratifying the IRR analysis by category, or reporting category-specific agreement statistics alongside overall kappa.
Do I need to report IRR for qualitative thematic analysis?
This is a contested area. In codebook thematic analysis (Braun & Clarke, 2021) and framework analysis where a structured, predefined coding scheme is applied, IRR is both appropriate and expected by peer reviewers. In fully reflexive thematic analysis, some methodologists — including Braun and Clarke themselves — argue that inter-rater reliability is philosophically incompatible with the paradigmatic assumptions of the approach, because themes are not discovered but constructed. Researchers should state their position explicitly in the methodology chapter and cite methodological literature to support it.
References
- Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104
- Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220.
- Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382.
- Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology (2nd ed.). Sage.
- Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.
- McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282. https://doi.org/10.11613/BM.2012.031
- Gamer, M., Lemon, J., Fellows, I., & Singh, P. (2019). irr: Various Coefficients of Interrater Reliability and Agreement (R package version 0.84.1). CRAN. https://CRAN.R-project.org/package=irr
- American Psychological Association. (2020). Publication Manual of the American Psychological Association (7th ed.). https://doi.org/10.1037/0000165-000
Write your thesis with AI
Structure, draft, cite, and format your thesis faster with Tesify’s AI writing tools, automatic bibliography, and plagiarism checker. Free to start, no credit card required.
Leave a Reply