Cohen’s Kappa Calculator
Enter how two raters classified the same cases to measure their agreement beyond chance. The calculator gives Cohen’s kappa with its standard error, confidence interval and p-value, linear or quadratic weighted kappa for ordered categories, the observed and chance agreement, and a sentence to paste into your methods or results.
Cohen’s kappa
95% CI 0.56 to 0.84
Substantial agreement on the Landis and Koch scale. The raters agreed on 85.0% of cases, against 50.0% expected by chance.
- Kappa (κ)
- 0.7000
- 95% CI 0.561 to 0.839
- Observed agreement
- 85.0%
- diagonal cells
- Chance agreement
- 50.0%
- from the marginal totals
- Standard error
- 0.0711
- for the interval
- z (test of κ = 0)
- 7.035
- SE₀ = 0.0995
- p (two-sided)
- < 0.0001
- Maximum possible κ
- 0.900
- given these marginal totals
- Cases rated
- 100
- 2 categories
Report it like this
Agreement between the two raters on 100 cases (2 categories: Yes, No) was substantial, Cohen’s kappa κ = .70, 95% CI .56 to .84, p < .001; observed agreement 85.0% against 50.0% expected by chance.
How to use the calculator
- Set the categories. Use the + and − buttons for 2 to 6 categories and type over the labels. Both raters must use the same categories.
- Fill in the table. Each cell counts the cases rater A put in the row category and rater B put in the column category. The diagonal holds the agreements.
- Choose weights if the categories are ordered (for example mild, moderate, severe). Leave kappa unweighted for unordered categories.
- Copy the report sentence.
Cohen’s kappa is for exactly two raters who each rate every case. For three or more raters, Fleiss’ kappa is used instead; for continuous measurements, use the intraclass correlation coefficient.
Kappa formula
κ = (po − pe) / (1 − pe)
po is the proportion of cases on which the raters agree. pe is the agreement expected if they rated independently with their own category frequencies: the sum over categories of (row total × column total) / N². Kappa is 1 for perfect agreement, 0 for agreement no better than chance and negative for worse than chance.
Weighted kappa gives partial credit for near misses. With k categories, the weight for rating categories i and j is 1 − |i − j|/(k − 1) (linear) or 1 − (i − j)²/(k − 1)² (quadratic). Quadratic weighted kappa is close to the intraclass correlation coefficient.
The standard error for the confidence interval and the separate standard error for the test of κ = 0 follow Fleiss, Cohen and Everitt (1969), the method used by statsmodels, SPSS and most statistics packages.
Worked examples
- Yes/no ratings (the default): two raters agree on 45 “yes” and 40 “no” cases out of 100, so po = 0.85. Rater A said yes 50 times and rater B 55 times, so pe = 0.50 × 0.55 + 0.50 × 0.45 = 0.50. κ = (0.85 − 0.50) / (1 − 0.50) = 0.70, 95% CI 0.56 to 0.84: substantial agreement.
- Ordered severity: for the mild/moderate/severe example (100 cases), unweighted κ = 0.65 (0.53 to 0.78), linear weighted κ = 0.71 (0.60 to 0.82) and quadratic weighted κ = 0.77 (0.67 to 0.88). Weighting raises kappa because most disagreements are between neighbouring categories.
- The kappa paradox: with 85 “normal” and 4 “abnormal” agreements out of 100, raw agreement is 89%, but kappa is only 0.36 because chance agreement is already 83% when nearly every case is normal.
How to interpret kappa
The most cited labels are those of Landis and Koch (1977):
| Kappa | Strength of agreement |
|---|---|
| Below 0 | Poor (less than chance) |
| 0.00 to 0.20 | Slight |
| 0.21 to 0.40 | Fair |
| 0.41 to 0.60 | Moderate |
| 0.61 to 0.80 | Substantial |
| 0.81 to 1.00 | Almost perfect |
The cut-offs were arbitrary, and McHugh (2012) argues they are too lenient for health research, where a kappa below 0.60 means much of the data may be wrong. Kappa also depends on prevalence: when one category is very common, kappa can be low even though the raters almost always agree. Report the observed agreement and the table alongside kappa so readers can judge.
How to report Cohen’s kappa
Name the type of kappa, give its value with a confidence interval, and describe the raters and the number of cases:
Two independent reviewers screened 100 records. Agreement was substantial, Cohen’s κ = .70, 95% CI .56 to .84, p < .001 (observed agreement 85.0%).
Systematic reviews often report kappa for title and abstract screening; the PRISMA flow diagram generator and PRISMA 2020 checklist cover the rest of the reporting. For agreement between a test and a reference standard rather than between raters, use the sensitivity and specificity calculator.
Questions
- How do I calculate Cohen’s kappa?
- Work out the observed agreement (the share of cases on the diagonal) and the chance agreement (the sum of row total × column total ÷ N² for each category). Then kappa = (observed − chance) ÷ (1 − chance). With 85% observed and 50% chance agreement, kappa = 0.35 ÷ 0.50 = 0.70.
- What is a good kappa value?
- By the Landis and Koch labels, 0.61 to 0.80 is substantial and above 0.80 almost perfect agreement. Many health researchers want at least 0.60, and above 0.80 for decisions that affect patients. Always interpret kappa together with the observed agreement and the category totals.
- When should I use weighted kappa?
- Use weighted kappa when the categories are ordered, such as grades or severity levels, so that disagreeing by one step counts as less serious than disagreeing by two. Quadratic weights are the most common choice. For unordered categories, use unweighted kappa.
- Why is kappa low when agreement is high?
- When one category is very common, two raters agree often by chance alone, so kappa, which only counts agreement beyond chance, can be low. This is known as the kappa paradox. Report the observed agreement, and consider also giving positive and negative agreement separately.
- What is the difference between Cohen’s kappa and Fleiss’ kappa?
- Cohen’s kappa measures agreement between exactly two raters who rate the same cases. Fleiss’ kappa extends the idea to three or more raters, and allows different raters for different cases.
- Can kappa be negative?
- Yes. A negative kappa means the raters agreed less often than chance would predict, which usually points to a misunderstanding of the categories or a data entry error, such as reversed codes for one rater.
Sources
- Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20(1):37–46. doi:10.1177/001316446002000104
- Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull 1968;70(4):213–220. doi:10.1037/h0026256
- Fleiss JL, Cohen J, Everitt BS. Large sample standard errors of kappa and weighted kappa. Psychol Bull 1969;72(5):323–327. doi:10.1037/h0028106
- Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33(1):159–174. doi:10.2307/2529310
- McHugh ML. Interrater reliability: the kappa statistic. Biochem Med 2012;22(3):276–282. PMCID: PMC3900052
- Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol 1990;43(6):543–549. doi:10.1016/0895-4356(90)90158-L
Calculations run in your browser; nothing you enter is sent or stored. Results are checked against SciPy and statsmodels in our test suite. Last updated 22 November 2026.