Cohen’s Kappa Calculator

Enter how two raters classified the same cases to measure their agreement beyond chance. The calculator gives Cohen’s kappa with its standard error, confidence interval and p-value, linear or quadratic weighted kappa for ordered categories, the observed and chance agreement, and a sentence to paste into your methods or results.

Ratings: rater A in rows, rater B in columns

Agreement table of rater A by rater B
A ↓ · B →Total
50
50
Total5545100

Cells on the diagonal are cases where both raters agreed. Type over a label to rename a category for both raters.

Confidence level

Load a worked example

Cohen’s kappa

0.70095% CI 0.56 to 0.84

Substantial agreement on the Landis and Koch scale. The raters agreed on 85.0% of cases, against 50.0% expected by chance.

Kappa (κ)
0.7000
95% CI 0.561 to 0.839
Observed agreement
85.0%
diagonal cells
Chance agreement
50.0%
from the marginal totals
Standard error
0.0711
for the interval
z (test of κ = 0)
7.035
SE₀ = 0.0995
p (two-sided)
< 0.0001
Maximum possible κ
0.900
given these marginal totals
Cases rated
100
2 categories

Report it like this

Agreement between the two raters on 100 cases (2 categories: Yes, No) was substantial, Cohen’s kappa κ = .70, 95% CI .56 to .84, p < .001; observed agreement 85.0% against 50.0% expected by chance.

How to use the calculator

  1. Set the categories. Use the + and − buttons for 2 to 6 categories and type over the labels. Both raters must use the same categories.
  2. Fill in the table. Each cell counts the cases rater A put in the row category and rater B put in the column category. The diagonal holds the agreements.
  3. Choose weights if the categories are ordered (for example mild, moderate, severe). Leave kappa unweighted for unordered categories.
  4. Copy the report sentence.

Cohen’s kappa is for exactly two raters who each rate every case. For three or more raters, Fleiss’ kappa is used instead; for continuous measurements, use the intraclass correlation coefficient.

Kappa formula

κ = (po − pe) / (1 − pe)

po is the proportion of cases on which the raters agree. pe is the agreement expected if they rated independently with their own category frequencies: the sum over categories of (row total × column total) / N². Kappa is 1 for perfect agreement, 0 for agreement no better than chance and negative for worse than chance.

Weighted kappa gives partial credit for near misses. With k categories, the weight for rating categories i and j is 1 − |i − j|/(k − 1) (linear) or 1 − (i − j)²/(k − 1)² (quadratic). Quadratic weighted kappa is close to the intraclass correlation coefficient.

The standard error for the confidence interval and the separate standard error for the test of κ = 0 follow Fleiss, Cohen and Everitt (1969), the method used by statsmodels, SPSS and most statistics packages.

Worked examples

  • Yes/no ratings (the default): two raters agree on 45 “yes” and 40 “no” cases out of 100, so po = 0.85. Rater A said yes 50 times and rater B 55 times, so pe = 0.50 × 0.55 + 0.50 × 0.45 = 0.50. κ = (0.85 − 0.50) / (1 − 0.50) = 0.70, 95% CI 0.56 to 0.84: substantial agreement.
  • Ordered severity: for the mild/moderate/severe example (100 cases), unweighted κ = 0.65 (0.53 to 0.78), linear weighted κ = 0.71 (0.60 to 0.82) and quadratic weighted κ = 0.77 (0.67 to 0.88). Weighting raises kappa because most disagreements are between neighbouring categories.
  • The kappa paradox: with 85 “normal” and 4 “abnormal” agreements out of 100, raw agreement is 89%, but kappa is only 0.36 because chance agreement is already 83% when nearly every case is normal.

How to interpret kappa

The most cited labels are those of Landis and Koch (1977):

Landis and Koch interpretation of kappa
KappaStrength of agreement
Below 0Poor (less than chance)
0.00 to 0.20Slight
0.21 to 0.40Fair
0.41 to 0.60Moderate
0.61 to 0.80Substantial
0.81 to 1.00Almost perfect

The cut-offs were arbitrary, and McHugh (2012) argues they are too lenient for health research, where a kappa below 0.60 means much of the data may be wrong. Kappa also depends on prevalence: when one category is very common, kappa can be low even though the raters almost always agree. Report the observed agreement and the table alongside kappa so readers can judge.

How to report Cohen’s kappa

Name the type of kappa, give its value with a confidence interval, and describe the raters and the number of cases:

Two independent reviewers screened 100 records. Agreement was substantial, Cohen’s κ = .70, 95% CI .56 to .84, p < .001 (observed agreement 85.0%).

Systematic reviews often report kappa for title and abstract screening; the PRISMA flow diagram generator and PRISMA 2020 checklist cover the rest of the reporting. For agreement between a test and a reference standard rather than between raters, use the sensitivity and specificity calculator.

Questions

How do I calculate Cohen’s kappa?
Work out the observed agreement (the share of cases on the diagonal) and the chance agreement (the sum of row total × column total ÷ N² for each category). Then kappa = (observed − chance) ÷ (1 − chance). With 85% observed and 50% chance agreement, kappa = 0.35 ÷ 0.50 = 0.70.
What is a good kappa value?
By the Landis and Koch labels, 0.61 to 0.80 is substantial and above 0.80 almost perfect agreement. Many health researchers want at least 0.60, and above 0.80 for decisions that affect patients. Always interpret kappa together with the observed agreement and the category totals.
When should I use weighted kappa?
Use weighted kappa when the categories are ordered, such as grades or severity levels, so that disagreeing by one step counts as less serious than disagreeing by two. Quadratic weights are the most common choice. For unordered categories, use unweighted kappa.
Why is kappa low when agreement is high?
When one category is very common, two raters agree often by chance alone, so kappa, which only counts agreement beyond chance, can be low. This is known as the kappa paradox. Report the observed agreement, and consider also giving positive and negative agreement separately.
What is the difference between Cohen’s kappa and Fleiss’ kappa?
Cohen’s kappa measures agreement between exactly two raters who rate the same cases. Fleiss’ kappa extends the idea to three or more raters, and allows different raters for different cases.
Can kappa be negative?
Yes. A negative kappa means the raters agreed less often than chance would predict, which usually points to a misunderstanding of the categories or a data entry error, such as reversed codes for one rater.

Sources