---
title: 'Cohen’s Kappa: Agreement Beyond Chance'
url: https://www.emergentmind.com/topics/cohen-s-kappa
type: topic
---

# Cohen’s Kappa: Agreement Beyond Chance

Cohen’s kappa (κ) is a canonical chance-corrected index of pairwise agreement for categorical data. It quantifies the extent to which observed agreement between two raters (or a rater and a classifier) exceeds that expected under independent labeling drawn from their marginal distributions. κ is widely used in medicine, psychology, machine learning, and annotation-based research to evaluate inter-rater reliability or classifier validity, and has yielded a large critical literature on its formal structure, interpretability, and practical limitations. The following sections provide a detailed exposition of its definition, mathematical properties, connections to information theory, computational issues, alternatives, and empirical benchmarking frameworks.

## 1. Mathematical Definition and Computation

Suppose two raters independently classify $n$ items into $K$ mutually exclusive categories. Let $n_{ij}$ be the number of items that Rater 1 assigns to category $i$ and Rater 2 to $j$ ($1 \leq i,j \leq K$). The observed agreement is

$$
P_o = \frac{1}{n} \sum_{i=1}^K n_{ii} \,,
$$

and the chance agreement under independent draws from the marginal category distributions is

$$
P_e = \sum_{i=1}^K p_i q_i \text{ where } p_i = \frac{\sum_j n_{ij}}{n}, \quad q_i = \frac{\sum_j n_{ji}}{n} \,.
$$

Cohen’s kappa is then

$$
\kappa = \frac{P_o - P_e }{1 - P_e} \,.
$$

For binary labels, this specializes to:

$$
\kappa = \frac{p_o - p_e}{1 - p_e}\,, \quad p_o = \frac{TP + TN}{N}\,, \quad p_e = \pi \hat{\pi} + (1-\pi)(1-\hat{\pi})\,,
$$

where $\pi$ and $\hat{\pi}$ are the marginal positive rates for the two raters or systems [2606.00093], [2402.08069].

**Interpretation:** $\kappa=1$ denotes perfect agreement, $\kappa=0$ agreement no better than chance, and $\kappa<0$ systematic disagreement beyond chance.

The denominator $1-P_e$ ensures normalization between systematic mismatch (potentially negative lower bound) and perfect concordance ($\kappa=1$). For multi-class tasks this generalizes directly by forming the confusion or agreement matrix and substituting each $n_{ij}$ as appropriate [2504.15325], [2106.07393].

## 2. Statistical Properties, Bounds, and Relation to Other Measures

Cohen’s original work established both explicit upper and lower bounds for $\kappa$ given fixed marginals [2404.14149], [2006.12904]. The maximum possible agreement for fixed marginals $f(i),g(i)$ is $A_{\mathrm{max}} = \sum_i \min\{f(i),g(i)\}$, yielding an upper bound

$$
\kappa_{\mathrm{max}} = \frac{A_{\mathrm{max}}-P_e}{1-P_e}\,.
$$

However, the minimum feasible agreement $A_{\mathrm{min}}$—once an open question—admits the closed form $A_{\min} = \max\{0, f(K) + g(K) - 1\}$ after appropriate permutation of categories (so $f(K)g(K)$ is maximal) [2006.12904]. The tight lower bound is

$$
\kappa_{\min} = \frac{A_{\min} - P_e}{1 - P_e}\,,
$$

which can be substantially negative, especially under marginal imbalance.

For binary variables, there is a linear relationship to the Pearson correlation $\rho$:

$$
\kappa = \rho \cdot C(p_1, p_2),\quad C(p_1,p_2) = \frac{2\sqrt{p_1 p_2(1-p_1)(1-p_2)}}{p_1 + p_2 - 2p_1p_2}
$$

This constant depends only on marginals. $\kappa$ and the Matthews Correlation Coefficient/phi coefficient have identical numerators but different normalizations, leading to $|\kappa|\leq |\phi|$ with equality if marginals match [2404.14149], [2606.00093].

In the presence of abstentions, $\kappa$'s value depends on preprocessing choice—exclusion, recode-as-negative, or three-class extension—each answering distinct inferential questions [2606.00093].

## 3. Theoretical Interpretation and Information-Theoretic Connections

$\kappa$ formalizes the idea of “agreement beyond chance” under a permutation (hypergeometric) null model, guaranteeing $E[\kappa]=0$ for random labeling with fixed marginals [2601.10641]. This property persists only when the null holds marginals fixed across resamplings; in multinomial or other nulls where marginals are random, $E[\kappa]$ departs from zero and interpretability collapses [2601.10641].

Recent developments have anchored $\kappa$ to information-theoretic quantities. Notably, there is a smooth, monotonic relationship between $\kappa$ and the Resistor Average Distance (RAD) between class-conditional densities in classification [2403.01571]:

$$
\kappa = 1 - 2^{-R(P,Q)},
$$

where $R(P,Q)$ is the resistor average of Kullback–Leibler divergences:

$$
R(P,Q)=\frac{D(P\|Q)D(Q\|P)}{D(P\|Q)+D(Q\|P)}\,.
$$

Thus, maximum achievable $\kappa$ is a direct function of the intrinsic information separation between classes. Empirical studies demonstrate that observed $\kappa$ matches the $1-2^{-R}$ upper bound closely on both synthetic and real datasets, confirming this theoretical link [2403.01571].

Information Agreement (IA), an alternative based on mutual information, avoids the “chance” baseline and normalizes by the minimum entropy of marginal distributions; IA always lies in $[0,1]$ and is robust to prevalence/bias artifacts that affect $\kappa$ [2008.11701].

## 4. Limitations, Prevalence Paradox, and Critique

$\kappa$’s behavior is unintuitive under prevalence or marginal imbalance, known as the “prevalence paradox”: for highly imbalanced marginals, $\kappa$ can be low even with near-perfect agreement on the dominant class [2106.07393], [2203.09628]. Moreover, $\kappa$'s lower bound is not always $-1$; extremely skewed marginals may force $\kappa$ only slightly negative or near zero even for maximal disagreement [2006.12904].

Empirical analysis over $>10^6$ 2×2 tables confirms:

- Severe underestimation of agreement under extreme prevalence
- Collapse or instability of $\kappa$ when row or column marginals are near zero
- High type I/II rates under moderate observed agreement
- Very similar “handicapped” behavior to Pearson’s $r$, and other classical measures

Consequently, Holley & Guilford’s $G$ and Gwet’s $AC_1$ have been recommended as more reliable alternatives in dichotomous settings, with $AC_1$ correcting the prevalence effect and correlating near-perfectly with $G$ [2203.09628].

## 5. Interpretation, Significance Indices, and Reporting Guidelines

Raw $\kappa$ values are widely classified into “slight,” “fair,” “moderate,” “substantial,” “almost perfect” following ad hoc thresholds (e.g., Landis–Koch), but these scales are arbitrary and not functionally linked to sampling variability, number of categories, or sample size [2504.15325].

Principled, probabilistic significance can be assigned by computing the likelihood (over all possible confusion matrices for given $n, m$) that a random labeler obtains a $\kappa$ at least as large as observed; this is termed the significance index $\varrho_{\kappa,n,m}(c)$. As samples grow, this approach converges to a distributional index $\rho_{\kappa,n}(c)$ based on the location of $\kappa$ in the probability simplex. This framework enables data-dependent, threshold-free interpretability for $\kappa$ [2504.15325].

Reporting best practice stipulates:

- Explicit statement of judgment scale and abstention handling
- Publication of full confusion matrices and marginal rates
- Reporting $\kappa$ alongside $AC_1$ or $G$ in dichotomous settings
- Cautious interpretation or replacement when prevalence is extreme or marginal bias is present
- Consideration of alternative or disattenuated indices when multi-rater or multi-class settings apply [2606.00093], [2601.10641]

## 6. Extensions and Empirical Frameworks

### Bayesian and Multilevel Extensions

For repeated measures, longitudinal, or multilevel data (e.g., multiple raters and time points, or hierarchical structures), generalized linear mixed models extend $\kappa$ estimation by incorporating subject, rater, and batch effects. Marginal or conditional $\kappa$ estimates then summarize agreement while propagating appropriate posterior uncertainty [2407.12700]. Bayesian models, e.g., BIN, BPN, and BFN frameworks, produce less biased $\kappa$ estimates with valid credible intervals, especially under small samples or deep nesting.

### Cross-Replication Reliability (xRR) Framework

Wong et al. propose benchmarking $\kappa$ through cross-replication reliability, introducing cross-kappa ($\kappa_x$) to quantify agreement across replications (e.g., different annotator pools or protocols) [2106.07393]. Given two annotation runs $X$ and $Y$ on the same items, $\kappa_x$ generalizes $\kappa$ to compare any two groups, with normalization correcting for low within-pool reliability:

$$
\kappa_x(X, Y) = 1 - \frac{d_o(X, Y)}{d_e(X, Y)}
$$

where $d_o$ and $d_e$ are observed and expected cross-replication disagreements.

Case studies demonstrate that $xRR$ reveals scenario-specific phenomena—such as population- or protocol-specific bias or reproducibility failures—unobservable with classical $\kappa$ alone.

## 7. Alternatives, Generalizations, and Practical Recommendations

Many limitations of $\kappa$ have motivated both conceptual and computational alternatives:

- Information Agreement (IA) and its zero-entry extension directly measure mutual information normalized by marginal entropy, avoiding arbitrary chance baselines and yielding $[0,1]$-scaled, always nonnegative agreement values [2008.11701].
- Gwet’s $AC_1$, Holley & Guilford’s $G$, Yule’s $Y$, and Bennett’s $S$ are empirically less biased and more robust across class-imbalance regimes [2203.09628], [2402.08069].
- Prevalence- and bias-adjusted $\kappa$ (PABAK) and generalized chance-corrected indices remedy specific defects, but do not address all issues.
- In simulated studies with correlated rater decision processes, $\kappa$ is consistently biased downward by about 0.08 units relative to probabilistic-certainty benchmarks; $AC_1$ and $S$ are closer to true corrected agreement [2402.08069].

Practical guidance includes:

- Avoiding $\kappa$ when outcome prevalence is extreme or marginal bias is strong.
- Supplementing $\kappa$ with prevalence and bias indices; when necessary, apply alternative indices and contextualize any negative or near-zero $\kappa$ against the true lower bound determined by marginals.
- Using $xRR$ or Bayesian models for benchmarking and quantifying reliability in annotation-intensive, crowdsourced, or multilevel designs.

## Summary Table: Cohen’s Kappa—Core Quantities

| Quantity           | Symbol        | Formula / Description                                          |
|--------------------|--------------|---------------------------------------------------------------|
| Observed agreement | $P_o$        | $\frac{1}{n}\sum_{i} n_{ii}$                                  |
| Marginal rates     | $p_i, q_i$   | Row/column sums per category ($p_i$, $q_i$)                   |
| Chance agreement   | $P_e$        | $\sum_{i=1}^K p_i q_i$                                        |
| Kappa              | $\kappa$     | $\frac{P_o - P_e}{1 - P_e}$                                   |
| Maximum kappa      | $\kappa_{\max}$ | $\frac{A_{\max} - P_e}{1 - P_e}$                            |
| Minimum kappa      | $\kappa_{\min}$ | $\frac{A_{\min} - P_e}{1 - P_e}$                            |
| Info distance bound| $1-2^{-R(P,Q)}$ | Resistor average distance between class-conditional densities |

Cohen’s $\kappa$ remains a central measure of chance-corrected agreement, yet its interpretation demands careful attention to prevalence, marginal distributions, and the explicit modeling of chance. Alternatives and recent generalizations offer more principled or informative reliability metrics in many empirical settings [2008.11701], [2106.07393], [2203.09628], [2403.01571], [2404.14149], [2504.15325], [2601.10641], [2606.00093], [2402.08069], [2407.12700].

Source: https://www.emergentmind.com/topics/cohen-s-kappa