Fleiss’ Kappa (κ): A Reliability Metric

• Fleiss’ kappa is a chance-corrected coefficient that measures inter-rater agreement when multiple raters classify subjects into nominal categories.
• It computes agreement by comparing the mean observed agreement with the expected chance agreement derived from marginal probabilities.
• Widely applied in systematic reviews, medical diagnostics, and educational assessments, it supports extensions for multi-label and hierarchical analyses.

Fleiss’ kappa ($\kappa$) is a chance-corrected coefficient quantifying inter-rater agreement for categorical items when each of $n$ raters classifies each of $N$ subjects independently into one of $k$ nominal categories. Unlike pairwise measures such as Cohen’s $\kappa$, Fleiss’ $\kappa$ generalizes agreement assessment to cases with more than two raters. Extensions further allow the analysis of scenarios in which raters can select multiple, possibly hierarchical, categories per subject. Its mathematical rigor and extensibility make $\kappa$ a standard reliability metric in systematic reviews, medical diagnostics, educational testing, and other multi-rater domains (Arenas, 2018, Moons et al., 2023).

1. Mathematical Foundations and Formal Definition

Let $N$ denote the number of subjects, $n$ the number of raters, and $k$ the number of categories. Denote by $n_{ij}$ the number of raters who assigned subject $i$ to category $j$, for $1 \leq i \leq N,\ 1 \leq j \leq k$. Fleiss’ $\kappa$ is given by:

$\kappa = \frac{\bar{P} - \bar{P}_e}{1 - \bar{P}_e}$

where:

• $\bar{P}$: mean observed agreement across subjects,
• $\bar{P}_e$: mean agreement expected by chance.

Observed agreement $P_i$ for subject $i$:

$$P_i = \frac{1}{n(n-1)} \sum_{j=1}^k n_{ij} (n_{ij} - 1)$$

Mean observed agreement:

$\bar{P} = \frac{1}{N} \sum_{i=1}^N P_i$

Marginal probability of category $j$:

$p_j = \frac{1}{Nn} \sum_{i=1}^N n_{ij}$

Chance agreement:

$\bar{P}_e = \sum_{j=1}^k p_j^2$

Inserting these into the $\kappa$ formula yields the classical Fleiss’ $\kappa$ (Arenas, 2018, Moons et al., 2023).

2. Extensions: Generalised and Hierarchical Fleiss’ κ

Classical Fleiss’ $\kappa$ requires each rater to assign each subject to exactly one category. In research contexts where raters may select multiple (non-exclusive) categories per subject, or where categories have a hierarchy or weights, a generalized $\kappa$ is used (Moons et al., 2023). Key definitions:

• $x_{ijc} \in \{0,1\}$: indicator that rater $j$ assigns subject $i$ to category $c$.
• $x_{ic}$: total “yes” votes for subject $i$ and category $c$.
• $w_c > 0$: context-dependent weights per category (e.g., score magnitude).
• $s_{ic}$: number of raters eligible to select category $c$ on subject $i$ (for hierarchical designs).

Agreement per category per subject:

$$P_{ic} = \frac{(x_{ic})(x_{ic} - 1) + (s_{ic} - x_{ic})(s_{ic} - x_{ic} - 1)}{s_{ic}(s_{ic} - 1)}$$

Aggregated observed agreement for category $c$:

$Po_c = \frac{\sum_{i=1}^N P_{ic}}{\sum_{i=1}^N 1}$

Expected agreement per category:

$$Pe_c = 2\left(\frac{\sum_i x_{ic}}{\sum_i s_{ic}}\right)^2 - 2\left(\frac{\sum_i x_{ic}}{\sum_i s_{ic}}\right) + 1$$

Generalized $\kappa$ for the hierarchical, weighted case:

$$\kappa = \frac{\sum_{c=1}^k w_c \, \phi_c \, (Po_c - Pe_c)}{\sum_{c=1}^k w_c \, \phi_c \, (1 - Pe_c)}$$

with $\phi_c = \frac{\sum_i s_{ic}}{NI}$, a scaling factor ensuring that main categories dominate deep subcategories when eligible rater counts differ.

When categories are exclusive and all raters assess all subjects, the generalization reduces to the classical Fleiss’ $\kappa$ (Moons et al., 2023).

3. Computational Procedure and Software Implementations

The canonical computational pipeline is implemented in the Inter-Rater Python package (Arenas, 2018), which accepts:

• An $N \times n$ data matrix (subjects $\times$ raters) of labels,
• Explicit category definitions,
• Handles missing or out-of-list ratings as abstentions.

Main computational steps:

1. Build the $N \times k$ count matrix $\{n_{ij}\}$.
2. Compute per-subject $P_i$, aggregate to $\bar{P}$, and compute $p_j$, $\bar{P}_e$, yielding $\kappa$.
3. Estimate $\kappa$’s confidence interval using the variance approximation from Fleiss et al. (1979).
4. For rater-specific diagnostics: permute all rater pairs, compute pairwise Cohen’s $\kappa$s, and average to obtain per-rater reliability scores (“permuted-κ”).

Efficiency is $O(N n^2)$, enabling practical analyses for hundreds or thousands of subjects and tens of raters.

4. Interpretation and Benchmarks

Interpretation typically follows the scale of Landis and Koch (1977):

$\kappa$ range	Interpretation
$<$0.00	Poor agreement
0.00–0.20	Slight agreement
0.21–0.40	Fair agreement
0.41–0.60	Moderate agreement
0.61–0.80	Substantial agreement
0.81–1.00	Almost perfect

These labels are descriptive conventions and may be context-dependent. For instance, a numerical example with $N=3$, $n=4$, $k=3$ resulting in $\kappa \approx 0.23$ would be interpreted as “fair” agreement (Arenas, 2018).

5. Practical Applications and Example Scenarios

Fleiss’ $\kappa$ and its generalizations are applied in domains requiring robust quantification of multi-annotator reliability:

• Systematic reviews involving multiple reviewers classifying abstracts,
• Medical diagnostics (e.g., inter-pathologist/radiologist reliability),
• Educational assessment with multiple scorers,
• Psychiatric diagnosis data where raters select multiple disorders per case,
• Item-level grading with possible partial credit and hierarchical feedback structures.

Explicit worked examples include checkbox-based mathematics grading with weights and hierarchy, and DSM-III psychiatric diagnoses with variable rater counts per case (Moons et al., 2023). The general approach handles missing data by varying the denominator per subject or category and supports rater-specific reliability diagnostics (Arenas, 2018, Moons et al., 2023).

6. Algorithmic Innovations and Limitations

Algorithmic enhancements present in current software include:

• Handling of incomplete designs (missing data, abstentions, or per-category eligibility),
• Retention of per-rater information via exhaustive pairwise permutations,
• Publication-ready visualizations (group agreement, per-pair, and per-user),
• Accommodation of hierarchical categories and context-driven weighting.

A commonly identified limitation is that the classical form of Fleiss’ $\kappa$ assumes exclusivity of categories and requires complete ratings. Generalizations address this by enabling (a) multi-label assignment, (b) weighted and hierarchical categories, and (c) variable rater eligibility per subject-category pair (Moons et al., 2023).

• "Inter-Rater: Software for analysis of inter-rater reliability by permutating pairs of multiple users" (Arenas, 2018)
• "Measuring agreement among several raters classifying subjects into one-or-more (hierarchical) nominal categories. A generalisation of Fleiss' kappa" (Moons et al., 2023)