---
title: Quadratic Weighted Kappa Overview
url: https://www.emergentmind.com/topics/quadratic-weighted-kappa-qwk
type: topic
---

# Quadratic Weighted Kappa Overview

Quadratic Weighted Kappa (QWK) is a metric for quantifying agreement between two ratings over a set of categorical, ordinal labels. It is widely utilized in human–machine evaluation pipelines where the target variable is an integer-valued grade or assessment, such as automated essay scoring, image annotation, or subjective ratings in natural language understanding. QWK extends Cohen’s kappa by incorporating a quadratic penalty for larger disagreements between raters, thus emphasizing the ordinal structure of the labels and providing interpretable, scale-sensitive agreement scores.

## 1. Mathematical Definition and Computation

Given two raters (or a rater and an automated system) assigning labels in $K$ ordered categories on $N$ samples, QWK is constructed as follows:

- **Observed Matrix ($O$):** $O_{ij}$ is the count of items with true label $i$ and predicted label $j$.
- **Marginals:** $R_i = \sum_j O_{ij}$ (row sum for true label $i$), $C_j = \sum_i O_{ij}$ (column sum for predicted label $j$).
- **Expected Matrix ($E$):** $E_{ij} = \frac{R_i C_j}{N}$, representing the count expected under statistical independence.
- **Quadratic Weights ($W$):** $w_{ij} = \frac{(i-j)^2}{(K-1)^2}$, imposing stronger penalties for greater rating disagreements.

QWK is then defined as:
$$
\kappa_{QW} = 1 - \frac{\sum_{i,j} w_{ij} O_{ij}}{\sum_{i,j} w_{ij} E_{ij}}
$$
This formulation represents one minus the ratio of observed weighted disagreement to the expected weighted disagreement under label independence. When agreement is perfect, the numerator is zero and $\kappa_{QW}=1$. When observed agreement is no better than chance, $\kappa_{QW}=0$ [2510.27131, 2110.12765, 2604.19131].

## 2. Interpretation and Properties

QWK is tailored to ordinal data and explicitly penalizes disagreements according to their magnitude: an error of $|i-j|=1$ is penalized much less than $|i-j|=K-1$. Its range is $[-\infty,1]$ in general, but is bounded in $[0,1]$ for labels restricted to a finite range, as in grading tasks [2111.08906].

A QWK near 1 indicates almost perfect agreement adjusted for chance, while a QWK near 0 implies agreement no better than chance. Negative values arise if observed disagreement exceeds that expected by chance—indicating systematic disagreement.

QWK is sensitive to marginal distributions. If rater marginals differ significantly, the expected matrix $E$ shifts accordingly, modulating the baseline for chance.

## 3. Step-by-Step Example

Suppose $K=3$ (labels 0,1,2), $N=100$, and the observed matrix is:
$$
O = \begin{pmatrix}
30 & 5 & 0 \\
4 & 25 & 6 \\
0 & 7 & 23
\end{pmatrix}
$$
- Compute marginals: $R=(35,35,30)$, $C=(34,37,29)$.
- Weights: $w_{ij}=0$ if $i=j$, $1/4$ if $|i-j|=1$, $1$ if $|i-j|=2$ (since $(K-1)^2=4$).
- Compute weighted observed sum: $\frac{1}{4}(5+4+6+7)=5.5$.
- Compute weighted expected sum as $\sum_{i,j} w_{ij} E_{ij}=11.525$.
- Then QWK: $1 - 5.5/11.525 \approx 0.523$ [2604.19131].

## 4. Practical Use in Human–Machine Evaluation

QWK is the principal metric in automated essay scoring (AES) and similar tasks, being the official objective for high-profile challenges such as those on Kaggle [2510.27131]. In AES, for each essay, a model predicts an integer grade; massive state-of-the-art models (e.g., BERT, RoBERTa, DeBERTa) are compared using QWK against one or more human ratings. In rationalized AES evaluation, QWK is computed for ensemble models or models leveraging LLM-generated rationales, both in standalone and stacking regimes. Reported QWKs for top models on Prompt 6 of ASAP data range from approximately 0.80 to 0.87, with ensemble strategies incrementally improving agreement [2510.27131].

In psychometrics, QWK is also employed to validate label proxies. For example, laughter-normalized “humour quotient” scores were validated against three human annotators, with mean QWK of 0.6, indicating that the automatic signal is as consistent as an additional human rater [2110.12765].

## 5. QWK in Active and Human-in-the-Loop Evaluation

Human-labeling cost motivates hybrid scoring pipelines that intelligently allocate human rater effort. Sampling approaches prioritize items whose manual correction will most increase QWK. 

- **Reward Sampling:** For each machine-predicted label $m$, its empirical agreement distribution with human labels $M[m][c]$ is used to compute the expected reward $E_R(d)$ of relabeling a record $d$. Records are sampled in proportion to $E_R(d)$, resulting in substantial QWK gains even with modest human annotation budgets (e.g., improvements of 25.6% with 30% re-graded samples) [2111.08906].
- **Uncertainty/Importance Sampling:** Selection based on cross-entropy or uncertainty in prediction.
- **Estimation With Guarantees:** For unbiased QWK estimation, importance sampling at the test-taker level, leveraging per-taker uncertainty weights $\zeta(t)$, yields empirical confidence intervals. With $N_{est}=200$, a 95% CI width of $\lesssim \pm 0.02$ is achievable.

These methods have been empirically validated across BERT and LSTM model baselines, consistently improving QWK under realistic annotation constraints [2111.08906].

## 6. Statistical Estimation, Significance, and Ceiling Effects

A critical methodological question concerns the achievable QWK ceiling in the presence of noisy human labels. Classical Test Theory (CTT) provides a principled approach:

- **Theoretical Ceiling ($\hat \kappa_{\max}$):** This is the QWK that an ideal model (predicting latent true scores) can reach against noisy observed labels:
  $$
  \hat\kappa_{\max} = \sqrt{\hat\rho_Y}
  $$
  where $\rho_Y$ is the reliability of the mean score of all raters.

- **Human-like Ceiling ($\hat\kappa_{\mathrm{HL}}$):** This is the QWK attainable by a model whose error variance matches that of a single human:
  $$
  \hat\kappa_{\mathrm{HL}} = \sqrt{\hat\rho_1\,\hat\rho_Y}
  $$
  where $\rho_1$ is the reliability of a single rater.

- **Computation:** From two-rater data, compute mean squares between and within items, estimate $\hat\rho_1$ and $\hat\rho_Y$, then apply the formulas above [2604.19131].

- **Interpretation:** Human–human QWK ($\kappa_{H}$) is strictly less than both ceiling values. Using empirical $\kappa_{H}$ as the “best possible” can underestimate what a model could achieve if it matched the reliability of human raters in the data.

## 7. Application-Specific Findings and Limitations

- In AES, QWK is robust to class imbalance, and model improvements are reliably reflected in QWK increases, as confirmed by ensemble and stacking experiments [2510.27131].
- In the context of multimodal evaluation, such as computational humor, QWK provides a nuanced measure of model–rater agreement, reflecting not just binary hits but proximity in ordinal space [2110.12765].
- QWK does not in itself quantify statistical significance in differences; bootstrapping or related resampling schemes are necessary for confidence intervals and hypothesis testing [2510.27131].
- Optimal QWK is constrained by rater reliability; as a result, further modeling gains may be unattainable unless label noise is reduced [2604.19131].

A plausible implication is that as models approach the QWK ceiling imposed by human reliability, further improvements should focus on label quality and procedural re-annotation rather than solely on improving model architecture. The explicit ceiling calculations provide actionable targets for both model selection and dataset curation in ordinal labeling tasks.

Source: https://www.emergentmind.com/topics/quadratic-weighted-kappa-qwk