AGInTRater: Ordinal Agreement Index

• AGInTRater is an ordinal statistical framework that measures interrater absolute agreement by normalizing Leti’s dispersion index.
• It offers unbiased estimation and explicit variance formulas to ensure robust and interpretable agreement measures even under restricted score variability.
• Empirical validation via simulation and real-world cases shows AGInTRater provides reliable confidence intervals and improved alignment with qualitative consensus over traditional metrics.

AGInTRater is an ordinal index and statistical framework for measuring interrater absolute agreement, specifically designed to address limitations of traditional agreement metrics—such as Cohen’s Kappa and the intraclass correlation coefficient (ICC)—when applied to ordinal data. Rooted in Giuseppe Leti’s cumulative dispersion index, AGInTRater provides a normalized, unbiased, and inferentially rigorous measure of rater consensus that is robust to variance restriction and applicable across diverse rating scenarios.

1. Theoretical Foundations and Motivation

AGInTRater generalizes agreement assessment by capitalizing on Leti’s ordinal dispersion index, defined for a $K$-category ordinal scale as

$D = \sum_{k=1}^{K-1} 2F_k(1 - F_k)$

where $F_k$ is the empirical cumulative proportion of ratings at or below category $k$. This index possesses desirable properties: $D = 0$ if all ratings are identical (no dispersion), and $D$ maximizes when ratings split between the two extreme categories. To facilitate interpretation and generalizability, the index is normalized,

$d = D / D_{\max}$

where $D_{\max}$ is the theoretical maximum for the observed $N$ (with $D_{\max} = (K-1)/2$ for even $N$). This yields $d \in [0, 1]$, where lower values indicate higher agreement.

AGInTRater was formulated to resolve critical weaknesses in conventional metrics:

• Null distribution assumptions: Standard metrics often presume a particular distribution under the null hypothesis (usually uniform), which is not justified in many applied problems.
• Restricted variance distortions: When ratings collapse onto a single or few categories (“restriction of variance” problem), Kappa-like or ICC-based indices approach zero, misrepresenting strong consensus.

2. Unbiased Estimation and Statistical Properties

Let $n_R$ be the number of raters and $n_T$ the number of targets (e.g., rated objects or instances). For target $i$, with ratings $X_{ij} \in \{1, \ldots, K\}$, the empirical CDF is

$$F_i(k) = \frac{1}{n_R} \sum_{j=1}^{n_R} \mathbf{1}\left(X_{ij} \le k\right)$$

and target-level dispersion

$D_i = \sum_{k=1}^{K-1} 2F_i(k)(1 - F_i(k))$

Alternatively,

$$D_i = \frac{1}{n_R(n_R - 1)} \sum_{j \ne j'} |X_{ij} - X_{ij'}|$$

which expresses $D_i$ as the mean of absolute pairwise distances among raters.

Under the assumption of independent and identically distributed ratings across both raters and targets, $D_i$ is an unbiased estimator of $D$, up to a factor $(n_R-1)/n_R$: $\mathbb{E}[D_i] = \frac{n_R - 1}{n_R} D$ Pooling $D_i$ across targets and normalizing yields the sample-level $d$; bias correction is achieved with

$d^* = \frac{n_R}{n_R - 1} d$

The sampling variance of $D_i$ (and thus $d^*$), is given explicitly in terms of combinatoric sums over indicator differences and is asymptotically

$$\operatorname{Var}(D_i) \sim \frac{4(J - D^2)}{n_R}$$

with $J$ an elementary symmetric sum over CDF indicators. This supports Central Limit Theorem–based inferential procedures for large $n_R$ and $n_T$.

3. Inference: Confidence Intervals

Two principal methods for constructing confidence intervals (CIs) for $d^*$ are supported:

• Asymptotic Normal Approximation: Using the estimated variance $V_{d^*}$, confidence bounds are

$$L = d^* - z_{\alpha/2} \sqrt{V_{d^*}}, \quad U = d^* + z_{\alpha/2} \sqrt{V_{d^*}}$$

Simulation studies confirm this method exhibits slightly conservative coverage (about $99.4\%$ for true $95\%$ intervals) and parsimonious interval length.

• Bootstrap-based CIs: Options include the percentile, bootstrap-$t$, and pivotal methods. The percentile bootstrap—especially under a parametric model matched to the data—delivers CIs close to nominal coverage, while the other bootstrap methods are sometimes sensitive to skewness or tail error inflation.

4. Empirical Validation and Use Cases

AGInTRater has been benchmarked using both simulated and real datasets:

• Simulation: For $n_T = 150$ targets and $n_R = 28$ raters (and subsampling to $n_T = 50$, $n_R = 7$), with a “true” $d$ value of $0.61$ under a 5-point ordinal model, normal-approximation CIs consistently covered the nominal level with short average intervals. Pseudo-nonparametric bootstrap, which respects the two-stage sampling design, mitigated resampling bias under restricted ratings.
• Real-world application: In a paper of language proficiency scored on a 6-point Likert scale (7 raters, focus on the “comprehensibility” subscore), classical ICC metrics (e.g., ICC(A,1)) yielded near-zero agreement ($0.14$), an artifact of range restriction. AGInTRater produced $d = 0.17$ and $d^* = 0.19$—values that more faithfully reflected the high interrater consensus visible in the clustered ratings.

5. Mathematical Formulation

Central formulas include:

• Leti’s Dispersion Index: $D = \sum_{k=1}^{K-1} 2F_k(1 - F_k)$
• Normalized Agreement: $d = D / D_{\max}$
• Pairwise Absolute Difference: $$D_i = \frac{1}{n_R(n_R - 1)} \sum_{j \ne j'} |X_{ij} - X_{ij'}|$$
• Unbiased Sample Estimate: $d^* = \frac{n_R}{n_R - 1} d$
• Variance for $d^*$:

$$V_{d^*} = \frac{\left(\frac{n_R - 1}{n_R^2}\right)[4Q + 4(n_R-2)J - 2(2n_R-3)D^2]/D_{\max}^2}{n_T}$$

where $J$ and $Q$ are combinatoric sums over category counts as described in the source.

These formulations ensure both efficient computation and theoretical soundness for inferential purposes.

6. Comparison to Established Measures

Conventional indices such as Cohen’s Kappa, ICCs, or rwG suffer from two technical issues in ordinal settings:

• Data-type misalignment: Many treat ordinal responses as interval-numeric or require arbitrary scaling.
• Variance restriction: When between-target variance is artificially suppressed (e.g., ratings cluster at the top), these indices approach zero, under-reporting actual agreement.

AGInTRater (the normalized Leti index) directly models the ordinal structure, does not depend on null distributional assumptions, and is immune to variance restriction at the between-target level due to its within-target, dispersion-focused construction. In side-by-side comparisons, AGInTRater yields consensus assessments that align more closely with qualitative researcher/clinician intuition when restricted variability or ordinal rating schemes are present.

7. Practical Implementation and Scope

AGInTRater computation requires only the ratings matrix, category labels, and rater/target identifiers. The estimator and normalization steps are trivial to implement; variance and CI calculation involve empirical moments and are manageable for typical dataset sizes. The method is broadly suited to rating applications in language assessment, psychology, medicine, and any field utilizing ordinal scales for subjective scoring. AGInTRater is particularly recommended where traditional agreement indices become unstable or deflate in the presence of restricted score ranges.

In sum, AGInTRater represents an ordinal-specific, theoretically justified, and practically robust measure for interrater absolute agreement, advancing statistical methodology for fields reliant on subjective, categorical rating architectures (Bove et al., 2019).