Liberal-Conservative Hierarchies of Intercoder Reliability Estimators (2410.05291v2)

Abstract: While numerous indices of inter-coder reliability exist, Krippendorff's {\alpha} and Cohen's {kappa} have long dominated in communication studies and other fields, respectively. The near consensus, however, may be near the end. Recent theoretical and mathematical analyses reveal that these indices assume intentional and maximal random coding, leading to paradoxes and inaccuracies. A controlled experiment with one-way golden standard and Monte Carlo simulations supports these findings, showing that {kappa} and {\alpha} are poor predictors and approximators of true intercoder reliability. As consensus on a perfect index remains elusive, more authors recommend selecting the best available index for specific situations (BAFS). To make informed choices, researchers, reviewers, and educators need to understand the liberal-conservative hierarchy of indices, i.e., which indices produce higher or lower scores. This study extends previous efforts by expanding the math-based hierarchies to include 23 indices and constructing six additional hierarchies using Monte Carlo simulations. These simulations account for factors like the number of categories and distribution skew. The resulting eight hierarchies display a consistent pattern and reveal a previously undetected paradox in the Ir index.