Overall Agreement for Multiple Raters with Replicated Measurements

Abstract: Multiple raters are often needed to be used interchangeably in practice for measurement or evaluation. Assessing agreement among these multiple raters via agreement indices are necessary before their participation. While the intuitively appealing agreement indices such as coverage probability and total deviation index, and relative area under coverage probability curve, have been extended for assessing overall agreement among multiple raters, these extensions have limitations. The existing overall agreement indices either require normality and homogeneity assumptions or did not preserve the intuitive interpretation of the indices originally defined for two raters. In this paper, we propose a new set of overall agreement indices based on maximum pairwise differences among all raters. The proposed new overall coverage probability, overall total deviation index and relative area under overall coverage probability curve retain the original intuitive interpretation from the pairwise version. Without making any distributional assumption, we also propose a new unified nonparametric estimation and inference approach for the overall indices based on generalized estimating equations that can accommodate replications made by the same rater. Under mild assumptions, the proposed variance estimator is shown to achieve efficiency bound under independent working correlation matrix. Simulation studies under different scenarios are conducted to assess the performance of the proposed estimation and inference approach with and without replications. We illustrate the methodology by using a blood pressure data with three raters who made three replications on each subjects.