Min-Mid-Max Scaling, Limits of Agreement, and Agreement Score (2006.12904v5)

Abstract: In this paper, I solve a 60-year old question posed by Cohen's seminal paper (1960) and offer an agreement measure centered around the chance-expected agreement while isolating marginally forced agreement and disagreement. To achieve this, I formulate the minimum feasible agreement given row and column marginals by devising a new algorithm that minimizes the sum of diagonals in contingency tables. Based on this result, I also formulate the lower limit of the most common agreement measure-Cohen's kappa. Finally, I study the lower limit of maximum feasible agreement and devise two statistics of distribution similarity for agreement analysis.