Removing Bias and Incentivizing Precision in Peer-grading (1807.11657v7)

Abstract: We study peer-grading with competitive graders who enjoy a higher utility when their peers get lower scores. We propose a new mechanism, PEQA, that incentivizes such graders through a score-assignment rule which aggregates the final score from multiple peer-evaluations, and a grading performance score that rewards performance in the peer-grading exercise. PEQA makes grader-bias irrelevant. Additionally, under PEQA, a peer-grader's utility increases monotonically with the reliability of her grading, irrespective of her competitiveness and how her co-graders act. In a reasonably general class of score assignment rules, PEQA uniquely satisfies this utility- reliability monotonicity. When grading is costly and costs are private information, a modified version of PEQA implements the socially optimal effort choices in an equilibrium of the peer-evaluation game. Data from our classroom experiments confirm our theoretical assumptions and show that PEQA outperforms the popular median mechanism.