Identifying bias mechanisms in Paper Copilot’s ARR self-reported data

Determine whether the upward bias observed in ARR self-reported review score samples collected via Paper Copilot is attributable to survivorship, complainer, or borderliner effects by incorporating decision outcomes (accept/reject) into the dataset and analyzing score distributions conditional on decision status.

Background

Beyond Zhihu and Reddit, the authors examine self-reported ARR scores collected by the Paper Copilot website. They find the same upward bias in these samples when compared to population distributions.

However, Paper Copilot’s dataset lacks decision outcomes, preventing the authors from decomposing the observed bias into the previously identified mechanisms (survivors, complainers, borderliners). Adding decision data would enable a mechanism-level analysis analogous to their main paper.