Peer-review in a world with rational scientists: Toward selection of the average (1008.4324v1)

Abstract: One of the virtues of peer review is that it provides a self-regulating selection mechanism for scientific work, papers and projects. Peer review as a selection mechanism is hard to evaluate in terms of its efficiency. Serious efforts to understand its strengths and weaknesses have not yet lead to clear answers. In theory peer review works if the involved parties (editors and referees) conform to a set of requirements, such as love for high quality science, objectiveness, and absence of biases, nepotism, friend and clique networks, selfishness, etc. If these requirements are violated, what is the effect on the selection of high quality work? We study this question with a simple agent based model. In particular we are interested in the effects of rational referees, who might not have any incentive to see high quality work other than their own published or promoted. We find that a small fraction of incorrect (selfish or rational) referees can drastically reduce the quality of the published (accepted) scientific standard. We quantify the fraction for which peer review will no longer select better than pure chance. Decline of quality of accepted scientific work is shown as a function of the fraction of rational and unqualified referees. We show how a simple quality-increasing policy of e.g. a journal can lead to a loss in overall scientific quality, and how mutual support-networks of authors and referees deteriorate the system.

An Analysis of Peer Review's Vulnerability within Scientific Publishing

This paper by Stefan Thurner and Rudolf Hanel presents an agent-based model to evaluate the peer review system's efficiency in scientific publishing, particularly focusing on deviations from optimal reviewer behavior. By examining a community of scientists who act as referees, the paper investigates the impact of 'rational' referees—individuals who may prefer not to approve work superior to their own—on the quality of published research. It also considers how networks of favoritism among reviewers can further undermine the system.

The model assumes several archetypes of referees: correct (those who objectively assess papers), random (incapable of discerning quality), and rational (who act out of self-interest by rejecting superior submissions). It explores how these types influence the overall quality of published scientific work. The authors establish that even a small number of rational reviewers significantly reduces the quality of selected papers. For instance, introducing 10% rational referees degrades accepted paper quality by approximately one standard deviation from the mean quality of submitted papers.

The simulations indicate a stark decline in system performance as the fraction of rational referees increases, with performance approaching pure chance when rational referees constitute over 70% of the reviewing body. This decline is exacerbated in the presence of random referees, further eroding the efficacy of quality selection.

Moreover, the influence of nepotism is significant. If 10% of the community functions within a mutual acceptance network, the model predicts that the quality disparity between network-influenced and non-network papers is considerable, with overall system efficiency noticeably impaired. These friendship networks effectively bypass quality checks, reducing the average quality of published work.

A striking, counterintuitive finding is that raising quality standards using quantiles based on recent publications, represented by an increase in the parameter $\alpha$, paradoxically leads to a decrease in publication quality. This occurs because rational referees' influence grows relative to that of correct referees, as the latter become more selective, further biasing the outcomes.

The implications of this paper highlight the inherent volatility of the peer review system in the presence of self-serving dynamics among reviewers. Practically, the results suggest a vulnerability in relying on traditional peer review unless the prevalence of rational and random referees is mitigated. Theoretically, the model questions the foundational robustness of a widely-accepted system under less-than-ideal conditions.

Future developments in AI, while not directly addressed in the paper, provide potential avenues for augmenting the peer review process by introducing automated checks that might counterbalance human biases. Machine learning algorithms could identify biased refereeing patterns or assist in preliminary quality assessments, thus preserving peer review's relevance while addressing some intrinsic inefficiencies exposed by this model. However, further research must assess the practicability and reliability of such technological interventions to supplement or even substitute aspects of the traditional peer review system.