Does selective error detection extend to favorable results?

Establish whether researchers’ probability of detecting coding errors is systematically lower when the errors produce favorable outcomes—defined as results more likely to be published or that support researchers’ hypotheses—thereby determining if selective error detection depends on result favorability in addition to whether results are expected or unexpected.

Background

The paper reports experimental evidence from a randomized coding task embedded in the World Bank’s recruitment process showing that individuals are significantly more likely to detect coding errors when those errors lead to unexpected results. The manipulated error involved failing to treat the value 99 as a missing outcome in regression analyses, and the randomized design ensured that the same error could yield either expected or unexpected findings.

In the Discussion, the authors propose that this mechanism might also apply to favorable results—i.e., outcomes perceived as publishable or supportive of hypotheses—suggesting a potential pathway for systematic bias if researchers are less inclined to scrutinize coding when it produces desirable findings. Clarifying whether result favorability influences error detection would have implications for research practices, particularly for placebo tests and broader concerns about replication, transparency, and publication bias.