The Natural Selection of Bad Science (1605.09511v1)

Abstract: Poor research design and data analysis encourage false-positive findings. Such poor methods persist despite perennial calls for improvement, suggesting that they result from something more than just misunderstanding. The persistence of poor methods results partly from incentives that favor them, leading to the natural selection of bad science. This dynamic requires no conscious strategizing---no deliberate cheating nor loafing---by scientists, only that publication is a principle factor for career advancement. Some normative methods of analysis have almost certainly been selected to further publication instead of discovery. In order to improve the culture of science, a shift must be made away from correcting misunderstandings and towards rewarding understanding. We support this argument with empirical evidence and computational modeling. We first present a 60-year meta-analysis of statistical power in the behavioral sciences and show that power has not improved despite repeated demonstrations of the necessity of increasing power. To demonstrate the logical consequences of structural incentives, we then present a dynamic model of scientific communities in which competing laboratories investigate novel or previously published hypotheses using culturally transmitted research methods. As in the real world, successful labs produce more "progeny", such that their methods are more often copied and their students are more likely to start labs of their own. Selection for high output leads to poorer methods and increasingly high false discovery rates. We additionally show that replication slows but does not stop the process of methodological deterioration. Improving the quality of research requires change at the institutional level.

• The paper’s main finding is that academic incentives drive researchers to adopt poor methodologies, leading to consistently low statistical power.
• It employs a comprehensive meta-analysis and computational modeling to demonstrate how publication pressures propagate flawed research practices.
• The study underscores the need for systemic institutional reforms to realign career incentives with rigorous, replicable scientific research.

The Natural Selection of Bad Science: An Overview

The paper "The Natural Selection of Bad Science" by Smaldino and McElreath presents a critical analysis of the systemic flaws in scientific research practices, arguing that institutional incentives favor poor research methods, leading to persistent issues such as low statistical power and high false positivity rates. The authors propose that the current academic landscape encourages a form of "natural selection" that propagates these inadequate methods, further complicating efforts to improve scientific validity and replicability.

The central thesis is that scientific practices are not solely shaped by methodological integrity but are significantly influenced by career incentives and publication pressures. The authors support this claim with empirical evidence, particularly focusing on a 60-year meta-analysis of statistical power in the behavioral sciences. They demonstrate that despite longstanding calls for increased statistical power, there has been little to no observable improvement.

Empirical and Theoretical Support

1. Empirical Analysis:
   • The authors present a comprehensive meta-analysis indicating that statistical power within behavioral sciences has remained virtually unchanged for decades. This stagnation persists despite numerous scholarly articles emphasizing the importance of improved research design and rigorous data analysis.
   • The paper reveals that although the academic output in terms of publication quantity has increased, the quality and reliability of these publications remain questionable due to methodological shortcomings inherent in the system.
2. Computational Modeling:
   • The research incorporates a dynamic model simulating scientific communities where laboratories compete for publishing novel findings. This model illustrates how cultural transmission of methods skews towards those facilitating higher publication output.
   • The simulations predict an increasing prevalence of low-effort, low-quality research methodologies, assuming logical consistency with evolutionary processes observed in natural selection.

Implications and Future Directions

This paper highlights the inherent conflict between individual researchers' career incentives and the collective scientific goal of reliable knowledge discovery. The authors suggest that addressing these issues requires institutional rather than individual reforms. They argue that merely correcting methodological misunderstandings is insufficient; instead, systemic incentives that prioritize understanding and replication over mere publication must be instituted to improve scientific integrity.

The implications of this research are profound, drawing attention to the need for academic institutions, funding bodies, and journals to reevaluate how they assess research quality and researcher productivity. Amplifying replicability and methodological rigor is crucial, though the authors caution against oversimplifying solutions such as punitive measures for non-replicable results, as they may not address the root causes of research deficiencies.

Concluding Thoughts

The paper makes a substantial contribution to metascience by elucidating the evolutionary dynamics within scientific practices, driven by institutional incentives. While the authors refrain from using hyperbolic language, their findings underscore the need for systemic overhaul to foster an environment that truly values scientific integrity and progress. This work serves as a calling for practitioners, policymakers, and institutions to collaboratively engineer a scientific culture where quality, rather than quantity, becomes the haLLMark of success.