Deep Impact: Unintended consequences of journal rank (1301.3748v3)

Abstract: Most researchers acknowledge an intrinsic hierarchy in the scholarly journals ('journal rank') that they submit their work to, and adjust not only their submission but also their reading strategies accordingly. On the other hand, much has been written about the negative effects of institutionalizing journal rank as an impact measure. So far, contributions to the debate concerning the limitations of journal rank as a scientific impact assessment tool have either lacked data, or relied on only a few studies. In this review, we present the most recent and pertinent data on the consequences of our current scholarly communication system with respect to various measures of scientific quality (such as utility/citations, methodological soundness, expert ratings or retractions). These data corroborate previous hypotheses: using journal rank as an assessment tool is bad scientific practice. Moreover, the data lead us to argue that any journal rank (not only the currently-favored Impact Factor) would have this negative impact. Therefore, we suggest that abandoning journals altogether, in favor of a library-based scholarly communication system, will ultimately be necessary. This new system will use modern information technology to vastly improve the filter, sort and discovery functions of the current journal system.

• The paper critically examines how relying on journal rank metrics, like the Impact Factor, negatively impacts scientific reliability and integrity.
• Evidence presented includes rising misconduct retractions in high-ranking journals, publication bias leading to the 'decline effect,' and a lack of correlation between high rank and methodological quality.
• The authors suggest that current journal rank reliance poses systemic risks and propose transitioning to alternative, article-focused evaluation systems to enhance scientific communication integrity.

An Analysis of the Unintended Consequences of Journal Rank

This paper, authored by Björn Brembs, Katherine Button, and Marcus Munafò, presents a critical examination of the impact of journal rank within the context of scholarly communication. The authors address the continued reliance upon journal rank, often symbolized by metrics like the Impact Factor (IF), as a key determinant for assessing scientific impact and quality. This review aggregates current data and studies to argue that the pervasive use of journal rank as an evaluation tool may have numerous negative consequences for scientific reliability and integrity.

Key Findings and Evidence

The authors support several significant claims with a robust body of empirical evidence:

1. Retraction Rates: The paper notes a rising trend in retraction rates from scientific journals starting in the early 2000s. While some retractions are due to honest errors, there's an alarming increase in retractions due to misconduct, which the authors associate with systemic pressures to publish in high-impact journals. This observation is supported by evidence showing that high-ranking journals are more likely to publish fraudulent work.
2. Publication Bias and the Decline Effect: Following initial publications in high-impact journals, ensuing attempts to reproduce results often fail or reveal much weaker effects. This decline effect highlights a systemic issue of publication bias where studies with significant or novel findings are favored over replication studies, which are frequently given less priority or are published in lower-ranking journals.
3. Methodological Soundness and Reliability: An intriguing contradiction discussed is that higher-ranked journals do not necessarily correlate with higher methodological quality. Studies in fields like crystallography and psychology reveal no significant correlation between journal rank and adherence to methodological guidelines or principles.
4. Impact and Citation Practices: While journal rank is thought to predict the novelty and impact of research, the authors argue that its actual predictive power regarding subsequent citations is weak to moderate. The entanglement of publication visibility and article quality clouds the attribution of a journal's perceived quality to inherent article importance.
5. Social and Economic Implications: The paper details the indirect costs of journal rank, noting the staggering expense of peer review processes and the journal rank cascade, which convolutes scientific publishing practices. There is also mention of potential distortions in interdisciplinary research and the dissemination of scientific findings both within and outside the scientific community.

Implications and Proposed Reforms

The authors extrapolate their findings to discuss the theoretical and practical implications for the scientific community:

• Systemic Risks: Continuing reliance on journal rank could perpetuate an environment inclined to favor marketability over scientific rigor, thereby potentially inflating the number of unreliable scientific claims.
• Public Trust: The correlation between journal rank, misconduct, and the decline effect may contribute to eroding public trust in scientific institutions, which is detrimental given the increasing importance of scientific literacy in addressing complex global challenges.
• Alternative Models: The paper suggests the need for a paradigm shift towards a library-based scholarly communication system that leverages modern information technology. Such a system would focus on improving the filtering, sorting, and discovery functions of scholarly publications, ultimately reducing the undue influence of journal rank.

Conclusion and Future Considerations

The paper makes a compelling case for reassessing the role of journal rank in scientific communication and proposes initiatives to transition to more dynamic and article-focused evaluation systems. Given the intricate relationship between publication practices and scientific reliability documented in this review, future developments in artificial intelligence and computational approaches could play a pivotal role in shaping innovative metrics for evaluating scientific contributions, ensuring they align closely with both scientific and societal interests. It is crucial for ongoing research to further explore and validate alternative systems that enhance the integrity and impact of scientific communication in a transparent and equitable manner.