The weakening relationship between the Impact Factor and papers' citations in the digital age (1205.4328v1)

Abstract: Historically, papers have been physically bound to the journal in which they were published but in the electronic age papers are available individually, no longer tied to their respective journals. Hence, papers now can be read and cited based on their own merits, independently of the journal's physical availability, reputation, or Impact Factor. We compare the strength of the relationship between journals' Impact Factors and the actual citations received by their respective papers from 1902 to 2009. Throughout most of the 20th century, papers' citation rates were increasingly linked to their respective journals' Impact Factors. However, since 1990, the advent of the digital age, the strength of the relation between Impact Factors and paper citations has been decreasing. This decrease began sooner in physics, a field that was quicker to make the transition into the electronic domain. Furthermore, since 1990, the proportion of highly cited papers coming from highly cited journals has been decreasing, and accordingly, the proportion of highly cited papers not coming from highly cited journals has also been increasing. Should this pattern continue, it might bring an end to the use of the Impact Factor as a way to evaluate the quality of journals, papers and researchers.

• The paper finds the strong correlation between journal Impact Factors and individual paper citations has significantly weakened since the 1990s, driven by digital dissemination.
• The study reveals an increase since 1990 in highly cited papers published outside high-Impact Factor journals, challenging traditional reliance on IF as a quality proxy.
• The results suggest researchers and institutions should reevaluate scholarly assessment, moving towards more sophisticated metrics and models beyond the traditional Impact Factor.

The Weakening Relationship Between the Impact Factor and Papers’ Citations in the Digital Age

The paper by Lozano, Larivière, and Gingras offers a comprehensive analysis of the evolving dynamics between journal Impact Factors (IF) and the citation rates of individual papers. This paper spans over a century, from 1902 to 2009, encapsulating a large dataset across numerous scientific disciplines. The core investigation focuses on whether the digital dissemination of scientific literature has significantly weakened the historical correlation between a journal's IF and the quality inferred by its citation count.

Research Context and Methodology

Traditionally, the IF has served as a pivotal metric designed to guide librarians in journal selection and acquisition decisions. The metric, calculated by Thomson Reuters, is based on a two-year citation window and has long been considered a surrogate for paper quality, albeit with noted limitations and criticisms. It is calculated as the average number of citations received in a particular year by papers published in a journal during the previous two years.

The paper employs an extensive dataset from the Web of Science, encompassing over 30 million papers and upwards of 819 million citations, focusing on natural and medical sciences, physics, and social sciences. The authors utilized indicators such as the coefficient of determination (r²) to assess the strength of the relationship between journals' IFs and papers' citations over time.

Key Findings

The paper reveals several noteworthy trends:

• A strong correlation between IF and paper citations persisted for the majority of the 20th century. However, since the 1990s, coinciding with the rise of electronic access to academic literature, this correlation has been diminishing.
• Specifically, in physics, which was an early adopter of electronic dissemination technologies, the weakening of this correlation became apparent as early as the late 1980s.
• The proportion of highly cited papers not published in high-IF journals has increased since 1990, highlighting a broader distribution of impactful papers across various journals.

Implications

The diminishing power of the IF as an indicator of paper quality is significant for both theoretical and practical aspects of academic publishing. Theoretically, this trend challenges the traditional reliance on IF as a proxy for quality, suggesting a paradigm shift where individual paper merit is judged independently of the journal in which it is published. This has implications for how researchers, academic institutions, and evaluators assess scholarly impact. Practically, the findings encourage the reevaluation of academic publishing models and merit assessment, necessitating more sophisticated and possibly multifactorial approaches to measure academic contributions.

Future Directions

In future research, it will be crucial to explore alternative bibliometric indicators that may offer a more nuanced understanding of paper quality and impact. As electronic and open-access journals continue to proliferate, developing robust metrics that reflect the diverse channels through which scientific work is disseminated and acknowledged becomes essential. Moreover, understanding how these trends differ across disciplines and the extent to which they influence decisions in research funding and career advancement would be invaluable.

In conclusion, this research highlights a potential shift in how the quality of research outputs is assessed in the digital age, with the IF losing its historic grip as the predominant measure of scholarly value. Such insights prompt a thoughtful reconsideration of academic performance metrics and expect changes in the landscape of scientific communication.