A principal component analysis of 39 scientific impact measures (0902.2183v2)

Abstract: The impact of scientific publications has traditionally been expressed in terms of citation counts. However, scientific activity has moved online over the past decade. To better capture scientific impact in the digital era, a variety of new impact measures has been proposed on the basis of social network analysis and usage log data. Here we investigate how these new measures relate to each other, and how accurately and completely they express scientific impact. We performed a principal component analysis of the rankings produced by 39 existing and proposed measures of scholarly impact that were calculated on the basis of both citation and usage log data. Our results indicate that the notion of scientific impact is a multi-dimensional construct that can not be adequately measured by any single indicator, although some measures are more suitable than others. The commonly used citation Impact Factor is not positioned at the core of this construct, but at its periphery, and should thus be used with caution.

• The paper reveals that scientific impact is a complex, multi-dimensional construct, with traditional metrics like the JIF positioned peripherally.
• It applies PCA to 39 measures from citation and usage data, finding that the first two components account for 83.4% of the variance.
• The analysis distinguishes rapid usage-based and delayed citation-based clusters, offering actionable insights for refining scholarly evaluation.

Principal Component Analysis of Scientific Impact Measures

The paper "A Principal Component Analysis of 39 Scientific Impact Measures" by Johan Bollen et al., explores the multifaceted nature of scientific impact through a comprehensive analysis of 39 different impact measures. Conducting a Principal Component Analysis (PCA), the authors assess how these measures relate to one another and evaluate their effectiveness in expressing scientific impact, especially in the context of digital advancements.

Methodology

The paper utilizes both citation data and usage log data from multiple sources, such as the Journal Citation Reports (JCR) and the MESUR project's usage data, to derive 39 impact measures. This includes existing measures like the Journal Impact Factor (JIF) and newly proposed metrics based on social network analysis.

Citation data were extracted and structured into networks, while usage data from over 346 million user interactions were used to form usage networks. Social network measures such as degree centrality, closeness centrality, betweenness centrality, and PageRank were then calculated for both citation and usage networks.

Key Findings

1. Multi-dimensional Impact: The results confirm that scientific impact is a complex, multi-dimensional construct that cannot be adequately captured by any single measure. The JIF, in particular, is positioned at the periphery of impact measures, suggesting it is not a central indicator of scientific impact.
2. Component Analysis: The PCA shows that the first two components account for 83.4% of the variance among the measures, with a clear distinction between "Rapid" (usage-based) and "Delayed" (citation-based) indicators on PC1, and "Popularity" versus "Prestige" indicators on PC2.
3. Clusters of Measures: The analysis reveals distinct clusters, with usage measures forming a tightly knit group, and citation measures being more dispersed across different clusters indicating their varied interpretative dimensions. Usage measures appear to provide a more immediate reflection of scientific impact compared to citation measures.
4. Comparison with JIF: The paper points out that the JIF, although widely used, does not align with the consensus of other impact measures, indicating it captures a less generalizable aspect of scientific impact.

Implications

The paper suggests that impact measures derived from usage data might offer a more immediate view of scientific influence, potentially serving as better indicators of "rapid" scientific activity. It also questions the standing of traditional measures like the JIF, advocating for a broader adoption and validation of alternative, possibly more representative measures of scientific impact.

Future Directions

The authors propose further exploration into the dimensions revealed through PCA, including the potential development of new metrics that better encapsulate the multifaceted nature of scientific influence. This research opens avenues for refining scholarly assessment tools to create a more comprehensive understanding of impact in the digital age.

In summary, this paper challenges the conventional understanding of scientific impact by revealing the complex structure underlying various impact measures. The insights provided here are crucial for further studies aimed at refining the metrics used in scientific evaluation, particularly in embracing the nuances introduced by digitalization and network analyses.