Do altmetrics correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective (1401.4321v1)

Abstract: An extensive analysis of the presence of different altmetric indicators provided by Altmetric.com across scientific fields is presented, particularly focusing on their relationship with citations. Our results confirm that the presence and density of social media altmetric counts are still very low and not very frequent among scientific publications, with 15%-24% of the publications presenting some altmetric activity and concentrating in the most recent publications, although their presence is increasing over time. Publications from the social sciences, humanities and the medical and life sciences show the highest presence of altmetrics, indicating their potential value and interest for these fields. The analysis of the relationships between altmetrics and citations confirms previous claims of positive correlations but relatively weak, thus supporting the idea that altmetrics do not reflect the same concept of impact as citations. Also, altmetric counts do not always present a better filtering of highly cited publications than journal citation scores. Altmetrics scores (particularly mentions in blogs) are able to identify highly cited publications with higher levels of precision than journal citation scores (JCS), but they have a lower level of recall. The value of altmetrics as a complementary tool of citation analysis is highlighted, although more research is suggested to disentangle the potential meaning and value of altmetric indicators for research evaluation.

• The paper finds a positive yet weak correlation between altmetrics and citations, demonstrating that they capture distinct dimensions of research impact.
• It employs factor, correlation, and precision-recall analyses on over 700,000 publications to compare altmetric activity with traditional citation metrics.
• Results highlight disciplinary variations, with altmetrics being more prevalent in biomedical and social sciences, underscoring their role as a complementary evaluation tool.

Correlation Between Altmetrics and Citations: A Multidisciplinary Analysis

The paper "Do ‘altmetrics’ correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective" by Rodrigo Costas, Zohreh Zahedi, and Paul Wouters offers a detailed investigation into the presence and reliability of altmetrics across different scientific disciplines. It specifically explores the relationship between altmetric indicators provided by Altmetric.com and traditional citation counts.

Methodology

The paper utilizes a comprehensive dataset of 718,315 publications, all of which are indexed by the Web of Science (WoS). Altmetric indicators are collected from Altmetric.com, including mentions in platforms such as Twitter, Facebook, blogs, Google+, and news outlets. The correlation between these altmetrics and various citation indicators is examined using factor analysis, correlation analysis, and precision-recall methods.

Key Findings

1. Presence and Density of Altmetrics:
   • Approximately 15% of publications from the latter half of 2011 exhibit some altmetric activity. This percentage increases for more recent publications, indicating a recency bias in altmetric coverage.
   • Altmetrics are most prevalent in the "Biomedical and Health Sciences" and "Social Sciences and Humanities," where over 22% of publications show altmetric activity. In contrast, fields like "Mathematics and Computer Science" and "Natural Sciences and Engineering" show less than 10%.
2. Correlation with Citations:
   • There is a positive but weak correlation between altmetrics and citations, corroborating the idea that altmetrics do not capture the same concept of impact as citations.
   • Altmetrics, especially Twitter mentions, correlate with both citations and journal impact metrics, albeit moderately. Blog mentions show a relatively stronger correlation with citations compared to other altmetric indicators.
3. Precision and Recall Analysis:
   • The precision-recall analysis reveals that journal citation scores (JCS) generally outperform altmetrics in identifying highly cited publications. However, for low recall levels, altmetrics, particularly blog mentions, exhibit higher precision in filtering highly cited papers.
4. Disciplinary Variations:
   • Significant disciplinary differences exist regarding the efficacy of altmetrics. For instance, in the "Social Sciences and Humanities," altmetrics show a higher activity level, potentially adding value in these fields where traditional citation metrics often fall short.
   • The "Life and Earth Sciences" show a rising trend but still underscore the dominance of traditional citation metrics in these fields.
   • For "Mathematics and Computer Science," both altmetrics and JCS display low precision and recall, highlighting the limited value of current altmetrics for these disciplines.

Practical and Theoretical Implications

The findings suggest that while altmetrics provide a timely, albeit less comprehensive, alternative to citation counts, their role as a replacement for traditional citation metrics is limited. Instead, altmetrics should be viewed as a complementary tool that offers additional insights, particularly for recent publications and in fields traditionally underrepresented in citation databases, such as the humanities and social sciences.

Speculations on Future Developments

Future research should focus on addressing the current limitations of altmetrics, including their low coverage and recency bias. Enhanced tracking and aggregation methods could improve the reliability and coverage of altmetrics. Moreover, interdisciplinary studies combining quantitative and qualitative approaches might better elucidate the societal and cultural impacts that altmetrics potentially capture. Technological advancements in AI could further refine altmetric data collection and sentiment analysis, providing more granular and insightful evaluations of scientific impact.

Conclusion

The paper by Costas, Zahedi, and Wouters offers nuanced evidence that, although altmetrics and citations are positively correlated, they capture different dimensions of scientific impact. Altmetrics should be leveraged as complementary tools in research evaluation, especially for recent works and in disciplines where they show higher relevance. Future research and technological improvements can enhance the utility and precision of altmetrics, contributing to a more holistic understanding of scientific impact.