Alternative metrics in scientometrics: A meta-analysis of research into three altmetrics

Abstract: Alternative metrics are currently one of the most popular research topics in scientometric research. This paper provides an overview of research into three of the most important altmetrics: microblogging (Twitter), online reference managers (Mendeley and CiteULike) and blogging. The literature is discussed in relation to the possible use of altmetrics in research evaluation. Since the research was particularly interested in the correlation between altmetrics counts and citation counts, this overview focuses particularly on this correlation. For each altmetric, a meta-analysis is calculated for its correlation with traditional citation counts. As the results of the meta-analyses show, the correlation with traditional citations for micro-blogging counts is negligible (pooled r=0.003), for blog counts it is small (pooled r=0.12) and for bookmark counts from online reference managers, medium to large (CiteULike pooled r=0.23; Mendeley pooled r=0.51).

• The paper performs a meta-analysis examining the correlation between altmetrics like Twitter, Mendeley, and blogs and traditional citation counts.
• The study found Twitter and blog counts show low correlation with citations, while Mendeley activity correlates moderately to strongly, indicating different types of engagement are captured.
• These varied correlations suggest altmetrics capture diverse impacts beyond traditional citations and require further research into user behavior and specific use cases for evaluation.

Insightful Overview of "Alternative metrics in scientometrics: A meta-analysis of research into three altmetrics"

The paper "Alternative metrics in scientometrics: A meta-analysis of research into three altmetrics" by Lutz Bornmann investigates the effectiveness of alternative metrics (altmetrics) within scientometric research, focusing on microblogging (Twitter), online reference managers (Mendeley and CiteULike), and blogging. The primary aim of the study is to assess the correlation between altmetrics and traditional citation counts as a measure of research impact, a topic of increasing interest in response to changing information dissemination methods.

The research includes a meta-analysis of existing studies to quantify these correlations. For microblogging, the outcome indicates an almost negligible correlation with traditional citation counts (pooled r=0.003). This suggests that Twitter activity does not align well with the more traditional measures of research impact and may capture a different type of engagement or popularity irrelevant to scholarly impact.

In the field of online reference managers, Mendeley showed a medium to large correlation with traditional citation counts (pooled r=0.51). The higher correlation here may reflect that Mendeley is more aligned with scholarly work and could potentially serve as a proxy for traditional impact measures. CiteULike results in a smaller pooled correlation (r=0.23), perhaps reflecting differences in user base or scope of coverage compared to Mendeley.

Blogging, on the other hand, yielded small correlation coefficients (pooled r=0.12) suggesting that blog counts do not strongly predict traditional citations. However, this indicates that blogging may capture a dimension of research impact outside the scope of traditional citational analysis, such as public engagement or societal impact.

Implications and Future Research

The practical implications of this study lie in the potential deployment of altmetrics in research evaluation. The different correlations across the three altmetric sources underscore their multifaceted nature, reflecting the diverse social media behaviors and the audiences they cater to. For instance, the modest correlation of Mendeley data suggests its possible use as a supplementary metric for research evaluation, particularly in disciplines where it has good coverage and user engagement.

Theoretically, the findings highlight the necessity for a broader understanding of what constitutes research impact. While traditional citations measure inter-researcher communication and endorsement, altmetrics could offer insights into how research permeates broader societal layers or informs policy and public discussions.

The paper calls for future research to explore the nature of these correlations. Rather than simply aggregating correlations across heterogeneous studies, it suggests examining the causes behind altmetrics data, evaluating the demographics of users who create altmetrics, and identifying specific scenarios where altmetrics provide unique insights into research impact, especially beyond academia.

Conclusion

In conclusion, Bornmann’s paper offers a comprehensive meta-analysis of altmetrics concerning traditional citation counts, stimulating further discussion on their role in contemporary research evaluation. As the scientific community continues to explore alternative pathways for assessing research influence, this analysis acts as a cornerstone for understanding altmetrics' usefulness and limitations and opens avenues for their optimization and integration into traditional bibliometrics. The consideration of altmetrics as an adjunct rather than a replacement to citations could lead to a more nuanced understanding of research influence in the digital age.