Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics (1406.7091v2)

Abstract: Today, it is not clear how the impact of research on other areas of society than science should be measured. While peer review and bibliometrics have become standard methods for measuring the impact of research in science, there is not yet an accepted framework within which to measure societal impact. Alternative metrics (called altmetrics to distinguish them from bibliometrics) are considered an interesting option for assessing the societal impact of research, as they offer new ways to measure (public) engagement with research output. Altmetrics is a term to describe web-based metrics for the impact of publications and other scholarly material by using data from social media platforms (e.g. Twitter or Mendeley). This overview of studies explores the potential of altmetrics for measuring societal impact. It deals with the definition and classification of altmetrics. Furthermore, their benefits and disadvantages for measuring impact are discussed.

• The paper demonstrates that altmetrics extend citation metrics by capturing diverse, real-time engagement from social media and online platforms.
• It highlights the benefits of evaluating research impact more broadly, rapidly, and openly through non-traditional scholarly outputs.
• It identifies challenges such as data quality issues, manipulation risks, and commercialization that may hinder the reliability of altmetrics.

Analyzing Altmetrics: Assessing the Societal Impact of Research

The paper "Do Altmetrics Point to the Broader Impact of Research?" by Lutz Bornmann offers a comprehensive exploration of the utility and limitations of alternative metrics (altmetrics) in evaluating the societal impact of scholarly works. As traditional bibliometrics focus predominantly on citation counts within academic circles, altmetrics endeavors to quantify engagement from broader societal sectors via web-based platforms, introducing new dimensions to research impact assessments.

Key Concepts and Classification

Altmetrics broadens the scope of research impact evaluation by considering interactions on social media and various online platforms, such as Twitter and Mendeley. This approach is posited as a means to capture the "public" engagement with research output, supplementing traditional citation metrics. Bornmann highlights the fundamental distinctions between altmetrics and bibliometrics and discusses various platforms where altmetrics are applied, including bookmarking, reference managers, and microblogging.

The paper provides classification schemes by organizations like ImpactStory and PLOS, which categorize altmetrics based on user engagement activities such as viewing, saving, discussing, recommending, and citing scholarly products.

Benefits and Advancements of Altmetrics

Altmetrics offers benefits in four main areas: broadness, diversity, speed, and openness.

1. Broadness: Altmetrics captures the societal impact beyond academic circles, enabling researchers to understand how their work resonates with the general public, policy-makers, and professionals.
2. Diversity: By assessing various scholarly products—beyond traditional peer-reviewed journals—altmetrics evaluate datasets, software, and other non-traditional outputs.
3. Speed: Unlike citation metrics, altmetrics can provide immediate feedback post-publication, offering a real-time perspective of impact.
4. Openness: Altmetrics data is increasingly accessible via APIs, easing the measurement of societal impact.

Challenges and Limitations

Despite its advantages, altmetrics face significant challenges:

• Commercialization: Social media platforms, driven by commercial motives, might inflate engagement figures, biasing impact measurements.
• Data Quality: Issues such as bias, ambiguity in user data, lack of measurement standards, and difficulty in normalizing data pose major hurdles.
• Manipulation Risks: The ease of artificially inflating altmetrics through fake social media accounts undermines their reliability.

Theoretical and Practical Implications

Altmetrics hold promise for expanding research assessments into societal and policy impacts. Yet, their adoption in formal evaluation systems remains cautious due to concerns of validity and susceptibility to manipulation.

Bornmann posits that while altmetrics ca potentially redefine how impact is measured, empirical studies are needed to verify their reliability and validity. Investigating whether altmetric data aligns with expert assessments of societal impact is pivotal for their integration into research evaluation frameworks.

Conclusion and Future Directions

The use of altmetrics in research evaluations is evolving, with growing interest in understanding broader societal impacts. As the field matures, developing rigorous methodologies to address data quality, representativeness, and manipulation remains crucial. Future research efforts should aim to establish robust theoretical foundations, provide empirical validation, and refine methodological approaches to enable altmetrics to complement traditional metrics in research evaluation.

In sum, altmetrics should be perceived as a supplementary tool offering additional insights rather than a wholesale replacement for traditional metrics. Their integration into informed peer review processes would enhance the comprehensiveness of research evaluation, fostering a nuanced understanding of scholarly impact in the digital age.