Grand challenges in altmetrics: heterogeneity, data quality and dependencies (1603.04939v1)

Abstract: As uptake among researchers is constantly increasing, social media are finding their way into scholarly communication and, under the umbrella term altmetrics, were introduced to research evaluation. Fueled by technological possibilities and an increasing demand to demonstrate impact beyond the scientific community, altmetrics received great attention as potential democratizers of the scientific reward system and indicators of societal impact. This paper focuses on current challenges of altmetrics. Heterogeneity, data quality and particular dependencies are identified as the three major issues and discussed in detail with a particular emphasis on past developments in bibliometrics. The heterogeneity of altmetrics mirrors the diversity of the types of underlying acts, most of which take place on social media platforms. This heterogeneity has made it difficult to establish a common definition or conceptual framework. Data quality issues become apparent in the lack of accuracy, consistency and replicability of various altmetrics, which is largely affected by the dynamic nature of social media events. It is further highlighted that altmetrics are shaped by technical possibilities and depend particularly on the availability of APIs and DOIs, are strongly dependent on data providers and aggregators, and potentially influenced by technical affordances of underlying platforms.

• The paper rigorously analyzes altmetrics challenges by dissecting the heterogeneity of social media interactions and their impact on research evaluation.
• It highlights data quality issues arising from dynamic online platforms and inconsistent metadata practices that undermine metric reliability.
• It uncovers dependencies on commercial data providers, which compromise the stability and academic legitimacy of altmetrics.

Analyzing the Core Challenges in Altmetrics in Scholarly Communication

Stefanie Haustein's paper examines the existing challenges faced by the field of altmetrics—an emerging area within research evaluation that attempts to gauge scholarly impact through social media metrics. The article primarily dissects the heterogeneity, data quality issues, and dependencies embedded within altmetrics, drawing upon historical and conceptual similarities with traditional bibliometrics. Altmetrics, a term that summarizes the variety of metrics derived from online social platforms, faces significant hurdles due to their diverse nature, the inconsistencies in data quality, and reliance on technical infrastructures.

Heterogeneity in Altmetrics

The paper identifies heterogeneity as a substantial challenge within altmetrics. This diversity stems from the range of social media interactions and platforms incorporated into altmetrics. Metrics can involve anything from social networking to blogging, each with different levels of scholarly engagement. For instance, while Mendeley readership correlates modestly with academic citations, Twitter activity does not, indicating disparate user bases and intentions. The paper argues that this variety confounds any attempt to define altmetrics with a clear-cut definition or integrate under a conceptual framework. This lack of uniformity also questions the validity of altmetrics as a unified substitute for traditional citation metrics.

Data Quality Concerns

Data quality is critically important within altmetrics, particularly given their rising use in research evaluation contexts. The dynamic nature of social media platforms, where user interactions can change or disappear, exacerbates issues of accuracy, consistency, and replicability. Haustein's analysis emphasizes how the data provided by aggregators like Altmetric and Plum Analytics may vary due to differing methodologies. Additionally, altmetrics data are profoundly influenced by metadata quality, such as the presence of DOIs, which affects usability and consistency across platforms. Ensuring high data quality remains a daunting challenge, mainly because these platforms are not fundamentally designed for academia.

Dependencies in Altmetrics

A key challenge in altmetrics stems from dependencies on data providers and platforms. The field's evolution is distinctly technology-driven, with a significant reliance on the APIs and infrastructure of platforms like Twitter and Mendeley—the continuation of which is uncertain. This technological dependence not only affects the collection of metrics but also influences user behavior. For-profit entities like Altmetric and Mendeley play an outsized role, potentially skewing the ecosystem toward commercial interests rather than scientific integrity. This concern raises pertinent questions about data reliability and the potential bias introduced in altmetrics.

Implications and Future Perspectives

Haustein’s paper underscores the urgency of addressing the shortcomings of altmetrics if they are to play a role in comprehensive research evaluation systems. While they offer a broadened perspective on research impact beyond traditional citations, the lack of a unified framework limits their integration into scholarly evaluation. The heterogeneity and data quality issues, coupled with significant dependencies, necessitate cautious application and ongoing refinement in altmetrics strategies. As researchers and institutions continue to navigate this evolving metric landscape, a balanced and robust approach to using altmetrics could ensure a more nuanced and inclusive assessment of scientific contribution. Moving forward, establishing robust standards and engaging in further theoretical explorations will be essential in shaping the future utility of altmetrics within academic and empirical contexts.