Measuring Research Interest Similarity with Transition Probabilities (2409.18240v1)

Abstract: We propose a method to measure the similarity of papers and authors by simulating a literature search procedure on citation networks, which is an information retrieval inspired conceptualization of similarity. This transition probability (TP) based approach does not require a curated classification system, avoids clustering complications, and provides a continuous measure of similarity. We perform testing scenarios to explore several versions of the general TP concept and the Node2vec machine-learning technique. We found that TP measures outperform Node2vec in mapping the macroscopic structure of fields. The paper provides a general discussion of how to implement TP similarity measurement, with a particular focus on how to utilize publication-level information to approximate the research interest similarity of individual scientists. This paper is accompanied by a Python package capable of calculating all the tested metrics.

• The paper introduces a novel TP-based measure that models literature search as a random walk to capture research interest similarity.
• It compares the TP method with Node2vec, demonstrating superior predictive performance with AUC scores around 0.9 for co-authorship prediction.
• The study highlights scalable approximations like estimated TP, enabling robust analysis of extensive citation networks without clustering bias.

Assessing Research Interest Similarity Using Transition Probabilities

Introduction

The paper "Measuring Research Interest Similarity with Transition Probabilities" by Varga et al. presents a novel method to gauge the similarity between academic papers and authors by modeling a literature search process as a random walk through citation networks. The transition probability (TP)-based approach is a significant departure from traditional methods, as it avoids the pitfalls of curated classification systems, clustering complications, and provides continuous similarity measures. This paradigm is contrasted with the Node2vec (N2V) machine learning technique, with the authors demonstrating the superiority of TP measures in capturing the macroscopic structure of academic fields.

Conceptual Framework

Traditional methods for assessing researcher similarity often rely on discrete representations of topics, whether through keyword thesauri, bibliographic coupling, or topic modeling. While continuous representations via vector spaces have advanced this domain, their opacity and the interpretability of the resulting embeddings pose challenges. This paper's main contribution is a family of similarity measures based on transition probabilities in citation networks, aligning with the conceptualization of literature search behavior. This random walk model interprets research similarity as the likelihood that two papers are retrieved together during a literature search.

Methodology

The TP measure is symmetrized and free from classification or clustering dependencies. It calculates the similarity between two papers proportional to the probability that a random walk from one paper reaches the other within a certain number of steps. This measure can be aggregated to represent research interest similarity at higher levels, such as authors, fields, or institutions. Specific alternatives to the TP measure include shortest path length (SP), average shortest path TP (𝑆𝑇), and an estimated TP (𝐸𝑇) via simulations. Node2vec (N2V) embeddings were also evaluated as a benchmark against these measures.

Experimental Design and Data

The empirical analysis employed co-authorship prediction to evaluate similarity measures at the local level, juxtaposing against macro-level disciplinary mappings. Citation data from the Web of Science’s Science Citation Index, coupled with author disambiguation from the Microsoft Academic Graph, formed the basis for these experiments. Selected fields included Astronomy & Astrophysics, Clinical Neurology, Sociology, and a multidisciplinary journal set.

Results and Implications

1. Predictive Performance:
   • The TP measure emerged as the most effective, with AUC scores around 0.9 for co-authorship prediction and 0.71 for disciplinary classification.
   • Node2vec performed well in local dynamics (co-authorship prediction) but poorly in macro structure mapping, aligning with the hypothesis about its coarse representation of lower similarity ranges.
2. Runtime and Scalability:
   • While direct computation of TP (𝑇) is computationally intensive, estimated TP (𝐸𝑇) and 𝑆𝑇 offer scalable solutions for network sizes unfeasible for exact 𝑇 calculations.
3. Correlations and Utility:
   • 𝐸𝑇 exhibited a strong correlation with 𝑇, validating it as a practical approximation despite some zero estimates due to network sparsity.
   • N2V showed no correlation with node degrees, contrasting with TP measures that tend to correlate with nodal degrees, reflecting the inherent citation network structure and visibility bias.

Practical and Theoretical Implications

The TP-based method advances the precision of measuring research interest similarity without the need for curated classifications or complex clustering. This methodological framework encourages a reconceptualization of scholarly communication and enhances our understanding of interdisciplinary and intra-disciplinary collaborations. By embracing a random walk model, the paper proposes a more intuitive and interpretable mechanism for mapping scientific domains.

Future Developments

Continued development of computational tools, such as the provided Python package, could streamline the application of these metrics in various research evaluation contexts. Future research might extend this framework's scope to explore temporal dynamics of research similarities or adapt it to emerging fields with evolving citation patterns.

Conclusion

Varga et al.'s paper presents a compelling case for the adoption of transition probability-based measures in scientometrics, offering significant improvements in evaluating research interest similarity. This approach not only enhances analytical precision but also integrates a more human-centric understanding of scholarly inquiry, promising better alignment between computational methods and conceptual research frameworks.