SciImpact: A Multi-Dimensional, Multi-Field Benchmark for Scientific Impact Prediction

Abstract: The rapid growth of scientific literature calls for automated methods to assess and predict research impact. Prior work has largely focused on citation-based metrics, leaving limited evaluation of models' capability to reason about other impact dimensions. To this end, we introduce SciImpact, a large-scale, multi-dimensional benchmark for scientific impact prediction spanning 19 fields. SciImpact captures various forms of scientific influence, ranging from citation counts to award recognition, media attention, patent reference, and artifact adoption, by integrating heterogeneous data sources and targeted web crawling. It comprises 215,928 contrastive paper pairs reflecting meaningful impact differences in both short-term (e.g., Best Paper Award) and long-term settings (e.g., Nobel Prize). We evaluate 11 widely used LLMs on SciImpact. Results show that off-the-shelf models exhibit substantial variability across dimensions and fields, while multi-task supervised fine-tuning consistently enables smaller LLMs (e.g., 4B) to markedly outperform much larger models (e.g., 30B) and surpass powerful closed-source LLMs (e.g., o4-mini). These results establish SciImpact as a challenging benchmark and demonstrate its value for multi-dimensional, multi-field scientific impact prediction. Our project homepage is https://flypig23.github.io/sciimpact-homepage/

• The paper presents a novel multi-dimensional benchmark that integrates 7 impact dimensions across 19 scientific fields to advance scientific impact prediction.
• The methodology employs contrastive artifact pairing with 215,928 curated pairs and strict quality controls to ensure reliable evaluation across diverse metrics.
• Empirical findings reveal that multi-task supervised fine-tuning enables smaller models to outperform larger LLMs, underscoring the value of high-quality multidimensional supervision.

SciImpact: A Comprehensive Multi-Dimensional Benchmark for Scientific Impact Prediction

Motivation and Contributions

The inability of conventional citation-based metrics to capture the complex, multi-faceted nature of scientific impact has catalyzed the need for more comprehensive evaluation frameworks. "SciImpact: A Multi-Dimensional, Multi-Field Benchmark for Scientific Impact Prediction" (2604.17141) responds to this by constructing a rigorous benchmark spanning 19 scientific fields and seven heterogeneous impact dimensions, including citation count, awards, patents, media attention, and adoption of code, datasets, and models.

The authors carefully curate 215,928 contrastive artifact pairs, leveraging both existing data repositories and extensive web crawling to address coverage gaps across disciplines and impact signals. The resulting resource notably exceeds the scope and flexibility of previous datasets, enabling robust model assessment and advancing the study of scientific impact as a complex, multi-dimensional phenomenon.

Figure 1: The end-to-end pipeline of the SciImpact benchmark curation process, integrating candidate retrieval, impact labeling, pair generation, and rigorous quality control.

Benchmark Construction and Data Composition

SciImpact's benchmark design addresses previously neglected dimensions of impact by operationalizing seven distinct measures:

• Citation Count: Standard bibliometric indicator sourced from MAPLE and OpenAlex.
• Award Recognition: Explicit inclusion of high-profile awards such as the Nobel Prize and conference Best Paper Awards, with rigorous negative sampling controls.
• Patent References and Media Attention: Integration of citation data from patents and statistics on societal attention, assembled through systematic linkage to SciSciNet.
• Artifact Adoption: Adoption rates for code, datasets, and model artifacts, with statistics drawn from platforms like GitHub and Hugging Face.

Each benchmark instance presents a binary classification problem between two artifacts (e.g., papers, datasets, or codebases), ensuring matched context (year, venue, author, or field) and statistically meaningful contrasts based on empirically defined thresholds.

Dataset statistics reveal pronounced variation in textual input lengths and semantic content across dimensions—abstracts for classic impact indicators, versus extended artifact documentation for code/dataset/model dimensions. This heterogeneity underscores the challenge of generalizing across formats and knowledge domains.

Figure 2: Input word count distributions across dimensions reflect the substantial variability in textual artifact content.

Experimental Design and Evaluation Protocol

The evaluation protocol standardizes prompt construction and model output, decoupling impact assessment from chronological or direct popularity leakage. LLM inputs are capped at 1,000 words per artifact to control for context length and enable fair cross-model comparison.

The study considers 11 LLMs (3 closed-source, 8 open-weight) and their supervised fine-tuned (SFT) counterparts. Notably, training is performed using multi-dimensional and multi-field aggregation with strict validation splits, facilitating statistical significance testing of model gains and robust ablation studies.

Empirical Findings

Fine-Tuning Efficacy and Model Scaling

A principal finding is that multi-task SFT on SciImpact consistently enables smaller open-weight models (e.g., SFT-Qwen3-4B, SFT-LLaMA-3.2-3B) to surpass large-scale open and closed-source LLMs in average pairwise impact prediction accuracy. This result challenges assumptions that pre-trained model scale alone determines scientific inference performance, instead highlighting the decisive leverage of high-quality, multi-dimensional supervision.

Figure 3: Supervised fine-tuning transforms a 4B open-weight model into a competitive rival to state-of-the-art closed-source systems across all impact dimensions.

Dimension and Field Heterogeneity

Performance stratifies sharply by impact dimension and scientific field. Predicting award recognition (especially Nobel Prizes) is empirically easier—average SFT accuracy on Nobel Prize assignments exceeds 0.89—potentially due to well-defined committee preferences and canonical accomplishment language within abstracts. In contrast, patent and media impact dimensions are substantially more challenging, likely reflecting the influence of non-textual, extrinsic context.

Field-wise breakdown further reveals that physics, chemistry, and medicine afford higher SFT accuracy than computer science or more heterogeneous "other fields," suggesting domain-specific textual regularities that are amenable to LLM reasoning.

These findings support the theoretical claim that both impact domain and disciplinary context substantially modulate the tractability of text-driven scientific influence prediction.

Supervision Complementarity

Ablation demonstrates that training exclusively on a single dimension yields lower average model performance than multi-dimensional SFT, even when evaluated on the original dimension. This indicates that supervision signals across different impact facets are largely complementary, enabling models to better generalize via multi-task learning.

Robustness to Temporal and Popularity Leakage

Empirical analysis reveals minimal sensitivity of artifact impact prediction accuracy to publication epoch, as well as insensitivity to explicit popularity cues within artifact texts (e.g., removal of badges and download metrics does not undermine accuracy). This establishes SciImpact's robustness to certain forms of leakage that compromise naive impact prediction datasets.

Field Performance Comparison

Figure 4: Model accuracy on citation prediction remains relatively consistent across publication periods, highlighting negligible temporal bias.

Implications and Future Directions

SciImpact's multidimensionality and strict protocol set a new standard for evaluating scientific impact prediction. Practically, its rigorous control for field, dimension, and textual form raises the bar for LLMs intended as decision-support for science policy, discovery, or strategy.

Theologically, the results demonstrate that LLMs' understanding of "impact" is shaped as much by training task and data coverage as by model scale or pre-training regime. The advantage imparted by SFT on SciImpact indicates that automating evaluative scientific reasoning is a learnable, data-centric problem—one not solved exclusively by scaling.

Several limitations remain—context truncation, formulation as binary comparison, and the presence of artifacts that may have been seen during LLM pre-training. Future work should focus on extending context, explicit forecasting, and enhancing temporal robustness, as well as expanding to further dimensions such as policy and societal impact beyond publicly available indicators.

Conclusion

SciImpact constitutes a methodologically rigorous, resource-rich benchmark for multi-dimensional scientific impact prediction, advancing the scope, resolution, and external validity of research in this area. The demonstrated utility of task-specific SFT, notable dimension- and field-related generalization effects, and stability against trivial leakage collectively highlight the benchmark's strength as a foundation for the next generation of scientific metareasoning systems.