MasterSet: A Large-Scale Benchmark for Must-Cite Citation Recommendation in the AI/ML Literature

Abstract: The explosive growth of AI and machine learning literature -- with venues like NeurIPS and ICLR now accepting thousands of papers annually -- has made comprehensive citation coverage increasingly difficult for researchers. While citation recommendation has been studied for over a decade, existing systems primarily focus on broad relevance rather than identifying the critical set of ``must-cite'' papers: direct experimental baselines, foundational methods, and core dependencies whose omission would misrepresent a contribution's novelty or undermine reproducibility. We introduce MasterSet, a large-scale benchmark specifically designed to evaluate must-cite recommendation in the AI/ML domain. MasterSet incorporates over 150,000 papers collected from official conference proceedings/websites of 15 leading venues, serving as a comprehensive candidate pool for retrieval. We annotate citations with a three-tier labeling scheme: (I) experimental baseline status, (II) core relevance (1--5 scale), and (III) intra-paper mention frequency. Our annotation pipeline leverages an LLM-based judge, validated by human experts on a stratified sample. The benchmark task requires retrieving must-cite papers from the candidate pool given only a query paper's title and abstract, evaluated by Recall@$K$. We establish baselines using sparse retrieval, dense scientific embeddings, and graph-based methods, demonstrating that must-cite retrieval remains a challenging open problem.

• The paper introduces a must-cite citation recommendation benchmark with a novel three-tier annotation scheme using 153,373 papers from top AI/ML venues.
• It evaluates diverse retrieval methods—sparse, dense, and graph-based—revealing that contrastive fine-tuning notably improves recall but still leaves many key citations unrecovered.
• The study underscores the challenge of balancing lexical matching with semantic relevance, offering practical insights for enhancing citation integrity in manuscript preparation.

MasterSet: Benchmarking Must-Cite Citation Recommendation in AI/ML

Motivation and Task Framing

The exponential expansion of AI and ML literature in the past decade has rendered comprehensive citation auditing increasingly infeasible in manual workflows. Existing citation recommendation systems focus on general relevance, but omit functional differentiation—critically missing the operational definition of "must-cite" papers: foundational methods, direct experimental baselines, datasets, and primary dependencies. MasterSet addresses this deficit by formally defining and benchmarking must-cite recommendation, a task characterized by a small positive set, high recall requirement, and substantial consequences for omission.

Must-cite recommendation diverges from local placeholder prediction and general bibliography expansion. Rather than optimizing for broad topical coverage, it demands precise retrieval of indispensable prior work central to scientific claims, comparison validity, and reproducibility.

Dataset and Annotation Framework

MasterSet comprises 153,373 papers from 15 top venues covering core ML, general AI, CV, NLP, and probabilistic modeling, collected strictly from official proceedings and websites to ensure authoritative paper counts and eliminate preprint conflation and aggregator artifacts. This procedure results in a domain-focused, well-validated candidate pool absent in prior open scholarly corpora.

Citation instances are labeled via a three-tier scheme:

• Type I: Binary assignment for direct experimental baselines—captures papers that are actively compared or whose resources (datasets/tasks) underlie analysis and evaluation.
• Type II: Scalar (1–5) relevance reflecting methodological and task overlap; must-cite threshold at $y^{\mathrm{II}} \geq 4$ operationalizes substantial functional role.
• Type III: Intra-paper citation frequency threshold ($N \geq 3$) denotes sustained engagement, detecting papers cited repeatedly across sections.

Labels are produced using an LLM-based judge (Gemini 2.5 Flash) with strict prompt design and extensive few-shot anchoring. Human expert validation (510 instances, 6 annotators) demonstrates LLM–human agreement within expert disagreement bounds, confirming annotation reliability. Krippendorff's $\alpha$ values of 0.461 (Type I) and 0.492 (Type II) are consistent with analogous subjective tasks.

Figure 1: Label distributions across the three must-cite tiers. (a) Type I exhibits a 1:3 positive-to-negative ratio. (b) Type II displays the 1–5 scale with the must-cite threshold ($y^{\mathrm{II}} \geq 4$) indicated. (c) Type III reveals heavy right skew; most references are mentioned only once or twice.

Benchmark Task Definition

Given only the title and abstract of a query paper, the must-cite retrieval task is to rank papers in the candidate pool such that the functionally indispensable set appears at the top. This strictly cold-start setting aligns with real-world manuscript preparation scenarios—authors seeking citation auditing prior to full draft completion.

Evaluation leverages Recall@$K$, prioritizing coverage of all must-cite targets. Additional metrics (MAP, MRR, nDCG@$K$, HR@$K$) facilitate nuanced ranking assessment. The benchmark pool, MasterSet-CoreML-v1, contains 67,761 papers (2018–2024) with evaluation queries drawn exclusively from NeurIPS, ICML, and ICLR (2025).

Figure 2: Empirical and cumulative distributions of in-paper mention frequency for cited works; threshold selection for Type III is motivated by sharp separation between incidental (1–2 mention) and sustained (≥3) citation engagement.

Baseline Methods and Experimental Results

Baseline paradigms include:

• Sparse Retrieval: BM25, optimized for term overlap.
• Dense Retrieval: SciBERT, SPECTER, SPECTER2, SciNCL, ColBERT, with contrastive fine-tuning (NT-Xent) evaluated for signal enhancement.
• Graph-Based Models: KTR, AR-GNN, LitFM—incorporate citation graphs via GNNs and reasoning paths.

Results establish must-cite retrieval as substantially harder than general citation recommendation. No method exceeds 50% Recall@100 on any ground truth tier. SciBERT-NTX yields highest recall (Type I: 0.383, Type II: 0.410, Type III: 0.493), but over half of must-cite papers remain unrecovered in top ranks. Improvements observed with NT-Xent contrastive fine-tuning vastly outpace architectural novelty; vanilla scientific embeddings are insufficient for task-specific signal.

BM25 retains competitive advantage for basis-rich tasks, exploiting shared terminology among baselines, datasets, and query papers, though it underperforms semantic methods on core relevance metrics. Graph methods perform poorly in cold-start due to absent citation neighborhood at inference.

Analysis and Theoretical Implications

The triple-label scheme exposes divergence in retrieval strengths:

• Surface lexical matching (BM25) recovers experimental baselines and frequently cited works but misses semantically central dependencies.
• Dense encoders with contrastive supervision (SciBERT-NTX, ColBERT-NTX) capture task/method interplay but cannot fully overcome sparse signal dominance for highly technical baseline comparisons.
• Type II (core relevance) labels subsume cases unaddressed by the other tiers, indicating need for more nuanced embeddings exploiting joint text and structure.

Contradicting expectations, models pretrained on citation structure (SPECTER, SciNCL) deliver only mid-tier performance. The dominant improvement arises from task-specific fine-tuning, suggesting that must-cite retrieval requires information not sufficiently encoded by general citation-informed pretraining.

Practical and Ethical Considerations

MasterSet’s sourcing guarantees data lineage, exact paper counts, and reproducibility, mitigating aggregator errors and preprint confusion. Annotation via LLM—while efficient—necessarily inherits bias from model training, but prompt design and human calibration keep reliability within community tolerances.

The benchmark's modular annotation pipeline enables adaptation to diverse scientific domains, permitting functional redefinition of must-cite status and transferability of infrastructure.

Automated must-cite auditing has significant utility for scholarly diligence, reducing inadvertent omission of foundational work and improving manuscript integrity. However, the tool should supplement, not replace, expert judgment; algorithmic benchmarking is not a prescription for normative citation practices.

Future Research Directions

Substantial headroom exists for innovation: architectures exploiting joint tier supervision, leveraging more granular input beyond title and abstract, and integrating citation graph augmentation could improve recall. Extending the benchmark to additional venues and tasks, integrating deep revision histories (e.g., reviewer-identified missed citations), and embedding auditing in manuscript workflows would expand practical impact.

Conclusion

MasterSet represents a large-scale, operational benchmark for must-cite citation recommendation, grounded in precise data collection, robust annotation, and rigorous baseline assessment. The persistent gap in recall across retrieval paradigms evidences the functional complexity of must-cite auditing. The dataset, annotation pipeline, and baseline implementations are released for reproducibility and continued research. The must-cite recommendation task remains open, presenting unique challenges for automated scientific knowledge discovery and citation integrity in AI/ML.