SPARBench: Academic Retrieval Benchmark

• SPARBench is an expert-annotated academic retrieval benchmark featuring realistic, multi-intent queries and rigorous, multi-stage annotation.
• It employs LLM-based coarse-to-fine filtering combined with expert validation over a diverse document pool from ArXiv, PubMed, OpenAlex, and Semantic Scholar.
• The benchmark provides balanced metrics like F1, precision, and recall while stress-testing retrieval systems under complex query conditions.

SPARBench is an expert-annotated benchmark for academic paper retrieval, introduced to enable systematic, realistic, end-to-end evaluation of scholarly search systems. It was developed to address the limitations of prior benchmarks, which tended to be small-scale, uni-domain, based on synthetic or oversimplified queries, or lacking in rigorous relevance annotations. SPARBench specifically incorporates multi-intent, reference-driven queries over a large-scale, multi-source corpus, with labels established via multi-stage LLM filtering and domain-expert validation. It is designed to simulate and stress-test real-world academic search tasks, particularly those involving complex and compound information needs (Shi et al., 21 Jul 2025).

1. Motivation and Scope

SPARBench was motivated by the exponential growth of scholarly output and the corresponding challenge of literature search, where users frequently pose multi-intent, open-ended, or reference-tracing queries. Existing academic retrieval benchmarks such as AutoScholar and RealScholarQuery were identified as inadequate for modern evaluation due to issues of limited scale, narrow coverage, reliance on synthetic queries, or insufficiently rigorous human relevance judgments. SPARBench addresses these gaps by introducing:

• Realistic queries reflecting actual literature review behaviors, including compound intents, linguistic variation, minor grammatical errors or typos, and multi-domain coverage.
• A multi-source document pool spanning ArXiv, PubMed, OpenAlex, and Semantic Scholar, capturing the diversity of sources encountered in academic search.
• Rigorous annotation, combining LLM-based coarse and fine filtering with subsequent expert validation to ensure high-precision relevance labels.
• Task variety that requires systems to handle complex intent, citation expansion, and cross-domain reasoning (Shi et al., 21 Jul 2025).

2. Dataset Construction

SPARBench construction follows a four-stage pipeline:

1. Seed Query Curation: Fifty seed queries spanning realistic literature-review information needs were collected by domain experts (35 in computer science, 15 in biomedicine), including survey requests, recent advances, methodological comparisons, and case studies.
2. GPT-4o-Based Expansion and Expert Screening: Each seed was expanded using GPT-4o to generate variants mirroring natural user phrasing (including partial grammar and typos). Experts manually curated these expansions to retain the most representative and challenging 50 queries.
3. Multi-Source Retrieval: Each query was issued to four major scholarly databases, yielding approximately 198,000 raw candidate documents.
4. Three-Stage Annotation and Filtering:
   • Rough LLM-based pruning using Qwen2.5-7B, narrowing to ~3,000 candidates.
   • Fine-grained LLM judgment using Qwen2.5-72B, reducing to ~2,000.
   • Final expert validation by graduate-level annotators, yielding 560 relevant documents (average 11.2 relevant documents per query) (Shi et al., 21 Jul 2025).

No fixed train/validation/test splits exist in the initial release, as emphasis was given to annotation quality. Standardized splits are suggested for future versions.

Stage	Documents Remaining	Tools/Methods Used
Initial multi-source retrieval	~198,000	ArXiv, PubMed, OpenAlex, Semantic Scholar
LLM-based coarse filter	~3,000	Qwen2.5-7B
LLM-based fine filter	~2,000	Qwen2.5-72B
Expert-annotated relevant documents	560	Graduate annotators

3. Evaluation Metrics

SPARBench applies standard retrieval metrics at the corpus level:

Given:

• TP (True Positives): Number of retrieved documents judged relevant
• FP (False Positives): Number of retrieved documents judged irrelevant
• FN (False Negatives): Number of relevant documents not retrieved

The metrics are computed as:

• Precision: $P = \frac{TP}{TP+FP}$
• Recall: $R = \frac{TP}{TP+FN}$
• F1-score: $F1=2\cdot\frac{P\times R}{P+R}$

These values are computed per query and averaged across all 50 queries. Additionally, Recall@5 is reported (the fraction of queries where at least one relevant document is retrieved in the top-5 system responses). This provides insight into both breadth and ranking quality in retrieval (Shi et al., 21 Jul 2025).

4. Query Spectrum and Benchmarking Challenges

SPARBench’s queries are designed to span the diversity and complexity of scholarly search. Categories include:

• Survey Retrieval: E.g., requests for “a systematic review of transformer-based vision models since 2020.” Requires breadth and synthesis.
• Recent Advances: E.g., “cutting-edge methods for domain generalization in ML across multiple fields,” incorporating temporal constraints.
• Methodological Comparison: Queries demand deep technical details, such as “Compare preconditioners used in solving LSTD systems under linear value-function approximation.”
• Application & Ethics: Cross-disciplinary queries like “ethical challenges of CRISPR in treating genetic diseases.”
• Multi-Intent, Noisy Input: Queries with partial grammar, typos, or run-ons to mirror real user scenarios.

These query types collectively stress intent interpretation, citation-driven expansion, source selection, and multifaceted ranking, exceeding the complexity of synthetic or single-intent benchmarks (Shi et al., 21 Jul 2025).

5. Baseline Performance and Comparative Analysis

SPARBench has been used to evaluate a spectrum of retrieval systems, including both commercial search engines and research baselines. Performance is reported as averaged F1, Recall, and Precision. Key results are tabulated below:

Method	F1	Recall	Precision
Google (G)	–	0.000	–
Google+GPT	0.0092	0.0082	0.0106
Google Scholar	0.0043	0.0038	0.0050
ChatGPT Search	0.0045	0.0038	0.0055
Google-ArXiv	0.2451	0.2800	0.2180
Google-ArXiv+LLM	0.1923	0.1613	0.2382
PubMed+LLM	–	0.000	–
OpenAlex+LLM	0.0242	0.0988	0.0138
SemanticScholar+LLM	0.0135	0.0449	0.0080
PaSa	0.1041	0.1009	0.1076
PaperFinder	0.0418	0.1474	0.0244
SPAR (ours)	0.3015	0.3103	0.2932

SPAR significantly exceeds the strongest prior baselines: for instance, compared to PaSa (F1=0.1041), SPAR achieves an absolute increase of +0.1974 F1 and a relative gain exceeding 190%, maintaining balanced precision and recall (Shi et al., 21 Jul 2025).

6. Component Ablation and System Analysis

Ablation studies quantify the impact of individual system components on SPARBench performance:

• Query Interpretation (QInterp): Increases F1 by +0.01 (from 0.21 to 0.22) and precision by +0.13, with a recall reduction (–0.05).
• RefChain Expansion: Adding single-layer citation chaining increases recall by +0.02 (raw recall +0.05), but drops precision by –0.06.
• Query Evolution: Iterative, citation-aware reformulation yields +0.02 F1 and +0.02 precision.
• Reranking: Incorporating authority and timeliness in reranking improves Recall@5 by +0.0074 (absolute; 4.7% relative).

These results collectively highlight that query understanding, citation chain traversal, dynamic query evolution, and sophisticated reranking all make measurable and complementary contributions to scholarly retrieval effectiveness on SPARBench (Shi et al., 21 Jul 2025).

7. Insights and Recommendations for Users

SPARBench’s development process and empirical analysis suggest several key recommendations:

• Multi-Stage Annotation Is Essential: Combining LLM-based filtering (coarse-to-fine) with expert validation enables reliable relevance labeling while minimizing expert effort.
• Source and Reference Expansion Models: Adaptivity in selecting sources and traversing at least one layer of citations results in more realistic and informative evaluation.
• Stress Testing with Diverse Queries: Challenging systems with multi-intent and noisy queries foregrounds weaknesses in intent understanding and ranking.
• Balanced Metrics for Task Diversity: Both precision- and recall-oriented evaluations are needed to accommodate different retrieval-augmented use-cases (e.g., summary generation versus focused review).
• Expandability: Future extensions should increase domain coverage (e.g., physics, social sciences), expand the query set, and introduce standardized train/validation/test splits to support ongoing system development and reproducibility.

Adoption of SPARBench and its construction methodology is posited to yield retrieval models and evaluation pipelines that mirror the diversity and difficulty of real-world academic literature search (Shi et al., 21 Jul 2025).