DeepScholar-Bench: Live Evaluation of AI Synthesis

• DeepScholar-Bench is a live benchmark for AI-driven generative research synthesis that evaluates long-form, citation-rich related work sections.
• It automates dataset curation from recent arXiv papers and rigorously measures synthesis, retrieval, and verifiability using holistic metrics.
• The framework highlights challenges in coherent information synthesis and citation precision, guiding future improvements in AI-supported academic research.

DeepScholar-Bench is a live benchmark and automated evaluation framework for generative research synthesis, designed to rigorously assess AI systems capable of creating long-form, citation-rich research outputs. The benchmark targets the task of synthesizing related work sections using data retrieved from up-to-date, high-quality arXiv papers, relying on a holistic suite of metrics to evaluate synthesis, retrieval, and verifiability. Its development addresses the shortcomings of short-form QA benchmarks and static, expert-curated datasets, establishing a robust protocol for measuring progress toward advanced AI-supported academic research capabilities (Patel et al., 27 Aug 2025).

1. Motivation and Problem Definition

The central goal of DeepScholar-Bench is the systematic evaluation of generative research synthesis—specifically, the production of scholarly related work sections through retrieval, synthesis, and citation of current literature. Traditional benchmarks focus either on short factual QA or static datasets that become stale and prone to contamination. DeepScholar-Bench, in contrast, ensures "live" evaluation by selecting recent queries sourced from ongoing arXiv publications, requiring systems to engage with evolving corpora rather than static knowledge stores.

The principal evaluation task is defined as follows: given a new research query (derived from an arXiv paper), the system must generate a well-organized related work section, supported by relevant citations, with all source documents and claims retrieved directly from the live web. This task encapsulates the core challenges of research synthesis—retrieval of authoritative sources, organization of content, and verifiable citation.

2. Dataset Curation and Query Pipeline

DeepScholar-Bench leverages an automated pipeline for dataset construction, continuously curating queries from recent arXiv papers to reflect timely research topics. Each query is paired with:

• The source paper’s metadata
• Background and abstract fields to guide synthesis
• A corpus of recent related works retrieved from arXiv (via controlled search APIs)

The dataset schema enforces strict documentation of retrieved documents, synthesized output, and citation structure. This live updating mechanism precludes staleness and contamination, ensuring that benchmarks reflect the current state of research synthesis challenges for both academic and AI systems.

3. Evaluation Framework and Metrics

The evaluation design is holistic, comprehensive, and multi-dimensional, rigorously quantifying system performance along the following axes:

A. Knowledge Synthesis

• Assessed by "Organization" (LLM-based pairwise preference judgments relative to human exemplars) and "Nugget Coverage" (counting the fraction of essential information units—"nuggets"—recovered in the generated output).

B. Retrieval Quality

• Relevance Rate (RR): $RR(S) = (1/(2|S|)) \sum_s Rel(s)$, where $Rel(s)$ is a graded relevance score per document.
• Reference Coverage (RC): $RC(S, E) = (1/|E|) \sum_s I[s \in E]$, quantifying the fraction of human-curated references present in the retrieved set.
• Document Importance: Median citation count comparisons between references in the system output and in human-written exemplars, using $median(\cdot)$ ratios capped at 1.

C. Verifiability

• Citation Precision: Measures how many citations in the generated text actually support their respective claims, as judged by LLM-based evaluators.
• Claim Coverage: Fraction of claims in the text that are fully covered by citations, evaluated using a "sliding window" (e.g., citations within ±1 sentence of claim location); ablation studies analyze the impact of varying window size.

All key metrics are computed automatically (with LLM judge modules or API-based evidence scoring), allowing efficient, scalable benchmarking. No system has yet achieved more than 19% combined score on all metrics, reflecting the strong discriminative power and inherent challenge of DeepScholar-Bench.

4. Baseline System: DeepScholar-base Pipeline

DeepScholar-base acts as an open, reference implementation for the benchmark task. It comprises:

• Retrieval: LLM-guided semantic query generation from the abstract/background, dispatched via APIs to collect candidate arXiv papers.
• Filtering: Semantic operators (Sem-Filter and Sem-TopK, implemented via the LOTUS API) to select and rerank relevant candidates.
• Aggregation/Generation: Final related work synthesis using Sem-Agg, which semantically aggregates candidate documents in a coherent multi-paragraph format, with citation anchors.

DeepScholar-base's performance establishes a competitive or superior baseline to several commercial and open-source search agents, particularly in citation precision and claim coverage (verifiability), where it realized improvements up to 6.3× versus prior methods. However, like other systems, it falls short of saturating the benchmark.

5. Experimental Findings and Insights

Systematic evaluation demonstrates that even the strongest models—including OpenAI's DeepResearch, current search AIs (GPT-4.1, Claude, Gemini), and DeepScholar-base—fail to reach satisfactory coverage and organization across the key metrics. Nugget retrieval, synthesis organization, and cross-reference coverage remain below 45% for all systems. This indicates fundamental bottlenecks in moving from factual retrieval to coherent synthesis and reliable citation.

Additionally, while commercial systems generate relatively coherent output, they often lack in recovering crucial facts and in aligning claims to supporting citations, underscoring deficiencies in retrieval and grounding. The live nature of queries ensures results are not skewed by overfitting to finite, static corpora.

6. Technical Details and Formulae

Several important evaluation metrics are defined mathematically in the framework:

• $RR(S) = (1/(2|S|)) \sum_s Rel(s)$
• $RC(S, E) = (1/|E|) \sum_s I[s \in E]$
• Document Importance Ratio: $$min(1, \frac{median(C_{\text{sys}})}{median(C_{\text{ref}})})$$

Citation alignment and claim coverage are operationalized via automated window-based matching procedures, with ablation studies provided on window size.

7. Future Directions and Community Resources

The persistent gap between automated and human-written synthesis revealed by DeepScholar-Bench points to pressing research needs:

• Improved document retrieval algorithms (e.g., advances in ranking, semantic filtering, diversity sampling)
• Enhanced synthesis modules for richer nugget extraction and logical organization
• More precise citation mechanisms and claim evidence alignment
• Ongoing live benchmarks to track progress against evolving research literature

Code, data, pipelines, and extensive documentation are openly available at https://github.com/guestrin-lab/deepscholar-bench, supporting reproducibility, extensibility, and integration into custom evaluation workflows.

DeepScholar-Bench thus sets a new standard for rigorous, live benchmarking of generative research synthesis, offering clear methodological advances and actionable insights for researchers developing the next generation of AI-supported academic synthesis systems.