SurveyBench: Benchmark for Automated Surveys

• SurveyBench is a benchmark framework that rigorously evaluates automated survey generation by emphasizing human reference surveys, structured outlines, and content fidelity.
• It leverages curated human-written surveys and large research corpora to construct detailed reference pools and evaluate generated surveys on literature coverage and organizational quality.
• The framework employs dual-mode evaluation—combining traditional content-based metrics with quiz-driven answerability tests—to identify strengths and persistent shortcomings in LLM-generated surveys.

SurveyBench is a benchmark framework for evaluating automatic survey generation, especially LLM- and agent-based systems that produce academic survey papers from large research corpora. In the literature, the name appears in two distinct 2025 formulations. In SurveyForge, SurveyBench is introduced as a multi-dimensional benchmark built around references, outlines, and content quality, with human-written survey anchors and win-rate comparison against experts (Yan et al., 6 Mar 2025). In “SurveyBench: How Well Can LLM(-Agents) Write Academic Surveys?”, the name denotes a fine-grained, quiz-driven benchmark designed to assess whether generated surveys satisfy readers’ informational needs through content-based scoring, answerability tests, and non-textual richness analysis (Sun et al., 3 Oct 2025).

1. Terminological scope and development

The earlier SurveyBench formulation arose from the claim that prior evaluations of LLM-generated surveys relied too heavily on holistic, model-internal judgments and did not expose failures in logical organization, literature coverage, or content quality. To address this, the SurveyForge paper paired automated survey generation with a benchmark and metric suite called SAM, spanning reference quality, outline quality, and content quality (Yan et al., 6 Mar 2025). The later SurveyBench paper retained the same basic target—rigorous evaluation of automated academic surveys—but recast the problem in reader-aligned terms, emphasizing whether a survey can answer general and topic-specific questions in addition to exhibiting strong structure and synthesis (Sun et al., 3 Oct 2025).

The two formulations are closely related in purpose but differ in emphasis.

Formulation	Corpus design	Evaluation emphasis
SurveyBench in SurveyForge	Approximately 100 human-written surveys across 10 CS topics	SAM-R, SAM-O, SAM-C, plus win-rate comparison
Quiz-driven SurveyBench	Topics sourced from 11,343 recent arXiv papers and 4,947 high-quality surveys; 20 vetted topics	Outline quality, content quality, non-textual richness, content-based and quiz-based evaluation

This dual usage has made “SurveyBench” both a specific benchmark name and a broader reference point for survey-generation evaluation. Later work explicitly situates itself against this benchmark lineage. SurveyEval presents a three-dimensional benchmark across 7 disciplines and 38 topics, SurGE offers a standardized computer-science benchmark with 205 expert-written ground-truth surveys, SGSimEval adds similarity-enhanced evaluation over 80 highly cited arXiv surveys, and SurveyLens generalizes the problem to 1,000 surveys across 10 disciplines (Zhao et al., 2 Dec 2025, Su et al., 21 Aug 2025, Guo et al., 15 Aug 2025, Guo et al., 11 Feb 2026). This suggests that SurveyBench helped define automatic survey generation as a distinct evaluation problem rather than a mere variant of long-form text generation.

2. Corpus construction and curation logic

In the SurveyForge formulation, SurveyBench is organized around approximately 100 human-written survey papers curated across 10 computer science topics. For each topic $t_i$, one “highest-quality” survey $S_i^*$ is selected as the evaluation reference, and the references cited by the curated human-written surveys are extracted to form a topic-specific reference database $\mathcal{R}_i$ (Yan et al., 6 Mar 2025). The ten topics span multimodal LLMs, evaluation of LLMs, 3D object detection for autonomous driving, vision transformers, hallucination in LLMs, generative diffusion models, 3D Gaussian splatting, LLM-based multi-agent systems, graph neural networks, and retrieval-augmented generation. The curated surveys are predominantly published within the last two years, and curation was performed by a panel of 20 doctoral-level researchers with topic expertise.

The data structure is explicitly componentized. Each topic includes a set of curated human-written surveys, a per-topic reference set $\mathcal{R}_i$, and one high-quality survey $S_i^*$ used to anchor content evaluation. No train/dev/test split is defined because the benchmark is used for evaluation rather than supervised learning. The SurveyForge paper also states that the dataset is released at Hugging Face and that the project code is available on GitHub, although licensing details are not specified (Yan et al., 6 Mar 2025).

The later SurveyBench paper adopts a different curation pipeline. It starts from 127 topic candidates drawn from authoritative CS conference CfPs, removes duplicates, unifies terminology, and then clusters recent arXiv papers from the past three months using embeddings from text-embedding-3-small and t-SNE-based visualization (Sun et al., 3 Oct 2025). In parallel, 4,947 survey papers are sampled from arXiv using the keywords “survey” or “review,” and “high-quality” filtering uses citation counts together with coverage depth. The final benchmark contains 20 well-vetted topics. The paper also reports a larger topic source base of 11,343 recent arXiv papers and positions those topics as representative of current research demand.

A notable difference between the two formulations is the role of human references. In the SurveyForge version, human-written surveys define reference pools and content anchors for expert-oriented evaluation. In the later SurveyBench, human-written surveys are still central, but they function more directly as reader-calibrated gold standards against which generated surveys are scored and from which quiz material is derived (Yan et al., 6 Mar 2025, Sun et al., 3 Oct 2025).

3. Evaluation architecture and metrics

The SurveyForge benchmark centers on SAM, or Survey Assessment Metrics. SAM-R evaluates reference quality by measuring overlap between the generated survey’s references and the expert-curated reference set. For topic $t_i$ and generated survey $\hat{S}_i$ with references $R_{\hat{S}_i}$, the paper defines

$$\mathrm{SAM_R}(\hat{S}_i)=\frac{|R_{\hat{S}_i}\cap \mathcal{R}_i|}{|R_{\hat{S}_i}|}.$$

Higher values indicate better coverage of key literature (Yan et al., 6 Mar 2025).

SAM-O assigns a single outline-quality score from 0 to 100. It is judged by LLMs using criteria decomposed from peer-review guidance, specifically topic uniqueness, structural balance, hierarchical clarity, and logical organization. SAM-C evaluates content quality along structure, relevance, and coverage, grounding assessment in the selected best survey $S_i^*$. In addition to these scalar metrics, the benchmark includes score-based win rate, comparative win rate, and human expert pairwise evaluation, with double-blind assessment and reported Cohen’s $S_i^*$0 for agreement between LLM and human evaluators (Yan et al., 6 Mar 2025).

The later SurveyBench retains outline and content assessment but introduces a broader metric hierarchy. Outline quality is evaluated on a 1–5 scale through coverage, relevance, and structure. Content quality is also scored on a 1–5 scale across coverage, depth, focus, coherence, and fluency. A distinct non-textual richness metric quantifies the density of figures, tables, and diagrams relative to chapter length:

$S_i^*$1

where $S_i^*$2 is the number of non-text elements, $S_i^*$3 is the accumulated length of all chapters, and $S_i^*$4 is a tunable hyperparameter (Sun et al., 3 Oct 2025).

The most distinctive addition is the dual-mode protocol. The first mode is content-based evaluation, with or without a human reference survey. In the human-reference setting, the judge sees the topic, a high-quality human survey, the generated survey, and an evaluation dimension; human-written surveys are assigned the maximum score of 5 in outline and content quality. The second mode is quiz-based answerability testing. General quizzes cover three difficulty levels and seven perspectives: Concept Definition, Knowledge Classification, Historical Understanding, Algorithmic Principles, Practical Guidance, Performance Insights, and Future Work. Topic-specific quizzes are built through retrieval-augmented generation from high-quality survey passages and are explicitly required to be answerable solely from the provided text (Sun et al., 3 Oct 2025).

The answerability protocol is intentionally strict. Retrieval first selects relevant sections from the survey outline, then segments text and finds relevant paragraphs with vector similarity, with optional LLM filtering for residual irrelevance. Answer generation must rely only on retrieved passages, must cite supporting passages, and must return exactly “No relevant content found in the survey” if the survey lacks sufficient evidence. For topic-specific quizzes, answers are automatically scored zero when evidence is judged insufficient (Sun et al., 3 Oct 2025). This design shifts evaluation from generic textual quality toward functional informativeness.

4. Experimental protocol and judged systems

In the SurveyForge experiments, systems were evaluated under standardized retrieval and generation conditions. A corpus of approximately 600,000 research papers and 20,000 review articles from arXiv was used to construct a vector retrieval database. For each topic, 1,500 candidate papers were retrieved for outline generation and 60 relevant papers for each chapter-writing stage. Generation was run on 10 topics with 10 independent trials per topic, yielding 100 outputs per model with averages reported (Yan et al., 6 Mar 2025).

The systems compared included AutoSurvey as the prior state-of-the-art baseline, SurveyForge as the proposed method, and an additional open-source evaluation with DeepSeek-v3. Generation used Claude-3-haiku-20240307, GPT-4o-mini-2024-07-18, and DeepSeek-v3, while outline and content scoring used GPT-4o-2024-08-06 and Claude-3.5-sonnet-20241022 (Yan et al., 6 Mar 2025). Human evaluation involved 20 PhD experts in a double-blind setup, with training and a guideline manual aligned to the LLM evaluation criteria.

The later SurveyBench evaluates a different system set: AutoSurvey, SurveyForge, LLM×MapReduce-V2, and OpenAI-DeepResearch. Its protocol explicitly attempts leakage avoidance by instructing systems not to use existing surveys on the topic as references. The prompt includes the directive “Do not use existing relevant surveys as reference,” and subsequent instructions emphasize breadth, rigor, inclusion of diagrams or tables, historical trajectory, open challenges, and construction from “fundamental sources and reasoning” (Sun et al., 3 Oct 2025).

This later benchmark also distinguishes between document-level and chapter-level content-based evaluation in the no-reference setting, then adds human-reference evaluation and quiz-based validation. A reported methodological finding is that scores are inflated when human-written references are absent, which the authors treat as evidence that human-reference calibration is necessary for faithful assessment (Sun et al., 3 Oct 2025). A plausible implication is that pure LLM-as-judge evaluation without external anchors tends to reward stylistic adequacy more than survey-specific rigor.

5. Quantitative findings and diagnosed failure modes

The SurveyForge benchmark reports that SurveyForge improves substantially over AutoSurvey on reference coverage, outline quality, and content quality. Under GPT-4o mini generation, AutoSurvey achieved Input Coverage $S_i^*$5 and Reference Coverage $S_i^*$6, while SurveyForge reached $S_i^*$7 and $S_i^*$8 respectively. Human-written surveys attained Reference Coverage $S_i^*$9. On outline quality, human-written surveys scored $\mathcal{R}_i$0, AutoSurvey scored $\mathcal{R}_i$1, and SurveyForge scored $\mathcal{R}_i$2. On content quality, SurveyForge with GPT-4o mini achieved Structure $\mathcal{R}_i$3, Relevance $\mathcal{R}_i$4, Coverage $\mathcal{R}_i$5, and Avg $\mathcal{R}_i$6, exceeding AutoSurvey’s Avg of $\mathcal{R}_i$7 (Yan et al., 6 Mar 2025).

The same benchmark reports strong alignment between automated and human judgment. LLM-versus-human Cohen’s $\mathcal{R}_i$8 is $\mathcal{R}_i$9 for outline and $\mathcal{R}_i$0 for content, while human cross-validation yields $\mathcal{R}_i$1 and $\mathcal{R}_i$2. In pairwise win-rate comparison, SurveyForge achieved an outline human-evaluation win rate of $\mathcal{R}_i$3 against AutoSurvey’s $\mathcal{R}_i$4, and a content human-evaluation win rate of $\mathcal{R}_i$5 against AutoSurvey’s $\mathcal{R}_i$6 (Yan et al., 6 Mar 2025). The paper also reports strong open-source performance from DeepSeek-v3, which under SurveyForge reached Input Coverage $\mathcal{R}_i$7, Reference Coverage $\mathcal{R}_i$8, Outline Quality $\mathcal{R}_i$9, and Content Avg $S_i^*$0.

The later SurveyBench yields a different performance picture because it scores surveys against human references on a 1–5 scale and probes answerability directly. In content-based evaluation, LLM×MapReduce-V2 attains the best outline average at $S_i^*$1, compared with SurveyForge’s $S_i^*$2, AutoSurvey’s $S_i^*$3, and OpenAI-DeepResearch’s $S_i^*$4. On content quality, OpenAI-DeepResearch attains the best average at $S_i^*$5, followed by LLM×MapReduce-V2 at $S_i^*$6, with SurveyForge and AutoSurvey both at $S_i^*$7. Human surveys are fixed at $S_i^*$8, and the abstract summarizes the gap as “on average 21% lower than human in content-based evaluation” (Sun et al., 3 Oct 2025).

Quiz-based results reveal a sharper weakness. On general quizzes, OpenAI-DeepResearch performs strongly on historical understanding, algorithmic principles, practical guidance, and future-work questions, with win rates of $S_i^*$9, $t_i$0, $t_i$1, and $t_i$2 respectively. LLM×MapReduce-V2 is comparatively strong on concept definition, profiling, and prediction, while SurveyForge peaks on profiling at $t_i$3 (Sun et al., 3 Oct 2025). Yet topic-specific quiz performance remains very low for all systems: AutoSurvey scores $t_i$4, SurveyForge $t_i$5, LLM×MapReduce-V2 $t_i$6, and OpenAI-DeepResearch $t_i$7, with the human survey set to $t_i$8.

The failure analysis in the later SurveyBench identifies four recurring deficits: insufficient detail, lack of associative reasoning, deficient synthesis and abstraction, and persistent weakness in fine-grained answerability despite fluent prose (Sun et al., 3 Oct 2025). The papers jointly suggest a recurring pattern: outline generation has improved markedly, general content fluency is often high, but citation grounding, cross-paper synthesis, and precise reader-oriented informational utility remain the main unresolved bottlenecks (Yan et al., 6 Mar 2025, Sun et al., 3 Oct 2025).

6. Position in the benchmark landscape, limitations, and significance

SurveyBench occupies a foundational position in ASG evaluation because it operationalizes survey quality as a composite of structure, literature grounding, synthesis quality, and user-facing informativeness. Related benchmarks extend different parts of this design space. SurveyEval formalizes overall content quality, outline coherence, and reference accuracy across seven disciplines and augments LLM-as-a-Judge with human-written anchors (Zhao et al., 2 Dec 2025). SurGE adds a million-scale arXiv retrieval pool and hierarchical citation relevance at document, section, and sentence levels (Su et al., 21 Aug 2025). SGSimEval combines outline, content, and reference evaluation with embedding-based similarity weighting to human surveys (Guo et al., 15 Aug 2025). SurveyLens generalizes evaluation to ten disciplines and introduces discipline-aware rubrics together with canonical alignment metrics such as RAMS and TAMS (Guo et al., 11 Feb 2026).

Several limitations are explicit. The original SurveyBench is confined to ten trending topics in computer science, with curated surveys mainly from the past two years, so older foundational literature or non-CS domains may be underrepresented (Yan et al., 6 Mar 2025). Its paper reports win rates and agreement statistics but does not provide confidence intervals or formal statistical test details beyond $t_i$9. The later SurveyBench also remains heavily AI- and CS-centric through its 20 final topics, and its excerpt does not report human inter-annotator agreement, significance testing, or release details for code and data (Sun et al., 3 Oct 2025). In addition, the richness metric measures the count of non-text elements normalized by length, not their correctness or explanatory value.

A further complication is terminological. Because “SurveyBench” names both the SurveyForge benchmark and the later quiz-driven benchmark, the term no longer refers to a single frozen protocol. Instead, it denotes a research line centered on benchmarking automatic survey generation under more demanding criteria than ROUGE-style long-text evaluation. This suggests that SurveyBench’s enduring contribution lies less in any single metric and more in establishing that survey generation should be evaluated against human-written surveys, explicit structural and bibliographic criteria, and reader-centered answerability tests rather than generic coherence alone.

Within that broader trajectory, SurveyBench marks a methodological shift. It treats academic survey writing not simply as long-form generation, but as literature-grounded synthesis with evaluable obligations: comprehensive topic coverage, hierarchical organization, citation fidelity, critical comparison, and the capacity to support downstream question answering. Subsequent benchmark proposals largely preserve that premise, even when they expand it to new disciplines, new corpora, or new scoring models (Zhao et al., 2 Dec 2025, Su et al., 21 Aug 2025, Guo et al., 15 Aug 2025, Guo et al., 11 Feb 2026).