SurveyForge (DeepSeek-v3): Automated Survey Generation

• SurveyForge (DeepSeek-v3) is an automated framework that generates high-quality scientific surveys by learning from human-written outlines and leveraging memory-driven retrieval.
• It employs a two-stage hierarchical outline generation and a Scholar Navigation Agent to integrate bibliographic references with temporal-aware reranking.
• Empirical evaluations show that SurveyForge outperforms previous systems in outline coherence, reference accuracy, and overall survey quality.

SurveyForge (DeepSeek-v3) is an automated framework for generating high-quality scientific survey papers, integrating architectural, algorithmic, and evaluation innovations rooted in the DeepSeek-v3 Mixture-of-Experts (MoE) LLM. Its design addresses long-standing deficits in outline structure and reference quality associated with LLM-generated surveys by explicitly learning from large corpora of human-written survey outlines, leveraging memory-driven scholarly retrieval, and employing multidimensional evaluation criteria. SurveyForge establishes new benchmarks for coherence, relevance, and bibliographic accuracy in automated survey writing (2503.04629).

1. System Architecture and Outline Generation

SurveyForge adopts a two-stage generation approach centered around hierarchical, semantically-informed outline construction. It employs a top-down heuristic learning strategy utilizing two domain-specific databases: a research paper database containing approximately 600,000 titles and abstracts, and a survey outline database with section/subsection hierarchies from approximately 20,000 human-written surveys. The system fuses these resources, retrieving thematically relevant papers and extracting outline skeletons representative of expert structuring.

Outline generation proceeds in two recursive levels:

• At the first level, top-level outline nodes ($\mathcal{O}_i$) and their semantic queries ($Q_i$) define each section's scope.
• For each $\mathcal{O}_i$, the system retrieves supporting literature, yielding second-level outlines ($\mathcal{O}_{ij}$) and refined sub-queries ($q_{ij}$) that guide subsequent retrieval and synthesis.

This process is algorithmically formalized, enabling the model to reproduce human-like hierarchical structures with improved thematic coverage and logical flow. Compared to uncontrolled LLM text generation, SurveyForge’s outline stage promotes logical coherence, depth, and structural fidelity to field conventions.

2. Memory-Driven Content Synthesis and Scholarly Retrieval

SurveyForge’s content generation is orchestrated by a Scholar Navigation Agent (SANA), responsible for leveraging retrieved literature as “memory” to guide all text synthesis:

• Memory for Sub-Query (MS): For each outline section, relevant literature $P_{R_i}$ is stored as memory $M_i$. Subsection-specific queries $q_{ij}$, formed from section descriptors and titles, are expanded using the memory: $q_{ijk} = LLM(q_{ij}, M_i)$.
• Memory for Retrieval (MR): Sub-queries are matched against the global memory set $M$, selecting the most relevant subset $L_{ijk}$ based on embedding similarity.
• Temporal-Aware Reranking Engine (TRE): Retrieved literature is further grouped by publication periods (every two years) and reranked based on temporal currency and citation counts within each period. This ensures literature quality and diversity, formalized by $k_g = (|n_g| / |L_{ijk}|)\cdot K_{O_{ij}}$, where $K_{O_{ij}}$ is a hyperparameter controlling citation set size.

LLMs then synthesize each subsection’s content in parallel using the filtered reference set, followed by a refinement stage that merges generated fragments, mitigating redundancy and ensuring consistency.

3. Multi-Dimensional Evaluation with SurveyBench

SurveyForge introduces SurveyBench, a benchmark suite comprising 100 human-written surveys across ten research topics, to evaluate AI-generated surveys using tailored Survey Assessment Metrics (SAM) along three axes:

• SAM-R (Reference Quality):

$$SAM_R(\hat{S}_i) = \frac{|R_{\hat{S}_i} \cap \mathcal{R}_i|}{|R_{\hat{S}_i}|}$$

This metric calculates reference overlap between the generated survey’s bibliography and curated topic-specific ground truth $\mathcal{R}_i$.

• SAM-O (Outline Quality): Single scalar (0–100) measuring clarity, logical organization, topic uniqueness, and structural balance, cross-validated by LLM and human raters.
• SAM-C (Content Quality): Composite of structure, relevance, and coverage, averaged as

$$SAM_C^{avg} = \frac{SAM_C^{struct} + SAM_C^{rel} + SAM_C^{cov}}{3}$$

with each component rated 0–100 based on content’s academic standards.

This multidimensional evaluation framework surpasses traditional fluency measures, enabling fine-grained comparison against expert-written surveys and previous systems such as AutoSurvey.

4. Benchmark Results and Comparative Effectiveness

Empirical evaluation shows that SurveyForge—when powered by open-source DeepSeek-v3—outperforms previous automated survey writers:

• With DeepSeek-v3 as the LLM backend, SurveyForge achieves an outline score of 87.42 and an average content quality score of 80.15, surpassing AutoSurvey and even rivaling commercial systems when normalized for API cost.
• Reference Quality and Input Coverage are practically doubled relative to earlier frameworks: for example, Reference Coverage rises from 0.23 (AutoSurvey) to 0.3960 (SurveyForge, using Claude-3-Haiku).
• Human and LLM-based win-rate comparisons show SurveyForge is preferred in approximately 70% of side-by-side evaluations for both outline and content, underscoring substantial advances in practical survey quality.

Key performance improvements arise from the joint optimization of outline heuristics, memory-based retrieval, and citation-aware reference selection.

5. Technical Innovations and Implications

SurveyForge’s framework embodies several methodological advances:

• Outline Heuristics: By explicitly modeling human academic structure, SurveyForge mitigates common LLM failures in hierarchical organization and thematic ambiguity.
• Memory-Driven Generation: The SANA agent’s integration of paper memory and reranking based on both semantic similarity and bibliometric quality marks a significant departure from naive retrieval-augmented generation.
• Automated Citation Filtering: Temporal-aware reranking ensures inclusion of high-impact and papers, directly addressing the citation drift problem in LLM-based generation.

The technical contributions, including formulaic definitions and data pipeline algorithms, render the framework extensible and adaptable to other domains requiring high-fidelity literature synthesis.

6. Applications and Future Directions

SurveyForge’s design enables diverse applications:

• Academic Research Automation: Accelerates comprehensive literature survey generation, aiding researchers in rapidly assimilating new topic areas.
• Interdisciplinary Review: Fosters domain-bridging surveys by systematically mapping overlapping literatures.
• Scientific Discovery and Policy Analysis: Supports hypothesis generation and evidence aggregation in automated discovery platforms and policy development contexts.

Future work includes adapting the framework for cross-domain literature mapping, integrating more sophisticated bibliometric indicators into the reranking module, and extending the evaluation suite to cover multilingual and multi-domain survey generation. The modular design allows substitution or augmentation of the LLM backbone, making improvements in underlying models immediately beneficial to survey quality.

7. Limitations and Prospective Enhancements

Although SurveyForge narrows the quality gap with human survey writing, notable challenges remain:

• Some residual redundancy and incoherence may persist in the merging stage, especially under heavy parallelization of section synthesis.
• The citation coverage, although superior to predecessors, is fundamentally bounded by the scope and recency of the underlying databases.
• Fine control over sub-topic specificity and divergence from prevailing survey conventions requires further research in outline semantic modeling.

Continual refinement of both the heuristic outline algorithms and retrieval reranking—potentially leveraging reinforcement learning from human preferences or adversarial evaluation strategies—represents a promising direction for progressive enhancement.

---

SurveyForge (DeepSeek-v3) therefore represents a significant advancement in automated survey generation, distinguished by its learning-based outline construction, memory-driven literature synthesis, and robust multidimensional evaluation regime. Through systematic benchmarking and empirical validation, it establishes technical foundations for scalable, high-quality scientific survey writing (2503.04629).