Agentic AutoSurvey Framework

• Agentic AutoSurvey is a multi-agent framework for automated literature surveys that orchestrates specialized agents to deliver state-of-the-art synthesis and evaluative rigor.
• It employs semantic embedding and optimized K-Means clustering to organize research topics, ensuring quality citation coverage and integration-first narrative generation.
• Demonstrated to outperform baseline systems, the framework offers a scalable, modular design with empirical improvements in core, writing, and depth metrics.

Agentic AutoSurvey is a multi-agent framework for automated literature survey generation designed to address limitations of prior approaches in coverage, synthesis quality, and evaluative rigor. By orchestrating a chain of specialized agents—each responsible for distinct research tasks—it demonstrates new state-of-the-art performance in the automated synthesis and critical analysis of research topics, particularly within rapidly evolving domains such as LLM research (Liu et al., 23 Sep 2025).

1. Core Architecture: Modular Multi-Agent Orchestration

Agentic AutoSurvey's architecture is modular, with four specialized agents collaborating to form an integrated pipeline:

1. Paper Search Specialist Agent
   • Performs multi-query generation (20–30 queries/topic), synonym/semantic expansion, and queries multiple sources (Semantic Scholar, arXiv).
   • Filters for quality (recency, citation count, venue) and deduplicates results by title overlap (90% threshold).
2. Topic Mining & Clustering Agent
   • Encodes title+abstract pairwise into a vector space using sentence-transformers (e.g., all-MiniLM-L6-v2).
   • Applies K-Means clustering (K selected via silhouette scores and cluster indices) for semantic grouping.
   • Assigns diagnostic metrics: silhouette score, Calinski-Harabasz, Davies-Bouldin, per-cluster confidence, and inter-cluster relationship strengths (using cosine similarity of centroids).
   • Names clusters using TF-IDF scoring on local word frequencies.
3. Academic Survey Writer Agent
   • Consumes structured clusters, mandates citation inclusion (≥50%; targets ≥80%), and generates synthetic, integration-first narrative reviews (not mere listing).
   • Organizes output by clusters; arranges into 8,000–12,000-word academic prose.
4. Quality Evaluator Agent
   • Performs structured diagnostic scoring across 12 dimensions, grouped into "Core" (60%; e.g., citation coverage, synthesis quality), "Writing" (20%; e.g., readability), and "Content Depth" (20%; e.g., critical analysis, future directions).
   • Emulates context-aware human peer review rather than simple rule-based LLM grading.

This pipeline, typically completed in 15–20 minutes for up to 443 papers per topic, is designed for resilience (dedicated caching, error handling, scalable orchestration) and robust to topic or domain expansion.

2. Algorithmic Foundations and Technical Implementation

Agentic AutoSurvey leverages contemporary NLP infrastructure and clustering methods:

• Semantic Embedding: Each paper’s title and abstract concatenated before embedding.

$e_i = \text{Encode}(t_i \oplus a_i)$

where $e_i$ is the embedding for paper $i$.

• Cluster Selection: K-Means clustering with $K$ picked to optimize silhouette score:

$K^* = \arg\max_{K \in [5,15]} S(K)$

with

$$S(K) = \frac{1}{n} \sum_{i=1}^{n} \frac{b_i - a_i}{\max(a_i, b_i)}$$

($a_i$, $b_i$ are sample proximity scores).

• Cluster Validity: Confidence and relationship metrics:

$$\mathrm{confidence}(i, C_j) = 1-\frac{d(i, \mathrm{centroid}(C_j))}{\max_k d(i, \mathrm{centroid}(C_k))}$$

$$\mathrm{strength}(C_j, C_k) = \cos(\mathrm{centroid}(C_j), \mathrm{centroid}(C_k))$$

• Citation Coverage and Depth: Writer agent incorporates hard minimums and targets for explicit citation count and representation across all clusters.
• Evaluation: 12-part agentic rubric, 0–10 per dimension with category weights, ensures nuanced and holistic quality assessment.

3. Empirical Performance and Evaluation

The system was benchmarked on six LLM research topics from COLM 2024, with literature sets ranging from 75 to 443 papers (total 847):

• Quality Scores (mean over topics):

| | Core | Writing | Depth | Overall | |---------------|------|---------|-------|---------| | Agentic AutoSurvey | 8.23 | 8.31 | 7.92 | 8.18 | | Baseline (AutoSurvey) | 4.77 | | | |

Category improvements: Core +99%; Writing +68%; Depth +33%; Overall +71% vs. baseline [AutoSurvey, Wang et al. 2024].

• Citation Coverage:
  • For 75–100 paper sets, 80%+ citation coverage is typical.
  • Very large corpora (e.g., RLHF, 443 papers): coverage drops (6–7%), indicating scalability bottlenecks in writing/extensive referencing.
• Clustering Quality: Example cluster metrics (e.g., for LLM Agents topic, $K=9$): silhouette score 0.055, Calinski-Harabasz 4.1, Davies-Bouldin 2.591—consistent with meaningful but complex topic structure in modern research areas.

4. Key Innovations Over Prior Automated Survey Systems

Agentic AutoSurvey outperforms prior automated systems (notably AutoSurvey):

• Orchestration Depth: Each agent is specialized and modular; prior single-agent approaches lack robust decomposition and error recovery.
• Breadth and Integration: Semantic clustering with optimal $K$ enables comprehensive and organized coverage of a topic's landscape, against baseline tools with limited or heuristic partitioning.
• Synthesis Quality: The synthesis-first writer agent integrates across clusters, focuses on comparative analysis and explicit research trends/gaps, and produces structurally academic outputs.
• Evaluation Rigor: The 12D Quality Evaluator agent offers multi-faceted, weighted assessment aligned more closely with human academic appraisal than simple scoring or LLM self-judgment.
• Scalability: Capable of handling hundreds of papers in real time, supporting dynamic and repeatable topic exploration at scale.

5. Limitations and Future Directions

• Scalability Bottlenecks: As collection sizes grow (e.g., >400 papers), citation coverage drops; digest/appendix strategies are needed for full representation without narrative overload.
• Domain Adaptation: System and evaluation framework are optimized for LLM/AI research; further generalization requires domain-informed retraining and prompt engineering.
• Feature Gaps: Human-in-the-loop interactive refinement is not yet present; singular writing pass in the current system constrains adaptability.
• Evaluation Subjectivity: While improved, certain qualitative dimensions (depth, insight) still challenge full automation and maintain a degree of subjectivity.
• Processing Time: 15–20 minutes is rapid for batch academic tasks but may not enable real-time interactive survey refinement.

Planned extensions include hierarchical writing, interactive user guidance during synthesis, full multimodal summarization (figures, tables, equations), and domain-adaptive expansion to fields beyond computational sciences.

6. Implications for the Future of Automated Scholarly Synthesis

Agentic AutoSurvey operationalizes the agentic services paradigm in a canonical research workflow. It demonstrates that specialized agent orchestration, robust semantic clustering, coverage-aware synthesis, and agentic evaluation can produce objectivity, repeatability, and integration quality previously unachievable in automated survey generation. A plausible implication is the emergence of on-demand, dynamically updated surveys for rapidly changing fields, provided the evaluation, narrative, and human oversight aspects continue to mature in parallel.

---

References:

• (Liu et al., 23 Sep 2025) Agentic AutoSurvey: Let LLMs Survey LLMs (main reference; all claims, metrics, and architectural details are from the cited paper).
• [AutoSurvey, Wang et al. 2024] for baseline comparison.

---