PaperOrchestra: Automated Manuscript Framework

• PaperOrchestra is a multi-agent framework that transforms unconstrained pre-writing inputs into high-quality LaTeX manuscripts with specialized agent roles.
• The system employs modular components—including outline, plotting, literature review, section writing, and content refinement agents—to ensure content consistency and reduce errors.
• Evaluated with the PaperWritingBench benchmark, PaperOrchestra demonstrates significant improvements in literature review quality, citation accuracy, and overall manuscript production.

PaperOrchestra is a multi-agent framework for automated research paper writing, designed to flexibly convert unconstrained pre-writing materials into high-quality, submission-ready LaTeX manuscripts. Distinct from prior autonomous writing systems, which are often narrowly tied to specific experimental pipelines and produce limited literature reviews, PaperOrchestra employs a modular architecture with specialized agent roles, comprehensive literature synthesis capabilities, and integrated generation of scientific visuals. Its performance is evaluated using PaperWritingBench, a benchmark constructed from reverse-engineered materials of 200 top-tier AI conference papers, and is demonstrated to yield substantial quality improvements in both literature review and overall manuscript production compared to prior systems (Song et al., 6 Apr 2026).

1. System Architecture and Agent Specialization

PaperOrchestra formalizes manuscript creation as a multi-agent orchestration problem, with five dedicated agents communicating via structured JSON payloads and shared files. Each agent uses the Gemini-3.1-Pro LLM backbone and implements an "anti-leakage" prompt to prevent cross-contamination of agent outputs. The core agents are:

• Outline Agent: Synthesizes idea summaries, experimental logs, templates, and guidelines into a structured JSON outline specifying plans for plotting, literature search, and section breakdowns.
• Plotting Agent: Consumes the outline to generate both empirical plots (from raw numbers) and conceptual diagrams using PaperBanana (a multimodal text-driven image generator with vision-LLM [VLM] critique). Outputs consist of finalized images and descriptive captions.
• Literature Review Agent: Executes a two-stage, LLM- and web-grounded retrieval-augmented generation pipeline, discovering and verifying citations via Semantic Scholar, then drafting Introduction and Related Work sections using only confirmed references.
• Section Writing Agent: Integrates all components to draft the remaining manuscript sections (Methodology, Experiments, Conclusion), enforces booktabs-compliant tables and figure inclusions, and ensures citation consistency.
• Content Refinement Agent: Iteratively revises the draft in a closed loop, guided by automated reviewer feedback, until consensus is reached or a maximum of three cycles is completed.

This division of labor supports both parallelism (e.g., plotting and citation gathering) and content consistency, producing coherent output without redundancy or hallucinated results.

2. End-to-End Pipeline and Data Flow

The PaperOrchestra manuscript generation process consumes a quintet of artifact types:

• $\mathcal{I}$ (Idea summary)
• $\mathcal{E}$ (Experimental log)
• $\mathcal{T}$ (LaTeX template)
• $\mathcal{G}$ (Submission guidelines)
• $\mathcal{F}$ (Raw figures)

The high-level function is:

$$(\mathcal{I}, \mathcal{E}, \mathcal{T}, \mathcal{G}, \mathcal{F}) \to (P_{tex}, P_{pdf})$$

The algorithmic pipeline is formally described as:

\begin{algorithmic}[1] \Function{W}{%%%%6%%%%} \State outline.json = OutlineAgent(%%%%7%%%%) \State \textbf{in parallel:} \State \quad figs = PlottingAgent(outline.json.plotting_plan, %%%%8%%%%) \State \quad bib = LitReviewAgent(outline.json.intro\_related\_work\_plan) \State draft.tex = SectionWritingAgent(template.tex, outline.json.section_plan, figs, bib, %%%%9%%%%) \State final.tex = ContentRefinementAgent(draft.tex, draft.pdf, reviewer\_feedback) \State final.pdf = \texttt{pdflatex}(final.tex) \State \Return (final.tex, final.pdf) \EndFunction \end{algorithmic}

All manuscript sections, figures, tables, and references are integrated into the output LaTeX file, which is then compiled to produce the final PDF. Agent communication is strictly regulated, with each agent operating on a defined subset of the inputs and outputs, minimizing error propagation and information leakage.

3. Literature Synthesis Methodology

PaperOrchestra advances literature review quality via a structured, two-stage RAG process:

• Parallel Candidate Discovery: Launches ten concurrent LLM (Gemini-3-Flash) web-grounded searches per plan query. Each returns a batch of 5–10 candidate titles.
• Sequential Citation Verification: Each candidate is authenticated using the Semantic Scholar API, filtered by a minimum Levenshtein title match of 0.7 and a publication date cutoff. Deduplication is performed using the Semantic Scholar paper ID. Only validated citations are retained for inclusion.
• Section Drafting: Introduction and Related Work are drafted using only the verified citation pool. At least 90% of citations in these sections are guaranteed to be present in the authenticated bibliography.

No further ranking or weighting is performed beyond temporal cutoff and duplicate removal. This approach enables both breadth and topical depth, limited only by coverage of search queries in the supplied outline.

4. Visual and Tabular Content Generation

Scientific visuals in PaperOrchestra are created through the Plotting Agent:

• Data Plots: Extracted from experimental logs and rendered via a Python microservice (e.g., Matplotlib), ensuring alignment with underlying results.
• Conceptual Diagrams: Generated via PaperBanana, which first creates a draft SVG/raster image through diffusion, then applies a VLM-based critique to assess semantic and visual fidelity to the specified objective. Failed attempts are automatically corrected and regenerated.
• Captions: Generated contextually by an LLM prompt, tailored to each figure's purpose.

Tabular results employ the booktabs LaTeX formatting standard, extracting metrics directly from logs to prevent fabrication or misrepresentation.

5. Benchmarking and Evaluation (PaperWritingBench)

Evaluation is standardized using PaperWritingBench, built from 200 anonymized, high-quality papers (split equally between CVPR 2025 and ICLR 2025):

Construction Pipeline:

• Manuscripts reverse-engineered to extract markdown (MinerU), raw figures and captions (PDFFigures2.0), idea and experimental log snippets (Gemini-3.1-Pro prompting).
• Explicit author identities, manual citations, and figure references are stripped.

Scoring Protocols:

• Side-by-side win rates: Direct human comparison of PaperOrchestra outputs against baselines, separately for literature review and full manuscript quality.
• Citation F1: Computed between generated (Gen) and ground-truth (GT) reference sets for both all cited papers (P0) and high-priority, key citations (P1).
• Holistic reviewer scores (AgentReview, ScholarPeer): End-to-end simulation of peer review and acceptance rates.

Sample scoring equations:

$$\mathrm{Precision} = \frac{|\mathrm{Gen}\cap \mathrm{GT}|}{|\mathrm{Gen}|},\quad \mathrm{Recall} = \frac{|\mathrm{Gen}\cap \mathrm{GT}|}{|\mathrm{GT}|},\quad F_1 = 2\cdot\frac{\mathrm{P}\,\mathrm{R}}{\mathrm{P}+\mathrm{R}}$$

6. Results Compared to Previous Systems

PaperOrchestra achieves state-of-the-art performance across multiple axes. Quantitative results include:

• Literature Review Win Rate: Outperforms existing autonomous baselines by an absolute margin of 50%–68%.
• Overall Paper Quality: Raises side-by-side win rates by 14%–38%.
• Holistic Acceptance Rates (simulated):

Pipeline	CVPR 2025	ICLR 2025
Human (GT)	86%	94%
AI Scientist-v2	70%	64%
SingleAgent	71%	72%
PaperOrchestra(PlotOff)	84%	81%

• Citation F1 (CVPR): SingleAgent: 28.0%, AI Scientist-v2: 17.2%, PaperOrchestra: 29.7% (notably, P1 recall climbs from ~3% to ~16%).
• Automated literature review quality (Gemini-3.1-Pro): SingleAgent: 43%, AI Scientist-v2: 45%, PaperOrchestra: 78%.

All scoring details are extracted from controlled, blinded trials; no author-invented statistics or fabrication are introduced.

7. Strengths, Limitations, and Future Directions

Strengths

• Modular, decoupled agent paradigm supports broad input flexibility and reliably grounded outputs.
• Literature synthesis employs multi-stage RAG with strong verification, minimizing citation hallucination.
• Conceptual diagram generation leverages VLM-based critique, enabling robust non-data-driven visualization.
• Iterative draft refinement uses agent-based reviewer simulation to eliminate fabrication and optimize clarity.

Limitations

• External tool dependencies (e.g., PaperBanana, MinerU, PDFFigures) pose risks of image or extraction failures, which may propagate into the final manuscript.
• Automated diagram generation can drift in quality if VLM critique fails.
• Content refinement is exclusively automated; absence of interactive, human-in-the-loop steering restricts customizability.
• Pretraining data leakage, while minimized by anti-leakage prompting, is not provably eliminated.

Proposed Extensions

• Tighter factual verification of diagrams using specialized vision-LLMs.
• Real-time, interactive authoring interfaces allowing live agent feedback or re-planning.
• Enhanced support for richer raw artifacts, including lab notebooks and simulation logs.
• Mechanisms for live, multi-paper joint synthesis (e.g., cross-article surveys) via orchestrated agent collaboration.

A plausible implication is that by enabling plug-and-play orchestration of agent capabilities, future versions of PaperOrchestra could serve as a general-purpose backbone for open-ended, grounded scientific manuscript automation across methodological domains (Song et al., 6 Apr 2026).