General Checklists in Deep Synthesis

• General Checklists are methodical tools that decompose complex synthesis tasks into atomic, verifiable items, ensuring objective evaluation.
• They leverage curated Oracle Contexts to isolate synthesis from retrieval, enabling reproducible and granular scoring of deep learning outputs.
• Agentic multi-turn workflows using these checklists improve factual coverage and structural organization, reducing hallucinations in model outputs.

DeepSynth-Eval is a comprehensive benchmarking framework developed to objectively evaluate deep learning models for information consolidation and synthesis, most notably in complex tasks such as deep survey writing and, by analogy, for learning-to-synthesize in audio domains. By shifting focus from retrieval to the post-retrieval synthesis stage, DeepSynth-Eval enables fine-grained, reproducible, and scalable assessments of model capabilities in aligning model outputs with human-curated ground truths across thousands of atomic requirements. The methodology addresses fundamental gaps in prior evaluation schemes, providing a critical infrastructure for next-generation LLM and agentic research (Zhang et al., 7 Jan 2026).

1. Motivation and Scope

The impetus for DeepSynth-Eval stems from the encountered limitations in traditional information retrieval (IR) and short answer extraction benchmarks, which are ill-equipped to measure the capacity of LLMs and autonomous agents to synthesize, organize, and distill information from vast corpora (often exceeding one million tokens and hundreds of documents). Benchmarking the “deep synthesis” capability—articulating the state of a research area, composing taxonomies, constructing comparative tables, and adhering to complex narrative or structural constraints—requires moving beyond holistic human evaluation and LLM-as-a-judge scoring, which are costly, subjective, and susceptible to bias or positional variance (Zhang et al., 7 Jan 2026).

DeepSynth-Eval is designed to:

• Isolate the synthesis competency from the confounders of retrieval or irrelevant context (“Oracle Context” paradigm).
• Decompose open-ended generation into verifiable, atomic checklist items for reproducible scoring.
• Support evaluation of both factual coverage and structural organization at scale (Zhang et al., 7 Jan 2026).

2. Dataset Construction and Oracle Contexts

For each evaluation task, DeepSynth-Eval utilizes high-quality existing survey papers as references. The process involves:

• Reverse-engineering the core research prompt from the title and abstract of a target survey, typically by prompting an LLM to formulate a generalized research question.
• Constructing the "Oracle Context" by aggregating the full text (usually as stable-ID-prefixed summaries) of all papers cited in the reference survey's bibliography, eliminating retrieval noise (e.g., web detritus or broken links).
• Ensuring that the model under evaluation has access only to the curated Oracle Context, thereby decoupling synthesis from retrieval (Zhang et al., 7 Jan 2026).

Each Oracle Context can encompass >1 million words and hundreds of sources, providing a scalable and challenging testbed for synthesis models.

3. Formal Evaluation Protocols

Evaluation in DeepSynth-Eval proceeds via a rigidly defined checkpointing and scoring system:

• Checklist Extraction: Every reference survey is manually annotated to produce:
  • General Checklist ($C_{\mathrm{gen}}$): atomic, verifiable facts—methods, datasets, metrics, timelines.
  • Constraint Checklist ($C_{\mathrm{con}}$): explicit requirements on higher-level structure—e.g., taxonomy, table columns, axis definitions, or specific organizational schemes.
• Automated Verification: For a generated report $\hat{R}$, each checklist item $c\in C$ is scored by a judge LLM:
  • $+1$ if correctly mentioned,
  • $-1$ if hallucinated or incorrect,
  • $0$ if omitted.
• Group Scoring with Fault Tolerance: Items are grouped by logical or topical anchor (e.g., all methods in a taxonomy). For each group $k$ of $N_k$ items, and a saturation threshold $\theta_k\leq N_k$,

$$\mathrm{Score}_k = \min\left(1,\,\tfrac{S_k}{\theta_k}\right),\quad S_k = \sum_{c\in k} r(c).$$

This prevents excessive penalty for underrepresented topics, better modeling expert “good enough” coverage.

• Aggregated Metrics: Overall scores are computed as weighted means over all general and constraint groups:

$$S_{\mathrm{gen}} = \frac{\sum_k w_{\mathrm{gen},k}\,\mathrm{Score}_{\mathrm{gen},k}}{\sum_k w_{\mathrm{gen},k}} \times 100\%,\quad S_{\mathrm{con}} = \frac{\sum_m w_{\mathrm{con},m}\,\mathrm{Score}_{\mathrm{con},m}}{\sum_m w_{\mathrm{con},m}}\times 100\%.$$

$S_{\mathrm{all}}$ averages all group scores by total weight.

• Precision, Recall, and F₁: At the item level, standard IR metrics are computed for finer diagnostic granularity.

This protocol transforms subjective, open-ended survey evaluation into atomic, objective—and automatable—metrics (Zhang et al., 7 Jan 2026).

4. Experimental Setup and Workflow Comparisons

In its principal evaluation regime, DeepSynth-Eval benchmarks models across 96 deep-survey tasks spanning diverse subfields (e.g., multi-document reasoning, architecture summaries, evaluation benchmarks). Two controlled workflows are compared:

• E2E Single-Turn Generation: Concatenation of Oracle Context and composite prompt; the model generates a report in a single pass.
• Agentic Multi-Turn Generation: Iterative, plan-and-write procedure involving:
  • Creation of an "Intellectual Skeleton" (section taxonomy and axis planning),
  • Section-wise relevance selection and note-taking (deep reading of context),
  • Section writing conditioned on local and prior content,
  • Global coherence polishing.

Evaluated models include GPT-5.2, Qwen-235B, and DeepSeek-v3.2, with all receiving identical context and prompts (Zhang et al., 7 Jan 2026).

5. Core Findings and Quantitative Results

The DeepSynth-Eval protocol exposes substantial bottlenecks in current LLM and agentic synthesis capabilities. Key results are summarized in the following table:

Workflow / Model	Overall (%)	General	Constraint	Precision
Reference Survey	96.1	95.5	98.9	99.6
E2E Single-Turn: GPT-5.2	28.3	26.4	36.1	85.5
Agentic Multi-Turn: GPT-5.2	33.3	34.8	26.2	95.3
E2E Single-Turn: Qwen-235B	24.8	24.7	23.3	79.9
Agentic Multi-Turn: Qwen-235B	35.5	37.5	27.5	92.9

Significant empirical findings include:

• Even top LLMs struggle to cover more than ~40% of atomic requirements when synthesizing over ≥100 references in a single pass.
• Agentic, multi-turn workflows systematically outperform single-turn generation, particularly in constraint adherence and reducing hallucinations (precision increase from ~85% to ~95%).
• Scaling model size yields consistent, if sublinear, improvements.
• The main bottleneck remains recall: model outputs omit large numbers of atomic factual/structural items, with major under-elaboration in E2E settings (Zhang et al., 7 Jan 2026).

6. Methodological and Practical Implications

DeepSynth-Eval establishes a new standard for evaluating the cognitive depth of LLMs and autonomous agents in information synthesis by:

• Providing reproducible, fine-grained metrics for long-form, post-retrieval generation tasks.
• Isolating synthesis errors from retrieval errors through Oracle Contexts.
• Enabling comparison across agent workflows, planning strategies, and model architectures using standardized, observable criteria.
• Enabling objective error analysis via checklist breakdowns (e.g., item-level, topic-level, group-level).

This suggests that progress in agentic planning and strategic reading/writing is more important than further gains in local context retrieval or surface-level linguistic metrics for next-generation research assistants.

7. Limitations and Future Directions

While DeepSynth-Eval represents a significant methodological advance for the evaluation of deep synthesis, several limitations are noted:

• All current tasks are within computer science. Extension to biology, social sciences, and other domains is needed to stress test domain transferability.
• The creation of exhaustive checklists remains human-intensive. Future work includes automated constraint discovery from unlabeled corpora and reinforcement learning with checklist-derived reward signals.
• Scaling to even larger Oracle Contexts will necessitate progress in memory and retrieval architectures.
• Current judging relies on LLMs for verification but achieves <5% false penalty on reference outputs, indicating high fidelity (Zhang et al., 7 Jan 2026).

Broader coverage and more automated item extraction will further standardize deep synthesis benchmarking.

---

By defining a rigorous, objective, and scalable protocol for evaluating the synthesis stage in information consolidation—an ability central to advanced LLM agents—DeepSynth-Eval accelerates scientific progress on a key open problem: the transformation of vast, fragmented literatures into coherent, domain-expert-grade narratives (Zhang et al., 7 Jan 2026).