DeepWeb-Bench: A Deep Research Benchmark Demanding Massive Cross-Source Evidence and Long-Horizon Derivation

Abstract: Deep research, in which an agent searches the open web, collects evidence, and derives an answer through extended reasoning, is a prominent use case for frontier LLMs. Frontier deep research products score high on existing benchmarks, making it difficult to distinguish their capabilities from current evaluation data alone. We introduce DeepWeb-Bench, a deep research benchmark that is substantially harder than existing benchmarks for the current frontier. Difficulty comes from three properties of the data itself: each task requires massive evidence collection, cross-source reconciliation, and long-horizon multi-step derivation. We represent these three sources of difficulty as four capability families (Retrieval, Derivation, Reasoning, and Calibration) and report results sliced by family. Every reference answer is accompanied by a source-provenance record with four disclosure levels and cross-source checks where available, making scores easier to audit against the underlying evidence. We evaluate DeepWeb-Bench on nine frontier models and report three findings: (1) retrieval is not the bottleneck, as retrieval failures account for only 12-14% of errors while derivation and calibration failures account for over 70%; (2) strong and weak models fail in qualitatively different ways, with strong models' errors dominated by incomplete derivation and weak models' by hallucinated precision; and (3) models exhibit genuine specialization across domains, with cross-model agreement of only rho = 0.61 and per-case disagreement reaching 18.8 percentage points. The public benchmark release includes the data, rubrics, and evaluation code.

• The paper introduces DeepWeb-Bench, emphasizing multi-source evidence gathering, explicit cross-source reconciliation, and long-horizon, multi-step derivation.
• It employs an 8×8 task matrix across four capability dimensions, demonstrating that even advanced models struggle with composite derivation and calibration.
• Empirical results reveal that while retrieval is largely solved, over 70% of errors arise from incomplete multi-step reasoning and overconfident responses.

DeepWeb-Bench: A Benchmark for Evaluating Deep Research Capabilities in Frontier LLMs

Motivation and Benchmark Design

The "DeepWeb-Bench" benchmark (2605.21482) emerges in response to rapidly saturating performance on prior web-based research benchmarks by state-of-the-art LLM agents, both in vertical research products and command-line research agents. Existing benchmarks inadequately discriminate high-end system capabilities, primarily due to their limited focus—commonly single-source lookup, short horizon reasoning, or evidence collection without explicit multi-step derivation nor robust calibration.

DeepWeb-Bench overcomes these limitations by enforcing three orthogonal sources of difficulty in every task: (1) evidence from numerous heterogeneous web sources, (2) explicit cross-source reconciliation, and (3) long-horizon, multi-step derivation. Each task is modeled as an 8×8 matrix (entity × analytical dimension), systematically covering four capability families: Retrieval (single-source factual extraction), Derivation (multi-step computation from multiple disclosed values), Reasoning (counterfactual or projection), and Calibration (conflict resolution and abstention when evidence is lacking).

Every cell's reference answer includes a precise value, a range estimate with derivation, or a not-available marker, each annotated with exhaustive source-provenance metadata (T1–T4, corresponding to regulatory documents through weakly verified sources) and agreement checkpoints. This rigorous design allows for granular, interpretable, and fully auditable scoring, contrasted with black-box, holistic LLM-judge evaluations.

Task Construction and Dataset Statistics

Domain experts curate entity sets and dimensions to guarantee intra-domain comparability and disclosure variability. The released dataset covers 100 tasks spanning six industrial domains: Technology, Energy & Materials, Industrials & Transport, Consumer, Finance, and Healthcare & Pharma, yielding 6,400 scored cells. Tasks reinforce non-redundancy; not-available responses are required where warranted, to distinguish between knowledge gaps and overconfident hallucination.

Reference answers average 3.2 supporting URLs and 2.4 publishers per cell, with derivation-type cells requiring 2.8 composed steps on average. The majority of dimensions (50%) require derivation, while retrieval constitutes only 12.5%.

Evaluation Protocol and Scoring

All agents are limited to 200 tool-use operations (web search, page visit, pdf fetch) and 30 minutes per task, ensuring comparability and preventing budget artifacts. Each model receives strict prompts specifying only tasks, schema, and units—without reference values or scoring cues.

A fixed rubric scores cell answers: exact match within a detailed tolerance for numeric, range, or not-available types, with explicit partial credit for plausible but imprecise or methodologically sound answers. Scoring reliability is cross-checked with human annotators (Cohen’s K=0.82). The design aggressively mitigates spurious performance due to luck or annotation ambiguity.

Empirical Results and Diagnostics

Nine leading model configurations are benchmarked, including Codex CLI + GPT-5.5, Claude Opus 4.7, DeepSeek V4. Pro/Flash, GLM 5.1, Qwen 3.6 Plus, MiniMax M2.7, and Kimi K2.6. Key findings include:

1. Retrieval is not the primary bottleneck: Retrieval failures account for only 12–14% of errors, indicating that information access (even with the same search tools) is mostly solved among frontier models. Over 70% of failures occur in Derivation (incomplete multi-hop reasoning, composition errors) and Calibration (improper abstention or spurious hallucination).
2. Qualitatively distinct failure modes manifest in strong versus weak models. Strong models are dominated by incomplete derivation errors (e.g., scope misalignment in calculations) at 31%, while weak models are characterized by hallucinated precision (confident, unsupported answers) at 38%. This is a robust indicator of a phase transition with capability scaling: margin for further improvement at the high end is driven by compositionality and calibration, not retrieval or surface pattern matching.
3. Domain-specialization and inconsistency: Models demonstrate specialization by case type, with a mean Spearman rank correlation of only 0.61 across models. There are substantial per-case differences (up to 18.8 percentage points), demonstrating nontrivial divergence in agent strategies and weaknesses. This necessitates domain-stratified evaluation in future work, especially in heterogeneous real-world analytical settings.

Detailed Analyses

Fine-grained, per-cell scoring and source-provenance annotations permit a precise breakdown of error types. Case studies show that even strong models can retrieve all necessary ingredients but misapply a derivation step, while weaker models more often generate unwarranted precision in the face of ambiguous or missing evidence.

Calibration analysis is particularly stringent: precise answers for cells explicitly marked as not-available yield a score of zero, while an explicit justified abstention scores one. This sharply penalizes overconfidence and is a major source of model differentiation.

Cross-task and per-domain diagnostics reveal that the hardest cells (e.g., those requiring financial modeling with non-standardized disclosures or counterfactual estimation) have compressed human-level performance and wide model-to-model spread.

Theoretical and Practical Implications

The study demonstrates that improvement in research agent performance on truly hard, high-stakes analytical tasks will not be unlocked by better retrieval or shallow tool augmentation alone. Explicit, model-internal mechanisms for multi-step quantitative composition, uncertainty quantification, and robust “don’t know” calibration are necessary. The lack of strong correlation across models and the prevalence of hallucinated precision in current systems indicate the limitations of pure model scaling or brute-force prompt engineering.

Methodologically, DeepWeb-Bench’s approach of per-cell scoring, rigorous provenance tracking, and capability-decomposed evaluation is likely applicable as a template for future evaluation of agentic LLMs in other complex settings, such as scientific synthesis, complex legal analysis, or medical question answering, where evidence must be collected, reconciled, and derived from multiple orthogonal sources.

Limitations and Future Directions

While DeepWeb-Bench covers a broad and demanding spectrum of tasks, there remain gaps for workflows involving non-public or private data, qualitative synthesis, or extended user-agent dialog. The benchmark’s fixed task matrix and source-provenance scheme could be further enhanced by extending disclosure and provenance standards to more jurisdictions or document types.

Future developments should explore models with built-in mechanisms for interpretability in multi-step derivation chains, explicit uncertainty modeling, and modular pipelines for cross-source conflict awareness. Human-in-the-loop evaluation and robust abstention metrics should also become more central, especially for models targeted at decision-critical domains.

Conclusion

DeepWeb-Bench constitutes a substantial advance in the evaluation of deep research agents by focusing on the full workflow—massive evidence gathering, cross-source reasoning, and long-horizon, multi-step synthesis. Retrieval is largely saturated, but derivation and calibration remain principal obstacles for frontier models. The capability-targeted, auditable, and provenance-enriched methodology sets a robust foundation for future research in agentic LLMs, with clear implications for both theoretical progress and high-stakes deployment in analytical domains.