How Far Are We from Genuinely Useful Deep Research Agents? (2512.01948v1)

Abstract: Deep Research Agents (DRAs) aim to automatically produce analyst-level reports through iterative information retrieval and synthesis. However, most existing DRAs were validated on question-answering benchmarks, while research on generating comprehensive reports remains overlooked. Worse, current benchmarks for report synthesis suffer from task complexity and subjective metrics -- this fails to reflect user demands and limits the practical utility of generated reports. To address these gaps, we present Fine-grained DEepResearch bench (FINDER), an enhanced benchmark consisting of 100 human-curated research tasks with 419 structured checklist items that standardize report structure, analytical depth, and factual grounding. Based on approximately 1,000 reports produced by mainstream DRAs, we further propose Deep rEsearch Failure Taxonomy (DEFT), the first failure taxonomy for deep research agents. DEFT contains 14 fine-grained failure modes across reasoning, retrieval, and generation, and is built upon grounded theory with human-LLM co-annotating and inter-annotator reliability validation. Our experimental findings reveal that current DRAs struggle not with task comprehension but with evidence integration, verification, and reasoning-resilient planning.

• The paper introduces Finder, a detailed, checklist-based benchmark, and DEFT, a taxonomy for failure analysis, to rigorously evaluate deep research agents.
• It shows that generative failures (≥39%) and retrieval deficiencies (32%) overwhelmingly impact performance, while reasoning capabilities are comparatively stronger.
• Experimental results across models and languages underscore systemic issues in evidence integration and strategic fabrication, urging improved verification methods.

Quantitative and Diagnostic Advances for Deep Research Agents

Motivation and Problem Statement

Deep Research Agents (DRAs) are engineered to autonomously synthesize complex, evidence-grounded research reports via multi-step information retrieval, analytical reasoning, and content generation. The current paradigm in DRA evaluation relies predominantly on question-answering benchmarks and subjective metrics, failing to replicate the multi-dimensional standards imposed in authentic research workflows. Existing benchmarks exhibit significant limitations: they either restrict assessment to closed-ended tasks with deterministic ground truths or derive open-ended tasks from LLM-driven sampling, resulting in a disconnect from real user requirements and insufficient analytical depth.

Finder: Comprehensive, Checklist-Based Benchmarking

The paper introduces Finder, a fine-grained benchmark composed of 100 expert-curated research tasks with 419 structured checklist items guiding report structure, analytical depth, and citation integrity. Each task is paired with explicit reporting specifications (length, scope, format) and multi-level checklists covering operationalization, mechanisms, methodology critique, domain contextualization, and actionable interventions.

Figure 1: Distributional shift toward significantly longer and more detailed queries in FInder compared to DRB.

This approach shifts the evaluation paradigm from generalized report scoring to reproducible, fine-grained procedural compliance and meta-reasoning discipline. Through iterative expert–LLM co-generation and manual review cycles, Finder queries exceed prior benchmarks in complexity and specificity, promoting robust coverage of real-world research demands.

Figure 2: Visualization of the distribution of checklist items per Finder query, illustrating the breadth of evaluation.

DEFT: Fine-Grained Failure Taxonomy

The authors propose Deep rEsearch Failure Taxonomy (DEFT), a hierarchical error classification scheme specific to DRAs. DEFT encompasses 14 axial failure categories mapped to three core dimensions: Reasoning, Retrieval, and Generation.

Figure 3: Schematic overview of the DEFT construction pipeline integrating human–AI open, axial, and selective coding iterations.

Reasoning failures include: failure to understand requirements, lack of analytical depth, limited analytical scope, and rigid planning strategy. Retrieval covers deficiencies in external information acquisition, misalignment in evidence representation, ineffective handling and integration, and lack of verification mechanism. Generation comprises redundant content piling, structural disorganization, specification deviation, deficient rigor, and strategic fabrication.

Figure 4: Detailed ontology of Level 1 and Level 2 DEFT categories enabling granular error decomposition.

DEFT is instantiated via grounded theory: failure reports from five distinct LLMs are iteratively coded, refined via embedding-driven similarity reduction and frequency thresholds, then fused through expert-led axial and selective coding, achieving saturation and validated consistency via Krippendorff’s alpha ($\alpha > 0.80$ for most categories).

Experimental Evaluation: Systemic Weaknesses and Quantitative Results

Experiments span proprietary (Gemini-2.5-Pro, O3 Deep Research, Kimi K2, Perplexity), open-source models (WebThinker, AFM, MiroThinker, Tongyi), and agentic frameworks (MiroFlow, OWL, OpenManus) across both English and Chinese queries.

Figure 5: Comparative agent performance on DRB and Finder benchmarks for RACE and FACT metrics.

Key findings highlight that content generation failures dominate (≥39%)—specifically strategic fabrication, where agents produce unsupported but professionally structured reports to simulate academic rigor. Retrieval failures are prominent (32%), driven by inability to integrate multi-source evidence and verify factuality, whereas reasoning failures are lower in frequency, indicating acceptable task comprehension but deficits in adaptability and resilience.

Strong quantitative results include: Gemini-2.5-Pro leads in overall RACE score (50.95), but even top models demonstrate substantial errors in generation and retrieval. MiroFlow’s explicit agentic orchestration yields the highest checklist accuracy (72.19), outperforming proprietary APIs in procedural discipline.

DEFT Failure Pathways and Correlation Structure

Taxonomic analysis shows balanced error propagation: 28.14% Reasoning, 33.10% Retrieval, 38.76% Generation. Correlation matrices reveal latent failure clusters—misunderstanding requirements links directly to irrelevant output, while poor source integration yields chaotic structure and redundancy. Antagonistic axes evidence mechanistic separation between 'concise but false' (fabrication) and 'verbose/disorganized' reports.

Figure 6: Correlation matrix among DEFT failure categories, delineating failure clusters and antagonistic axes.

Qualitative case studies demonstrate that DRAs systematically misrepresent empirical content when faced with ill-specified tasks and tend to mimic factual grounding for checklist compliance, rather than verifying actual evidence.

Information Retrieval Closed-Loop and Failure Modalities

Retrieval analysis indicates DRAs treat evidence collection as a request–response cycle, neglecting multi-stage acquisition, integration, and verification. Failures cluster at initial information gathering, intermediate integration, and final cross-check.

Figure 7: Closed-loop information retrieval workflow in deep research and mapping to specific agentic failure modes.

The absence of explicit knowledge anchoring and dynamic verification mechanisms results in unreliable outputs, evident in fabricated citations or outdated/irrelevant references.

Cross-Lingual and Stability Analysis

Systematic evaluation of MiroFlow under Finder with both English and Chinese prompts demonstrates robust, stable performance and low run-to-run variability. English tasks benefit from marginally higher reasoning and organization scores, but instruction-following and readability remain consistent across languages.

Figure 8: Cross-lingual comparison of RACE scores for MiroFlow under Finder benchmark (mean over three runs).

Implications and Future Directions

This research provides multiple actionable insights for both system development and evaluation methodology:

• Evidence-centric agent evaluation must move beyond answer correctness to structured checklist compliance and error taxonomy diagnosis, enabling reproducibility and transparency.
• Reasoning resilience—defined as the agent’s ability to adapt and recalibrate reasoning in dynamic, evolving tasks—is a superior predictor of performance relative to pure reasoning intensity.
• To mitigate strategic fabrication and citation hallucination, generative constraints and post-hoc evidentiary verification should be extended across both retrieval and generation stages.
• Benchmark complexity, length, and explicit instruction specification—modeled in Finder—expose previously hidden systemic weaknesses.

Theoretical implications include the necessity of closed-loop semantic modeling for information seeking, embedding-based error reduction for taxonomic coding, and geometric performance metrics (cosine-based) for interpretable category-level analysis.

Practical advancement centers on integrating DEFT as a diagnostic layer in agentic pipelines, informing tool extension, retrieval orchestration, and procedural checklists for future DRAs.

Conclusion

The paper "How Far Are We from Genuinely Useful Deep Research Agents?" (2512.01948) establishes Finder and DEFT as a unified evaluation and diagnosis framework, quantitatively exposing that current deep research agents are constrained by evidence integration, methodological rigor, and resilience in dynamic reasoning—rather than by comprehension or surface-level capabilities. The checklist-based, taxonomy-grounded analysis signifies a shift in research agent benchmarking, driving the field toward reliability, transparency, and actionable progress in autonomous research report synthesis.