Towards Knowledgeable Deep Research: Framework and Benchmark

Abstract: Deep Research (DR) requires LLM agents to autonomously perform multi-step information seeking, processing, and reasoning to generate comprehensive reports. In contrast to existing studies that mainly focus on unstructured web content, a more challenging DR task should additionally utilize structured knowledge to provide a solid data foundation, facilitate quantitative computation, and lead to in-depth analyses. In this paper, we refer to this novel task as Knowledgeable Deep Research (KDR), which requires DR agents to generate reports with both structured and unstructured knowledge. Furthermore, we propose the Hybrid Knowledge Analysis framework (HKA), a multi-agent architecture that reasons over both kinds of knowledge and integrates the texts, figures, and tables into coherent multimodal reports. The key design is the Structured Knowledge Analyzer, which utilizes both coding and vision-LLMs to produce figures, tables, and corresponding insights. To support systematic evaluation, we construct KDR-Bench, which covers 9 domains, includes 41 expert-level questions, and incorporates a large number of structured knowledge resources (e.g., 1,252 tables). We further annotate the main conclusions and key points for each question and propose three categories of evaluation metrics including general-purpose, knowledge-centric, and vision-enhanced ones. Experimental results demonstrate that HKA consistently outperforms most existing DR agents on general-purpose and knowledge-centric metrics, and even surpasses the Gemini DR agent on vision-enhanced metrics, highlighting its effectiveness in deep, structure-aware knowledge analysis. Finally, we hope this work can serve as a new foundation for structured knowledge analysis in DR agents and facilitate future multimodal DR studies.

• The paper presents a Hybrid Knowledge Analysis (HKA) framework that integrates both structured and unstructured data for comprehensive multimodal deep research.
• It demonstrates statistically significant improvements in report depth and key point supportiveness compared to 12 existing deep research systems.
• The KDR-Bench benchmark systematically evaluates multimodal report synthesis across nine domains using 41 research questions and 1,252 curated tables.

Knowledgeable Deep Research: A Hybrid Framework and Benchmark for Multimodal Agentic Reasoning

Problem Motivation and Task Definition

LLM agents are increasingly required to perform multi-step, autonomous research beyond simple information retrieval, notably in Deep Research (DR) applications that demand complex reasoning, information integration, and comprehensive report generation. Existing DR frameworks largely constrain themselves to unstructured web data, neglecting the rigorous incorporation of structured resources (tables, charts, etc.) necessary for quantitative analysis, computation, and deeper insight generation. The paper introduces Knowledgeable Deep Research (KDR), which formalizes the task of integrating both structured and unstructured knowledge in DR agents to yield robust, multimodal (text/figure/table) research reports. KDR requires explicit reasoning chains over diverse data modalities, setting it apart from prior "web-centric" DR platforms.

Hybrid Knowledge Analysis (HKA) Framework

To operationalize KDR, the authors propose the Hybrid Knowledge Analysis (HKA) architecture, a multi-agent system leveraging both LLMs and Vision-LLMs (VLMs) for comprehensive analysis and report synthesis. HKA comprises four key agentic roles:

1. Planner: Decomposes an input research query into explicit subtasks, managing control flow and the invocation of other agent roles based on the knowledge modality required.
2. Unstructured Knowledge Analyzer: Conducts intent-guided web retrieval, aggregation, and summarization of textual data. Query expansion is handled via prompting, and relevance filtering is performed to optimize context usage.
3. Structured Knowledge Analyzer: The architecturally unique module, integrating a code LLM for table-oriented computation and chart generation, and a VLM to analyze output visualizations and extract natural-language insights. Table retrieval is performed via dense semantic search with LLM reranking, and code execution is robustified by error-driven code regeneration.
4. Writer: Resolves conflicting evidence, integrates multimodal content, and generates coherent, logically consistent final reports constrained by context window limits.

   Figure 1: The HKA framework showing planner-controlled invocation of unstructured/structured analyzers and the final writer for multimodal report synthesis.

The architecture places heavy emphasis on explicit, error-tolerant computation over structured sources, multimodality (text, tables, figures), and workflow modularity, distinguishing it from conventional retrieval-augmented generation or naive information-aggregation agents.

KDR-Bench Benchmark Construction and Analysis

To enable systematic evaluation, the paper introduces KDR-Bench, a purpose-built benchmark spanning nine domains (e.g., Finance, Energy, Society, Technology) populated with 41 complex research questions and 1,252 expertly curated tables. Questions are crafted to elicit reasoning that leverages both qualitative and quantitative data, and annotation includes both high-level main conclusions and fine-grained key-point analyses mapped directly to supporting tables.

Figure 2: KDR-Bench construction pipeline and evaluation flow: structured knowledge curation, multifaceted question design, and LLM-centered scoring.

Table distribution statistics (Figure 3) demonstrate domain diversity and the systematically broad coverage of structured knowledge sources underlying the benchmark.

Figure 3: Empirical domain-wise statistics for table resources within KDR-Bench.

Evaluation is carried out along three axes:

• General-purpose: Readability, logical coherence, and comprehensiveness scored by cutting-edge LLMs as judges.
• Knowledge-centric: Alignment with annotated conclusions, coverage of key points, and supportiveness (i.e., correct use of relevant tables).
• Vision-enhanced: Multimodal LLMs assess the utility and informativeness of generated figures/tables in report PDF format, addressing limitations of text-only LLM evaluation.

Experimental Results

The proposed HKA agent is comprehensively compared against 12 baseline DR systems, including both closed (e.g., Gemini, Perplexity) and open-source (e.g., LangChain, ThinkDepth) agents, as well as LLMs with web search augmentation. Despite using similar backbone models, HKA delivers consistent, statistically significant improvements in both general-purpose and knowledge-centric scores over existing open (and several closed) DR agents. Notably:

• HKA achieves superior report depth and key point supportiveness, indicating effective structured knowledge utilization.
• HKA's vision-enhanced win rates (evaluated by a state-of-the-art MLLM on multimodal reports) surpass even Gemini, highlighting the practical utility of its table/figure generation capabilities (See Table 2 in the paper).
• Ablation studies affirm the indispensability of both structured and unstructured sub-agents, as well as the LLM rerank strategy in table selection.
• Evaluation reliability analyses demonstrate consistent scores across judge models and report generations, and high alignment (86.3% pairwise agreement) with human expert preference judgments.

Case studies (Figure 4) outline complex reasoning chains involving both table-based computation and textual evidence triangulation, culminating in domain-specific, insight-rich multimodal output.

Figure 4: Qualitative case trajectory, showing multi-agent invocation of analyzers, table-based computation, and writer integration into the final report.

Theoretical and Practical Implications

This work exposes the limitations of shallow knowledge integration and text-centric report generation in current DR baselines. By isolating and enhancing structured knowledge analysis, the authors move DR agents closer to the requirements of academic, policy-making, or enterprise research, where in-depth evidence synthesis from both files and web is non-negotiable. The modular HKA architecture provides a blueprint for scalable, extensible agent systems, and the rigorous KDR-Bench should steer future DR evaluation to more faithfully reflect real-world multimodality and data diversity.

On the theoretical front, the results underscore the necessity for explicit reasoning over structured sources (beyond append-and-prompt) and the value of multi-agent orchestration in breaking down and reconstructing complex research questions. The deployment of VLMs for chart/table validation within agentic workflows paves the way for richer forms of multimodal LLM integration.

Expected future directions include: tighter integration with external computation engines, extension to live database interaction, advanced cross-modal validation cycles (beyond LLM-only vision), and adaptation to personalized, user-steerable research/reporting scenarios enabled by user profiles or context memory.

Conclusion

The paper presents a principled approach to advancing Deep Research agents towards robust, multimodal, and evidence-grounded reasoning via the HKA framework and KDR-Bench benchmark. By empirically demonstrating the critical role of explicit structured knowledge analysis, it sets a new baseline for the development and evaluation of DR agents aspiring to academic and professional rigor.