M4DocBench: Multimodal Document Research Benchmark

• M4DocBench is a benchmark for evaluating deep research in multimodal, multi-page documents through iterative, multi-hop reasoning.
• It integrates cross-modal evidence from text, tables, figures, and equations to enable advanced document synthesis and retrieval.
• The benchmark’s evaluation uses detailed annotations and adaptive granular retrieval to simulate realistic multi-document research workflows.

M4DocBench is a benchmark designed to rigorously evaluate deep research capabilities in systems that process multimodal, multi-page, and multi-document corpora. Its architecture supports comprehensive assessment along axes that include multi-hop reasoning, cross-modal evidence integration, iterative multi-turn workflows, and adaptive granular retrieval. M4DocBench provides detailed evidence chains and annotated multimodal content, setting a standard for future advancements in the domain of large-scale, multimodal document research.

1. Concept and Objectives

M4DocBench establishes the first benchmark specifically for evaluating “deep research” systems in multimodal document collections (Dong et al., 24 Oct 2025). The focal objectives are:

• Measuring complex reasoning that requires linking evidence across multiple extraction steps (“multi-hop”).
• Dedicated support for cross-modal integration (text, tables, figures, equations), reflecting real-world heterogeneous content.
• Benchmarking systems on their ability to synthesize evidence from sets of documents, simulating “multi-document” research synthesis.
• Enabling evaluation in iterative research workflows (“multi-turn”), benefiting interactive scenarios with progressive query refinement.

These aspects collectively go beyond the scope of standard document QA or visual question answering datasets, aligning the benchmark with the sophisticated requirements in academic, financial, legal, and technical research.

2. Structure and Data Composition

The dataset consists of 158 expert-annotated questions crafted by PhD/Master-level specialists, mapped out over 304 multimodal documents from four primary topical domains: research (scientific papers, conferences), insurance (brochures, policy documents), education (university materials), and finance (market reports) (Dong et al., 24 Oct 2025).

Every question is matched with a “complete evidence chain,” including:

• Document-level, page-level, and fine-grained layout annotations (bounding box coordinates).
• Subquery decomposition for multi-hop tasks.
• Specification of optimal information retrieval granularity for different inquiry types (full document, summary, page, chunk).

The benchmark’s annotation schema ensures full coverage of complex research logic—multi-hop, cross-modal, multi-document, and multi-turn interactions.

3. Evaluation Metrics and Methodology

M4DocBench utilizes metrics tailored to its deep research design:

• Multimodal Retrieval Metrics: Recall@k for document-level, page-level, and layout-level (bounding box overlap with gold standard).
• Document Selection: Precision, recall, and F1 for identifying relevant documents amidst distractors.
• Iterative Deep Research: Formalized by:

$$\mathcal{R}_t = \text{Search}(\tilde{q}_k, \mathcal{D}', \theta),\quad \mathcal{R}_t^* = \text{Refine}(\mathcal{R}_t, \tilde{q}_k),\quad \sigma_t = \text{Evaluate}\left(\bigcup_{i=1}^t \mathcal{R}_i^*, q_i\right)$$

where $\sigma_t$ accumulates evidence over multi-turn iterations until a threshold $\tau$ is reached.

• Answer Accuracy: Only responses satisfying all atomic facts in the benchmark’s explicit checklist are considered correct.

These metrics collectively ensure assessment of retrieval, synthesis, iterative accumulation, and factual correctness.

4. Technical Innovations

M4DocBench is distinguished by several engineering advances:

• Deep Multimodal Parsing: Utilizes sophisticated layout-aware chunking and transcription pipelines preserving both visual semantics and structural coherence (bounding box and content type: text, table, figure, equation).
• Systematic, Hybrid Retrieval: Benchmarked systems employ both text-only (embedding models, dense representations) and vision-only (image embeddings) paradigms, with dynamic selection of retrieval granularity.
• Multi-Agent Iterative Research: Supports workflows where agents filter noisy document sets, select retrieval granularity adaptively, and progressively refine query decomposition in multi-turn settings.
• Rich Evidence Annotation: Annotation covers document, page, and fine-grained regions, enabling detailed evaluation of retrieval precision and reasoning depth.

These innovations facilitate comprehensive deep research workflows not achieved by prior benchmarks.

5. Comparative Evaluation

Experimental results on M4DocBench highlight significant advances:

• The Doc-Researcher system achieves 50.6% accuracy, outperforming the strongest baseline (MDocAgent) by 3.4× (Dong et al., 24 Oct 2025).
• Deep multimodal parsing offers an absolute gain of ≈10% over OCR-based approaches by retaining layout features and modality-specific details.
• Hybrid retrieval strategies—combining dense textual and visual embeddings—consistently outperform single-modality approaches.
• Iterative search-refine loops lead to marked increases in document recall and retrieval quality in the first few iterations.
• Ablation studies reveal a 6–8% accuracy drop upon removal of the adaptive Planner agent, confirming its critical role in effective deep research.

The benchmark’s comprehensive annotation and multi-faceted evaluation underscore the importance of synergistic parsing, retrieval, and iterative reasoning for high-performance document research systems.

6. Distinctiveness Compared to Prior Benchmarks

M4DocBench advances over existing work in several aspects:

• Prior datasets are typically limited to single-document or single-modality QA, often using OCR and lacking multi-document or deep multi-hop reasoning.
• Evidence chains in M4DocBench are annotated for completeness, spanning document/page/layout levels with subquery breakdown.
• The benchmark simulates realistic research by supporting multi-turn analysis and specifying optimal retrieval granularity, yielding finer-grained insights into system behavior.
• Evaluation extends beyond simple QA—requiring systems to aggregate, filter, and synthesize information from noisy multi-document corpora in conversational interaction.

These features establish M4DocBench as a critical resource for the next generation of research systems targeting real-world, multimodal document collections.

7. Implications and Future Research

The introduction of M4DocBench signals several ongoing and future research directions:

• Development of robust multimodal LLM architectures capable of simultaneously processing complex document layouts and integrating signals across text, tables, images, and equations.
• Improvement of multi-agent, iterative workflows that mimic human research by dynamic query decomposition and progressive evidence accumulation.
• Enhanced benchmarks may promote investigation into advanced cross-modal retrieval models, fine-grained chunking strategies, and scalable iterative aggregation algorithms.
• M4DocBench’s comprehensive settings could foster research into dynamic retrieval strategies (e.g., reinforcement learning), as well as into efficient online/offline document corpus management.

A plausible implication is an acceleration of research into scalable and interpretable multimodal document analysis systems for scientific, financial, legal, and educational applications.

---

M4DocBench embodies the rigorous and multifaceted demands of deep research workflows in multimodal document processing, providing the technical foundation, annotated data richness, and evaluation protocols necessary to benchmark and stimulate the development of advanced research systems.