---
title: 'UniDoc-Bench: Multimodal Retrieval Benchmark'
url: https://www.emergentmind.com/topics/unidoc-bench
type: topic
---

# UniDoc-Bench: Multimodal Retrieval Benchmark

UniDoc-Bench is a large-scale, realistic benchmark for document-centric multimodal retrieval-augmented generation (MM-RAG), designed to evaluate systems that retrieve and reason over text, tables, and figures within real-world PDF documents [2510.03663]. It is built from 70,000 PDF pages across eight domains and contains 1,600 multimodal question-answer pairs spanning factual retrieval, comparison, summarization, and logical reasoning queries. A central design goal is apples-to-apples comparison across four retrieval paradigms—text-only, image-only, multimodal joint embedding, and multimodal text-image fusion—under a unified protocol with standardized candidate pools, prompts, and evaluation metrics. The benchmark’s reported findings indicate that multimodal text-image fusion RAG systems consistently outperform both unimodal baselines and jointly multimodal embedding-based retrieval, while also revealing systematic failure modes in current MM-RAG pipelines [2510.03663].

## 1. Benchmark scope and design target

UniDoc-Bench was introduced to address a specific evaluation gap in multimodal retrieval-augmented generation: prior evaluations were described as fragmented, often focusing on either text or images in isolation or on simplified multimodal setups that do not capture document-centric multimodal use cases [2510.03663]. Its target setting is document-centric MM-RAG, where evidence is distributed across heterogeneous document elements and may require cross-modality grounding rather than purely textual lookup or purely visual search.

The benchmark is explicitly organized around four question families: factual retrieval, comparison, summarization, and logical reasoning. These are paired with four evidence-modality regimes: text-only, image-only, image-plus-text, and table-required. This structure makes the benchmark diagnostic as well as comparative. It does not merely ask whether a system answers correctly; it probes whether success depends on textual evidence, visual evidence, structured tabular evidence, or multimodal composition.

A plausible implication is that UniDoc-Bench is intended less as a narrow leaderboard and more as a controlled evaluation substrate for MM-RAG design choices. The paper’s emphasis on unified protocol, balanced modality conditions, and systematic failure analysis supports that reading [2510.03663].

## 2. Corpus construction and document grounding

The corpus is drawn from PDFA, described as a large-scale dataset of real-world PDF documents, and the benchmark curates 70,000 PDF pages across eight industries: Finance, Legal, Healthcare, Commerce and Manufacturing, CRM, Energy, Education, and Construction [2510.03663]. Documents are manually filtered and tagged by domain, subdomain, content modality, quality, language, and format. The reported average is about 8,000 pages per domain.

The extraction pipeline separates and links multiple content forms. Text chunks are extracted via PDF-to-text tooling with careful chunking. Tables and figures are parsed as images, and in-text references are replaced by unique placeholders such as `<<fig-XXX>>` and `<<tab-XXX>>`. Table content is also extracted as text where possible. Captions and surrounding text are retained and linked, so that figure and table evidence remains connected to nearby discourse rather than becoming isolated image artifacts [2510.03663].

To support multi-hop and multi-modal evidence tracing, chunks across modalities are linked via entity overlap into a knowledge graph for each domain:
$$
\mathcal{G}_i = (N_i, E_i), \quad N_i = \{n_{i1}, n_{i2}, ...\}
$$
where nodes are content chunks and edges represent entity overlaps [2510.03663]. In operational terms, this graph structure functions as the grounding layer for evidence assembly during QA construction. This suggests that UniDoc-Bench is structured around document-internal relational coherence, not merely around independently retrievable snippets.

## 3. QA synthesis, modality balance, and annotation quality

The benchmark contains 1,600 QA pairs, with 200 per domain and 400 per question type [2510.03663]. Generation is semi-automatic: domain-specific templates are combined with LLMs—GPT-4.1 for generation and Gemini-Pro-2.5 for verification—and then further human curation is applied for realism, diversity, and balance. The resulting questions are intended to be self-contained, grounded, and not artifactually dependent on seeing an image or table outside document context.

The answer-modality distribution is balanced within each 200-question block: 50 text-only, 50 image-only, 50 text-plus-image, and 50 table-only. The benchmark therefore enforces a 50% single-modality and 50% multi-modality split [2510.03663]. On average, each QA requires 2.15 evidence items, which directly encodes cross-evidence composition into the benchmark rather than treating multi-hop cases as marginal.

Quality assurance is a prominent part of the construction. Twenty percent of QAs, or 320 pairs, are triply annotated, with dispute adjudication by experts. The stated criteria are factuality, completeness, grounding, self-containment, and human-likeness [2510.03663]. This annotation regime matters because the benchmark evaluates not only retrieval but also answer faithfulness and completeness, which are sensitive to annotation ambiguity. The multiple-annotator and adjudication setup is intended to reduce such ambiguity.

## 4. Evaluation protocol and scoring framework

UniDoc-Bench supports four MM-RAG paradigms under identical candidate pools, prompt instructions, and scoring metrics [2510.03663]. The paradigms are:

| Paradigm | Retrieval setup | Reported role |
|---|---|---|
| Text-only RAG | Retrieves textual chunks only | Ignores non-textual evidence at retrieval time |
| Image-only RAG | Retrieves page images via VLMs | Treats each PDF page as a JPEG image |
| MM-Joint | Uses joint text-image embeddings | Shared-space multimodal retrieval |
| MM-Fusion, T+I | Retrieves text and images separately, then combines both sets | Downstream fusion with strong unimodal retrievers |

A critical fairness mechanism is built into the protocol. When a text-only system retrieves text chunks, associated images and tables are matched using in-text links and passed to the generator if relevant, so that the information supply is equalized across systems [2510.03663]. This prevents evaluation artifacts in which a multimodal generator is disadvantaged simply because evidence linkage was withheld upstream.

Retrieval is scored with Recall@10 and Precision@10. For each query, top-10 retrieved items are checked against all ground-truth evidence, matched by domain, page, and type. A retrieved item is counted as a true positive if page number, file, and type match the gold context; partial overlaps are also treated as correct when the retrieved item covers the answer’s page [2510.03663]. The paper summarizes this as a page- and file-grounded hit criterion.

End-to-end QA is evaluated with Faithfulness and Completeness. The judging protocol is LLM-based: necessary facts are extracted from gold evidence and system output; faithfulness checks whether stated facts appear in the retrieved contexts, and completeness checks whether all gold facts appear in the system answer [2510.03663]. This makes answer evaluation explicitly evidence-conditioned rather than relying on surface-form string overlap.

## 5. Empirical findings and failure modes

The benchmark reports that multimodal text-image fusion achieves the strongest average retrieval precision and the best end-to-end completeness, while image-only retrieval attains high recall but low precision, and multimodal joint embeddings do not match the robustness of fusion [2510.03663]. The following aggregate results are reported:

| Paradigm | Precision@10 / Recall@10 | Completeness (top-10) |
|---|---|---|
| Text-only | 0.406 / 0.796 | 0.653 |
| Image-only | 0.275 / 0.829 | 0.545 |
| MM-Joint | 0.341 / 0.870 | 0.641 |
| T+I Fusion | 0.479 / 0.867 | 0.684 |

These numbers support several of the paper’s headline conclusions. First, T+I Fusion achieves the highest precision and the highest completeness. Second, Image-only has higher recall than Text-only but markedly lower precision, consistent with page-level image retrieval being broader and less targeted. Third, Text-only outperforms the current best MM-Joint model on end-to-end completeness, which the paper uses to argue that not all multimodal models are robust in practice [2510.03663].

The benchmark also reports modality-specific asymmetries. Text retrieval is strong on text-only QAs and weak on image-required ones; image retrieval shows the converse pattern; T+I fusion is the most robust across answer types. Question type—factual, comparison, summary, reasoning—is said to have minimal impact relative to answer modality, with image-only and text-plus-image conditions driving difficulty [2510.03663]. QAs requiring multi-hop or multi-modality evidence especially benefit from fusion.

The failure analysis is correspondingly structured. Text-only RAG fails on questions requiring interpretation of figures, charts, and spatial or visual cues. Image-only RAG fails to extract nuanced, contextually situated facts, entities, or logical chains apparent only in text. MM-Joint embeddings are reported to have lower precision than fusion and, in practice, to resemble image retrieval more than balanced multimodal retrieval [2510.03663]. The benchmark further notes that image-required questions remain the most challenging for all systems, and that domains with a higher percentage of content-rich images, such as Finance and Construction, pose greater challenges. Evidence-type analytics describe text RAG as strong at entity recognition, comparative analysis, contextual numerical reasoning, and specialized terminology, while image RAG is strong at chart and table interpretations, timelines, technical diagrams, and spatial or geographical reasoning [2510.03663].

## 6. Relation to UniDoc and UniDoc-RL

Despite the shared naming, UniDoc-Bench is distinct from both UniDoc and UniDoc-RL. The earlier "UniDoc" model is a universal large multimodal model for simultaneous text detection, recognition, spotting, and understanding, and its paper does not explicitly introduce a benchmark named UniDoc-Bench. Instead, it establishes a comprehensive evaluation protocol over existing public benchmarks spanning text detection, recognition, spotting, VQA, KIE, and handwritten mathematical expression recognition [2308.11592]. The connection is therefore nominal and thematic rather than benchmark-continuous.

The relation to UniDoc-RL is also non-identity. UniDoc-RL is a unified reinforcement learning framework for visual RAG in which an LVLM agent jointly performs retrieval, reranking, active visual perception, and reasoning through a hierarchical action space and dense multi-reward training [2604.14967]. Its evaluation uses ViDoSeek, SlideVQA, and MMLongBench, and its training data is aggregated from SlideVQA, DoubleBench, VisR-Bench, DocBench, and DUDE. The paper explicitly states that it is not directly benchmarked on a dataset named UniDoc-Bench; instead, it curates a custom dataset of high-quality multi-step reasoning trajectories that serves a similar unifying function for RL in visual RAG settings [2604.14967].

A common misconception is therefore to treat UniDoc-Bench as the benchmark introduced by UniDoc or as the evaluation substrate used by UniDoc-RL. The available descriptions do not support either claim. UniDoc-Bench is a separate benchmark focused specifically on document-centric MM-RAG, whereas UniDoc is a model-centered universal multimodal OCR and understanding system, and UniDoc-RL is an RL framework for coarse-to-fine visual RAG [2308.11592] [2604.14967].

## 7. Research significance and methodological implications

UniDoc-Bench’s reported contribution is not limited to scale. Its methodological significance lies in its attempt to standardize cross-paradigm MM-RAG evaluation under a unified protocol, with candidate-pool control, document-grounded evidence matching, and LLM-based faithfulness and completeness scoring [2510.03663]. This combination makes it possible to compare retrieval paradigms without confounding changes in prompts, evidence availability, or answer judging.

The benchmark’s principal substantive conclusion is that multimodal text-image fusion currently provides the most robust operating point for document-centric MM-RAG. The paper presents this as evidence that neither text nor images alone are sufficient and that current multimodal joint embeddings remain inadequate [2510.03663]. It further derives explicit engineering guidance: use strong dedicated text and image retrievers and fuse them at generation time; balance candidate pools carefully; rely on LLM-based answer judging rather than simple string matching; prioritize multimodal fusion for documents rich in figures and tables; and evaluate cross-modality conditions rather than only unimodal cases [2510.03663].

This suggests a broader research interpretation. UniDoc-Bench operationalizes document-centric multimodality as an evidence-integration problem rather than as a simple extension of image retrieval or text retrieval. In that framing, the benchmark’s diagnostic value lies in showing where current MM-RAG systems lose information: at retrieval precision, at modality-specific grounding, and at answer synthesis under mixed evidence requirements.

Source: https://www.emergentmind.com/topics/unidoc-bench