ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios

Abstract: Retrieval-Augmented Generation (RAG) pipelines must address challenges beyond simple single-document retrieval, such as interpreting visual elements (tables, charts, images), synthesizing information across documents, and providing accurate source grounding. Existing benchmarks fail to capture this complexity, often focusing on textual data, single-document comprehension, or evaluating retrieval and generation in isolation. We introduce ViDoRe v3, a comprehensive multimodal RAG benchmark featuring multi-type queries over visually rich document corpora. It covers 10 datasets across diverse professional domains, comprising ~26,000 document pages paired with 3,099 human-verified queries, each available in 6 languages. Through 12,000 hours of human annotation effort, we provide high-quality annotations for retrieval relevance, bounding box localization, and verified reference answers. Our evaluation of state-of-the-art RAG pipelines reveals that visual retrievers outperform textual ones, late-interaction models and textual reranking substantially improve performance, and hybrid or purely visual contexts enhance answer generation quality. However, current models still struggle with non-textual elements, open-ended queries, and fine-grained visual grounding. To encourage progress in addressing these challenges, the benchmark is released under a commercially permissive license at https://hf.co/vidore.

• The paper introduces ViDoRe V3, a comprehensive human-annotated benchmark that evaluates multimodal retrieval-augmented generation in realistic settings.
• It leverages a three-stage annotation pipeline and diverse query types to assess retrieval, synthesis, and cross-modal integration performance.
• Experiments reveal that hybrid context improves answer accuracy, yet visual grounding remains significantly behind human performance.

ViDoRe V3: An End-to-End Benchmark for Multimodal Retrieval-Augmented Generation in Real-World Environments

Motivation and Background

Retrieval-Augmented Generation (RAG) has become the dominant paradigm for knowledge-intensive Natural Language Processing, yet evaluation benchmarks have historically failed to capture the complexity of practical deployment scenarios. Typical benchmarks over-index on single-document, textual-only comprehension, or attempt to disentangle retrieval and generation, ignoring the interplay between multimodal content (e.g., tables, charts, infographics) and composite user information needs. Real-world deployment often requires synthesizing information across heterogeneous sources, localizing supporting evidence, and robust grounding to avoid hallucinations in responses. Existing visual document understanding datasets rarely engage with multi-page or multimodal reasoning, and prior RAG benchmarks provide insufficient coverage of complex user queries and rich document modalities.

Benchmark Design and Annotation Protocol

ViDoRe V3 addresses these deficiencies by introducing a comprehensive human-annotated benchmark with 3,099 queries (spanning six languages) over ~26,000 document pages derived from ten industrial and public corpora, encompassing domains such as finance, computer science, pharmaceuticals, HR, maintenance, and energy. The benchmark leverages a rigorous three-stage human-in-the-loop annotation pipeline comprising:

• Document Curation: Sourcing diverse, openly licensed domain-specific documents in English and French, provided as images, text, and PDFs.
• Query Taxonomy and Generation: Defining seven query types (open-ended, extractive, numerical, multi-hop, compare-contrast, boolean, enumerative) and three formats (question, keyword, instruction). Queries are created from both raw documents (extractive) and synthetic, annotator-blinded contexts to mitigate extractive bias and stimulate authentic, cross-document information seeking.

  Figure 2: Overview of the benchmark creation process; query sourcing, VLM pre-filtering, multi-stage human annotation, and aggregation into reference answers and fine-grained evidence.

• Relevant Page and Evidence Annotation: Initial relevance pre-filtering is performed using a vision-LLM (VLM); subsequent human annotation includes relevance scoring, bounding box localization for supporting evidence (with modality tags), and answer synthesis based strictly on marked pages.
• Stratified Aggregation and Quality Control: Multiple annotators per query where needed, domain-trained annotator pool, reviewer oversight, and aggregation of multiple candidate answers to a single ground truth via VLM-based consolidation.

The resulting dataset is notable for its scale, consistency (Gwet’s AC2 = 0.76), fine-grained multi-modal bounding box annotations, and multilingual coverage.

Figure 3: Example ViDoRe V3 sample illustrating query, relevant page annotation, visual grounding (bounding boxes), and multi-modal page formats.

Query Diversity, Modality Coverage, and Dataset Structure

ViDoRe V3 captures a broad spectrum of real-world information needs and document types:

• Query Type Distribution: Extractive queries dominate due to annotator tendencies, but the benchmark achieves substantial coverage of open-ended and composite queries through targeted sampling and synthetic pipelines.
• Content Modality Distribution: While text is most frequent, tables, charts, images, and infographics are well represented, ensuring models must handle visually dense and structured content.

  Figure 1: Distribution of content types per domain, highlighting strong representation of tables, charts, images, and infographics alongside text.

• Multi-page and Multi-document Complexity: Many queries require integration across multiple pages and documents, reflecting realistic search and synthesis tasks.

  Figure 6: Representative examples from ViDoRe V3 displaying diverse query forms, answer localizations, and cross-modal complexity typical of real industry settings.

Evaluation Methodology and Empirical Findings

Retrieval Performance

A comprehensive evaluation of state-of-the-art retrievers (textual, visual, and late-interaction) establishes:

• Superiority of Visual and Late-Interaction Retrievers: Visual retrievers consistently outperform textual ones of equivalent parameter count, especially for modalities beyond text. ColEmbed-3B-v2 yields top NDCG@10 across modalities and languages.
• Query Complexity Effect: Performance is inversely correlated with query complexity. Boolean/numerical queries achieve much higher NDCG@10 than open-ended and multi-hop variants.

  Figure 4: ColEmbed-3B-v2 NDCG@10 retrieval scores dissected by query type and format.

• Modal and Multi-page Challenges: Mixed and image-heavy queries are the hardest; retriever performance systematically declines as the number of relevant pages increases.

  Figure 7: ColEmbed-3B-v2 retrieval effectiveness by page modality, underscoring the increased challenge posed by images, tables, and mixed content.

  

  Figure 8: Retrieval accuracy as a function of the number of annotated relevant pages for each query, highlighting the sharp decay under increasing context size.

• Textual Rerankers Surpass Multimodal Counterparts: Adding a state-of-the-art textual reranker to base retrievers delivers substantial gains (e.g., +13.2 NDCG@10), surpassing comparable visual rerankers, which often provide negligible improvements.

Answer Generation and Context Utilization

The benchmark assesses end-to-end RAG pipelines on final answer correctness using an LLM judge, stratified by context modality and hybrid context inclusion:

• Visual Context Boosts Hard-case QA Performance: On difficult queries—where parametric knowledge is unhelpful—providing image context yields 2.4–2.8 absolute percentage point gains over textual context for Gemini 3 Pro as the generator.
• Hybrid Pipelines Provide Complementarity: Combining top-ranked pages from visual and textual retrievers produces the strongest performance on challenging queries (54.7% accuracy on hard queries), outstripping pure-text (52.1%) and pure-image (54.5%) setups.
• Substantial Room for Improvement: The gap between best non-oracle pipelines and the oracle (retrieval upper bound) context (>10 percentage points on hard queries) underscores significant headroom for retrieval and grounding advances. Even under oracle context, generation remains imperfect, especially on non-extractive queries requiring synthesis.

Visual Grounding

ViDoRe V3 uniquely stress-tests visual grounding: the ability of models to return bounding boxes pinpointing answer-supporting regions.

• Model vs. Human Performance: While human annotator F1 for evidence box overlap is ~0.60, leading VLMs (Qwen3-VL-30B-A3B-Instruct, Gemini 3 Pro) achieve only 0.065–0.089, with primary errors in recall.
• Qualitative Grounding Analysis: Qwen3-VL produces coarse, often over-inclusive boxes; Gemini's output is more precise but suffers from off-by-one page indexing and lower recall. Both remain far from human agreement.

  Figure 9: Bounding box localization F1 for Qwen3-VL-30B-A3B and Gemini 3 Pro compared to inter-annotator human agreement, revealing a pronounced model–human gap.

  

  Figure 5: Side-by-side examples of Qwen3-VL-30B-A3B visual grounding predictions versus human annotations, illustrating common model localization tendencies and misses.

  

  Figure 13: Comparable Gemini 3 Pro grounding outputs for the same queries as above, demonstrating differences in precision and recall relative to Qwen3-VL.

Implications and Theoretical Considerations

ViDoRe V3 decisively demonstrates real-world bottlenecks in RAG deployment:

• Retrieval Limitations: Even state-of-the-art models struggle with queries demanding multi-hop reasoning, synthesis of information over disparate page ranges, and robust cross-modal integration. Visual retrievers and late-interaction models provide a necessary foundation, but hybrid context retrieval and effective reranking remain open areas for research.
• Modal Synergy: The hybrid context’s non-additive benefit demonstrates that text and image views encode complementary facets of document semantics, motivating continued exploration of multi-stream retrieval architectures and late-fusion techniques.
• Fine-grained Grounding: Visual evidence localization lags far behind human ability, especially for supporting fine-grained QA and interpretable systems. Progress here is essential for reliable citation, regulatory compliance, and user trust.
• Multilinguality and Cross-linguality: Mild performance degradation (2–3 points NDCG@10) in cross-lingual setups suggests that further work is required for robust retrieval in globally deployed RAG applications.

This benchmark brings focus to evaluated tasks and modalities (multimodal, multi-hop, cross-lingual, visually grounded). It also aligns with ongoing trends in RAG towards long-context capability (2601.08620, Xu et al., 7 Jul 2025), adaptive chunking [mixofgranularity], and the emergence of visual citation [ma2024visaretrievalaugmentedgeneration].

Future Directions and Potential in AI

The empirical gaps revealed here compel several research trajectories:

• Cross-modal Retrieval and Reranking: Novel hybrid retrieval/fusion architectures and improved visual rerankers are needed for robust context selection.
• Grounding and Explainability: Improving model alignment for visual grounding—potentially via multi-annotator contrastive learning, better supervision, or retrieval-aware citation—is required for systems to provide robust source evidence.
• Synthetic Data and Annotation Efficiency: Incorporating model-in-the-loop, task-adaptive synthetic query generation may enable broader domain and language coverage with manageable annotation budgets [lu2024machine].
• Long-context RAG: Architectures must scale to reasoning over hundreds of pages, integrating advancements in long-context encoding [jiang2024longragenhancingretrievalaugmentedgeneration] with grounding and multimodal alignment.

Adoption of benchmarks like ViDoRe V3 will inform the development of next-generation document intelligence and regulatory-safe enterprise assistants.

Conclusion

ViDoRe V3 constitutes the first multimodal, multilingual, human-curated benchmark for retrieval-augmented generation in complex, realistic settings. Its design enables fine-grained, interpretable evaluation of retrieval, answer generation, and visual grounding, unifying concerns previously addressed in isolation. The findings reveal both the promise and limitations of current RAG pipelines: while visual and hybrid retrieval methods advance the state of the art, complex open-ended reasoning, multi-hop aggregation, and evidence localization remain unsolved. ViDoRe V3 supplies both a challenge and a systematic reference point for future advances in robust, real-world, multimodal information retrieval and generation (2601.08620).