---
title: 'VisR-Bench: Multilingual Visual Retrieval Benchmark'
url: https://www.emergentmind.com/topics/visr-bench
type: topic
---

# VisR-Bench: Multilingual Visual Retrieval Benchmark

VisR-Bench is a benchmark for **multilingual, multimodal retrieval-augmented question answering on long documents**. It is specifically designed to stress-test visual retrieval and RAG pipelines that must locate evidence in multi-page PDFs containing a mix of text, figures, and tables, across sixteen languages. The benchmark comprises **1,286 documents** with an **average length of ~18 pages** and **35,571 high-quality QA pairs**, and it evaluates retrieval models on whether they can find the correct **evidence page** for a question before answer generation is attempted [2508.07493].

## 1. Scope and defining problem

VisR-Bench formulates **question-driven multimodal retrieval on long documents** as a page-level evidence retrieval problem. Given a **query question** and a **multi-page document**, the retriever must return the **single evidence page** that contains the information required to answer the question. For full visual RAG evaluation, a vision-language model is then given the retrieved page image and asked to generate the answer [2508.07493].

The benchmark is motivated by several limitations in prior work. Existing document and VQA benchmarks are often **English-only**, assume the evidence image or page is already given, emphasize **text-only retrieval**, or do not systematically cover **text, figures, and tables inside PDFs**. VisR-Bench is positioned as the first **question-driven multilingual visual retrieval benchmark for long documents**, and it explicitly includes queries without explicit answers, preventing models from relying on superficial keyword matching [2508.07493].

A central design choice is that retrieval is judged by whether the retrieved page is **answer-bearing evidence**, not by whether it is visually or lexically similar to the query. This makes the benchmark a retrieval benchmark for **long, visually rich documents**, rather than a single-page document QA dataset or a text retrieval dataset with OCR appended.

## 2. Corpus construction and dataset composition

The document corpus starts from **Common Crawl PDF** data and is parsed with the **Adobe Document Extract API**, which produces per-page **Markdown** and extracts **figures** as separate image files. After filtering out PDFs with extraction or quality issues, the retained pool contains **301,553 documents** [2508.07493].

The English multimodal split is constructed by manually selecting approximately **4,000 PDFs** of varied types and then choosing **10 visually rich types** for the benchmark, including **product manuals, presentations, newsletters, magazines, brochures, flyers, itineraries, instructions, information sheets, recipes, etc.** Figure images are classified with **CLIP ViT-L/14-336** into **19 categories**, and the benchmark retains the **6 informative figure types** while discarding decorative ones. The final English multimodal set contains **373 English documents**, with **210 table-rich**, **310 text-rich**, and **125 figure-rich** documents; all are manually checked for harmful content, PII, and valid licenses explicitly permitting research use [2508.07493].

The multilingual split contains **913 non-English documents** across **Spanish, Italian, German, French, Dutch, Arabic, Croatian, Japanese, Swedish, Vietnamese, Portuguese, Finnish, Czech, Slovenian,** and **Danish**. Together with the English split, this yields **1,286 documents** across **16 languages** [2508.07493].

The paper also reports a **210K QA** multilingual training resource across **39.5K documents** for fine-tuning experiments. By contrast, the benchmark itself is presented primarily as an evaluation resource; the paper does not describe standard train/dev/test splits for the benchmark and instead emphasizes page-level retrieval evaluation and downstream QA analysis.

## 3. Evidence types and QA construction

VisR-Bench organizes questions by the **primary evidence type** on the page: **figures**, **text**, and **tables** [2508.07493].

| Question type | Evidence | Construction constraint |
|---|---|---|
| Figure-related questions | Figures plus local context | Retain only questions where the figure is required to answer |
| Text-based questions | Text-only pages | Answers must be explicitly present in the text |
| Table-related questions | Tables but no figures | Generate reasoning-heavy questions; avoid direct fact look-up |

For **figure-related QA**, GPT‑4o is prompted with the figure image and nearby textual context. The crucial filtering step is a **visual dependence check**: the page’s **Markdown text** is given to GPT‑4o without the figure, and if GPT‑4o can answer correctly from text alone, the QA pair is discarded. The retained figure questions therefore require actual figure inspection rather than page-text matching [2508.07493].

For **text-based QA**, pages are filtered so that they contain **only text**, with no tables and no figures. GPT‑4o is then prompted to generate up to three QA pairs per page, under the constraints that the questions are **page-specific**, not global or cross-page, and that the answers are **explicit spans** in the text [2508.07493].

For **table-based QA**, pages must contain **tables but no figures**. GPT‑4o is instructed to generate up to three **reasoning-based** QA pairs per page, emphasizing **trend analysis, comparisons, rankings,** and **simple correlations**, while avoiding direct one-cell lookups. This makes table questions structurally different from string-matching table QA [2508.07493].

In the multilingual pipeline, GPT‑4o detects the page language and generates question–answer pairs in both the **document language** and **English**. Across all settings, the **page** is the retrieval unit. The benchmark does not provide explicit bounding boxes or coordinate-level grounding for evidence locations; the supervision is page-level.

## 4. Representations and evaluation protocol

The document representation combines **per-page Markdown**, **Markdown tables**, and **references to extracted figure images**. For retrieval, each page is treated as the main evidence object. For answer generation, the retrieved **page image** is passed to a vision-language model [2508.07493].

Retrieval is evaluated with **top‑1** and **top‑5** page retrieval accuracy. In the benchmark’s single-relevant-page setting, this is equivalent to Recall@\(k\):

$$
\text{Accuracy@k} = \frac{1}{|Q|} \sum_{q \in Q} \mathbf{1}[\text{rank}_q \le k].
$$

The paper reports this metric both as an overall score and as a breakdown by **figure**, **table**, and **text** questions on the English split, and by **language** on the multilingual split [2508.07493].

Answer generation is evaluated with **PNLS** and **GPT-based Evaluation (GAcc)**. **PNLS** is described as a variant of normalized Levenshtein similarity that rewards **partial correctness** and does not over-penalize shorter correct answers. **GAcc** uses **GPT‑4o** as a judge: if the model answer is semantically equivalent to the ground-truth answer, the instance receives a score of 1; otherwise 0 [2508.07493].

This evaluation design separates two problems that are often conflated in document QA: locating the correct page and extracting or reasoning over the content on that page. A plausible implication is that VisR-Bench is useful both for pure retrieval evaluation and for end-to-end visual RAG evaluation.

## 5. Models evaluated and empirical findings

The retrieval baselines fall into three groups. **Text-based methods** include **BM25**, **SBERT**, **BGE-large**, **BGE-M3**, and **NV-Embed-v2**. **Multimodal encoders** include **CLIP** and **SigLIP**. **MLLM-based retrievers** include **VisRAG**, **VLM2Vec**, **GME**, **ColInternVL2**, **ColPhi**, **ColPali-v1.2**, and **ColQwen2-v0.1**. For answer generation, the paper evaluates **GPT‑4o**, **InternVL2-4B**, **Phi-4-multimodal**, and **Paligemma2-3B** [2508.07493].

On the English split, **ColQwen2-v0.1** is the strongest retriever, with **average top‑1 accuracy: 75.23%** and **average top‑5 accuracy: 94.08%**. Its top‑1 accuracy is **74.58** on figures, **67.43** on tables, and **83.68** on text. **ColPali** reaches **72.51** average top‑1, **ColPhi** **72.03**, **ColInternVL2** **70.44**, and **GME** **68.63**. The best text-only model, **NV-Embed-v2**, reaches **46.95** average top‑1, while **CLIP** and **SigLIP** reach **32.68** and **34.26** respectively [2508.07493].

The English results show a consistent content-type pattern. **Text retrieval is easiest**, **figures are intermediate**, and **tables are hardest**. Even the best model has **67.43** top‑1 accuracy on tables, compared with **83.68** on text. The paper explicitly notes that **late interaction beats single-vector encoding**: **ColQwen2** outperforms **GME** despite the latter using a larger base model. This suggests that **token-level interaction and context preservation matter more than base model size for retrieval on structured documents** [2508.07493].

On the multilingual split, **ColQwen2-v0.1** is again the best or near-best in most languages, with examples including **Spanish: 75.04 / 94.34**, **German: 78.63 / 95.77**, **French: 77.81 / 93.69**, and **Portuguese: 76.32 / 93.53** for top‑1 / top‑5 accuracy. Performance drops sharply for some low-resource settings, especially **Arabic: 12.05 / 27.16** and **Vietnamese: 35.39 / 64.51**. The multilingual results also show that **text-based methods remain competitive, often beating MLLMs** in specific languages: **BGE-M3** reaches **77.19** top‑1 in Finnish, and **BM25** is strong in **Spanish**, **Vietnamese**, and **Czech** [2508.07493].

The paper further compares **ColQwen2 (E)**, trained on original mostly English data, with **ColQwen2 (M)**, trained on the same data plus the **210K multilingual QA** resource. The overall average top‑1 score rises from **53.84** to **56.07**, with especially large gains in **Portuguese: 49.44 → 71.86** and **Czech: 52.87 → 61.32**. This indicates that multilingual fine-tuning on VisR-Bench data improves retrieval, although gains remain uneven across languages [2508.07493].

For answer generation, **GPT‑4o (all page)** is the strongest model, with **PNLS ~0.82 average** and **GAcc ~0.62–0.64** on the English split. Among the listed open models, **InternVL2-4B** performs best, with **average GAcc: 0.58** and **PNLS: 0.71** on the English split, while **Phi‑4-multimodal** and **Paligemma2-3B** are substantially lower. This indicates that retrieval quality is necessary but not sufficient: even when the correct page is available, structured tables and multilingual pages remain difficult for current open MLLMs [2508.07493].

## 6. Significance, limitations, and naming context

VisR-Bench exposes several systematic failure modes. Hard negatives often contain **similar pages or figures**, and models confuse them because of shared keywords or similar chart style. Table questions remain difficult because current retrievers and MLLMs are not yet **table-structure aware**. The multilingual results further show uneven robustness across scripts and language families, with especially weak performance on **Arabic**, **Finnish**, and **Vietnamese** [2508.07493].

The benchmark’s limitations are also explicit. The documents are drawn from **web-crawled PDFs**, so some domains remain underrepresented; the language set, while broad, does not cover scripts and languages such as **Chinese, Hindi,** or **Russian**; the QA pairs are **synthetic**, generated by GPT‑4o and then filtered; and the supervision is at the **page level**, not the region level. The paper identifies several future directions: **multi-page evidence retrieval**, **explicit region-level annotations**, and **table-specific and chart-specific retrieval and reasoning modules** [2508.07493].

Within the broader benchmark landscape, VisR-Bench is specialized. It is not a multi-turn browsing benchmark, a purely visual-native search benchmark, or a natural-image visual reasoning benchmark; it is a benchmark for **question-driven multilingual visual retrieval in long documents**. The name should also be distinguished from **VisRes Bench**, which is **often written VisR-Bench in the paper’s text** but studies visual reasoning in naturalistic settings without contextual language supervision [2512.21194]. That naming overlap can create confusion, but the actual scopes are substantially different.

Taken as a whole, VisR-Bench establishes a benchmark for **multilingual multimodal long-document retrieval** in which **page-level evidence supervision**, **figure-, text-, and table-based questions**, and **visual-dependent queries** are all central. This suggests that it functions not only as a scorecard for current retrievers, but also as a diagnostic resource for document-aware multimodal architectures, multilingual fine-tuning, and end-to-end visual RAG research [2508.07493].

Source: https://www.emergentmind.com/topics/visr-bench