VisR-Bench: Multilingual Visual Retrieval Benchmark
- VisR-Bench is a benchmark for multilingual, multimodal retrieval-augmented QA on long documents, focusing on accurate page-level evidence retrieval.
- It evaluates models by testing retrieval on diverse document types with text, figures, and tables, ensuring queries require visual-dependent inspection.
- Empirical results highlight that token-level interactions and fine-tuning improve retrieval performance, especially on challenging table and multilingual queries.
VisR-Bench is a benchmark for multilingual, multimodal retrieval-augmented question answering on long documents. It is specifically designed to stress-test visual retrieval and RAG pipelines that must locate evidence in multi-page PDFs containing a mix of text, figures, and tables, across sixteen languages. The benchmark comprises 1,286 documents with an average length of ~18 pages and 35,571 high-quality QA pairs, and it evaluates retrieval models on whether they can find the correct evidence page for a question before answer generation is attempted (Chen et al., 10 Aug 2025).
1. Scope and defining problem
VisR-Bench formulates question-driven multimodal retrieval on long documents as a page-level evidence retrieval problem. Given a query question and a multi-page document, the retriever must return the single evidence page that contains the information required to answer the question. For full visual RAG evaluation, a vision-LLM is then given the retrieved page image and asked to generate the answer (Chen et al., 10 Aug 2025).
The benchmark is motivated by several limitations in prior work. Existing document and VQA benchmarks are often English-only, assume the evidence image or page is already given, emphasize text-only retrieval, or do not systematically cover text, figures, and tables inside PDFs. VisR-Bench is positioned as the first question-driven multilingual visual retrieval benchmark for long documents, and it explicitly includes queries without explicit answers, preventing models from relying on superficial keyword matching (Chen et al., 10 Aug 2025).
A central design choice is that retrieval is judged by whether the retrieved page is answer-bearing evidence, not by whether it is visually or lexically similar to the query. This makes the benchmark a retrieval benchmark for long, visually rich documents, rather than a single-page document QA dataset or a text retrieval dataset with OCR appended.
2. Corpus construction and dataset composition
The document corpus starts from Common Crawl PDF data and is parsed with the Adobe Document Extract API, which produces per-page Markdown and extracts figures as separate image files. After filtering out PDFs with extraction or quality issues, the retained pool contains 301,553 documents (Chen et al., 10 Aug 2025).
The English multimodal split is constructed by manually selecting approximately 4,000 PDFs of varied types and then choosing 10 visually rich types for the benchmark, including product manuals, presentations, newsletters, magazines, brochures, flyers, itineraries, instructions, information sheets, recipes, etc. Figure images are classified with CLIP ViT-L/14-336 into 19 categories, and the benchmark retains the 6 informative figure types while discarding decorative ones. The final English multimodal set contains 373 English documents, with 210 table-rich, 310 text-rich, and 125 figure-rich documents; all are manually checked for harmful content, PII, and valid licenses explicitly permitting research use (Chen et al., 10 Aug 2025).
The multilingual split contains 913 non-English documents across Spanish, Italian, German, French, Dutch, Arabic, Croatian, Japanese, Swedish, Vietnamese, Portuguese, Finnish, Czech, Slovenian, and Danish. Together with the English split, this yields 1,286 documents across 16 languages (Chen et al., 10 Aug 2025).
The paper also reports a 210K QA multilingual training resource across 39.5K documents for fine-tuning experiments. By contrast, the benchmark itself is presented primarily as an evaluation resource; the paper does not describe standard train/dev/test splits for the benchmark and instead emphasizes page-level retrieval evaluation and downstream QA analysis.
3. Evidence types and QA construction
VisR-Bench organizes questions by the primary evidence type on the page: figures, text, and tables (Chen et al., 10 Aug 2025).
| Question type | Evidence | Construction constraint |
|---|---|---|
| Figure-related questions | Figures plus local context | Retain only questions where the figure is required to answer |
| Text-based questions | Text-only pages | Answers must be explicitly present in the text |
| Table-related questions | Tables but no figures | Generate reasoning-heavy questions; avoid direct fact look-up |
For figure-related QA, GPT‑4o is prompted with the figure image and nearby textual context. The crucial filtering step is a visual dependence check: the page’s Markdown text is given to GPT‑4o without the figure, and if GPT‑4o can answer correctly from text alone, the QA pair is discarded. The retained figure questions therefore require actual figure inspection rather than page-text matching (Chen et al., 10 Aug 2025).
For text-based QA, pages are filtered so that they contain only text, with no tables and no figures. GPT‑4o is then prompted to generate up to three QA pairs per page, under the constraints that the questions are page-specific, not global or cross-page, and that the answers are explicit spans in the text (Chen et al., 10 Aug 2025).
For table-based QA, pages must contain tables but no figures. GPT‑4o is instructed to generate up to three reasoning-based QA pairs per page, emphasizing trend analysis, comparisons, rankings, and simple correlations, while avoiding direct one-cell lookups. This makes table questions structurally different from string-matching table QA (Chen et al., 10 Aug 2025).
In the multilingual pipeline, GPT‑4o detects the page language and generates question–answer pairs in both the document language and English. Across all settings, the page is the retrieval unit. The benchmark does not provide explicit bounding boxes or coordinate-level grounding for evidence locations; the supervision is page-level.
4. Representations and evaluation protocol
The document representation combines per-page Markdown, Markdown tables, and references to extracted figure images. For retrieval, each page is treated as the main evidence object. For answer generation, the retrieved page image is passed to a vision-LLM (Chen et al., 10 Aug 2025).
Retrieval is evaluated with top‑1 and top‑5 page retrieval accuracy. In the benchmark’s single-relevant-page setting, this is equivalent to Recall@:
The paper reports this metric both as an overall score and as a breakdown by figure, table, and text questions on the English split, and by language on the multilingual split (Chen et al., 10 Aug 2025).
Answer generation is evaluated with PNLS and GPT-based Evaluation (GAcc). PNLS is described as a variant of normalized Levenshtein similarity that rewards partial correctness and does not over-penalize shorter correct answers. GAcc uses GPT‑4o as a judge: if the model answer is semantically equivalent to the ground-truth answer, the instance receives a score of 1; otherwise 0 (Chen et al., 10 Aug 2025).
This evaluation design separates two problems that are often conflated in document QA: locating the correct page and extracting or reasoning over the content on that page. A plausible implication is that VisR-Bench is useful both for pure retrieval evaluation and for end-to-end visual RAG evaluation.
5. Models evaluated and empirical findings
The retrieval baselines fall into three groups. Text-based methods include BM25, SBERT, BGE-large, BGE-M3, and NV-Embed-v2. Multimodal encoders include CLIP and SigLIP. MLLM-based retrievers include VisRAG, VLM2Vec, GME, ColInternVL2, ColPhi, ColPali-v1.2, and ColQwen2-v0.1. For answer generation, the paper evaluates GPT‑4o, InternVL2-4B, Phi-4-multimodal, and Paligemma2-3B (Chen et al., 10 Aug 2025).
On the English split, ColQwen2-v0.1 is the strongest retriever, with average top‑1 accuracy: 75.23% and average top‑5 accuracy: 94.08%. Its top‑1 accuracy is 74.58 on figures, 67.43 on tables, and 83.68 on text. ColPali reaches 72.51 average top‑1, ColPhi 72.03, ColInternVL2 70.44, and GME 68.63. The best text-only model, NV-Embed-v2, reaches 46.95 average top‑1, while CLIP and SigLIP reach 32.68 and 34.26 respectively (Chen et al., 10 Aug 2025).
The English results show a consistent content-type pattern. Text retrieval is easiest, figures are intermediate, and tables are hardest. Even the best model has 67.43 top‑1 accuracy on tables, compared with 83.68 on text. The paper explicitly notes that late interaction beats single-vector encoding: ColQwen2 outperforms GME despite the latter using a larger base model. This suggests that token-level interaction and context preservation matter more than base model size for retrieval on structured documents (Chen et al., 10 Aug 2025).
On the multilingual split, ColQwen2-v0.1 is again the best or near-best in most languages, with examples including Spanish: 75.04 / 94.34, German: 78.63 / 95.77, French: 77.81 / 93.69, and Portuguese: 76.32 / 93.53 for top‑1 / top‑5 accuracy. Performance drops sharply for some low-resource settings, especially Arabic: 12.05 / 27.16 and Vietnamese: 35.39 / 64.51. The multilingual results also show that text-based methods remain competitive, often beating MLLMs in specific languages: BGE-M3 reaches 77.19 top‑1 in Finnish, and BM25 is strong in Spanish, Vietnamese, and Czech (Chen et al., 10 Aug 2025).
The paper further compares ColQwen2 (E), trained on original mostly English data, with ColQwen2 (M), trained on the same data plus the 210K multilingual QA resource. The overall average top‑1 score rises from 53.84 to 56.07, with especially large gains in Portuguese: 49.44 → 71.86 and Czech: 52.87 → 61.32. This indicates that multilingual fine-tuning on VisR-Bench data improves retrieval, although gains remain uneven across languages (Chen et al., 10 Aug 2025).
For answer generation, GPT‑4o (all page) is the strongest model, with PNLS ~0.82 average and GAcc ~0.62–0.64 on the English split. Among the listed open models, InternVL2-4B performs best, with average GAcc: 0.58 and PNLS: 0.71 on the English split, while Phi‑4-multimodal and Paligemma2-3B are substantially lower. This indicates that retrieval quality is necessary but not sufficient: even when the correct page is available, structured tables and multilingual pages remain difficult for current open MLLMs (Chen et al., 10 Aug 2025).
6. Significance, limitations, and naming context
VisR-Bench exposes several systematic failure modes. Hard negatives often contain similar pages or figures, and models confuse them because of shared keywords or similar chart style. Table questions remain difficult because current retrievers and MLLMs are not yet table-structure aware. The multilingual results further show uneven robustness across scripts and language families, with especially weak performance on Arabic, Finnish, and Vietnamese (Chen et al., 10 Aug 2025).
The benchmark’s limitations are also explicit. The documents are drawn from web-crawled PDFs, so some domains remain underrepresented; the language set, while broad, does not cover scripts and languages such as Chinese, Hindi, or Russian; the QA pairs are synthetic, generated by GPT‑4o and then filtered; and the supervision is at the page level, not the region level. The paper identifies several future directions: multi-page evidence retrieval, explicit region-level annotations, and table-specific and chart-specific retrieval and reasoning modules (Chen et al., 10 Aug 2025).
Within the broader benchmark landscape, VisR-Bench is specialized. It is not a multi-turn browsing benchmark, a purely visual-native search benchmark, or a natural-image visual reasoning benchmark; it is a benchmark for question-driven multilingual visual retrieval in long documents. The name should also be distinguished from VisRes Bench, which is often written VisR-Bench in the paper’s text but studies visual reasoning in naturalistic settings without contextual language supervision (Törtei et al., 24 Dec 2025). That naming overlap can create confusion, but the actual scopes are substantially different.
Taken as a whole, VisR-Bench establishes a benchmark for multilingual multimodal long-document retrieval in which page-level evidence supervision, figure-, text-, and table-based questions, and visual-dependent queries are all central. This suggests that it functions not only as a scorecard for current retrievers, but also as a diagnostic resource for document-aware multimodal architectures, multilingual fine-tuning, and end-to-end visual RAG research (Chen et al., 10 Aug 2025).