---
title: 'VDR-Bench: Vision-DeepResearch Benchmark'
url: https://www.emergentmind.com/topics/vision-deepresearch-benchmark-vdr-bench
type: topic
---

# VDR-Bench: Vision-DeepResearch Benchmark

Vision-DeepResearch Benchmark (VDR-Bench) is a benchmark for evaluating Vision-DeepResearch systems: multimodal large language model agents that answer complex questions by combining image understanding, visual search over the web, text search or browsing, and multi-step reasoning across both modalities. It is designed as a visual-search-centric benchmark for “joint visual and textual search for fact-finding,” and comprises 2,000 VQA instances constructed to reduce text-only shortcutting, prior-knowledge guessing, and overly idealized whole-image retrieval scenarios [2602.02185].

## 1. Definition and benchmark scope

VDR-Bench targets a setting in which the model is not expected to answer from the image alone. Instead, the intended workflow is to inspect the image, identify salient visual entities, search for those entities visually, use retrieved evidence to launch textual search, aggregate evidence across multiple steps, and then produce a factually grounded answer. In this formulation, the benchmark is neither a conventional static VQA dataset nor a pure retrieval benchmark. It is an evaluation framework for multimodal deep-research behavior under open-web conditions, with image-grounded entity identification as the entry point to broader factual reasoning [2602.02185].

A later system paper, "Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Models" [2601.22060], treats VDR-Bench as a central external benchmark and evaluates on its **test-mini split**. That paper reinforces the same task interpretation: image-plus-question input, noisy and cluttered visual scenes, multi-turn visual and textual search, and long-horizon evidence aggregation. The benchmark is therefore used not merely as a perception test, but as an instrument for measuring whether a multimodal agent can sustain iterative, agentic fact-finding under realistic search noise [2601.22060].

## 2. Motivation and anti-shortcut design

The benchmark is motivated by two stated limitations in prior multimodal search evaluation. First, earlier benchmarks are described as insufficiently visual-search centric: answers may leak through cross-textual cues in the question, through parametric world knowledge already stored in the model, or through text-only retrieval that does not require genuine visual grounding. Second, prior evaluation scenarios are described as overly idealized: on the image-search side, the required information can often be obtained by near-exact matching against the full image, while on the text-search side the query phrasing is often so direct that shallow lookup suffices [2602.02185].

The benchmark paper illustrates these problems with several failure modes. One example involves an image of the Norman Tower in Bury St Edmunds, where textual clues such as “early 12th-century gate,” “English town,” and later use as a “belfry” for a church that became a cathedral make the answer recoverable by cross-referencing text alone. Another example asks for the developer of the game shown in an image evidently depicting *Grand Theft Auto*, making “Rockstar Games” guessable from prior world knowledge. A third example contrasts whole-image search with cropped search on a crowd scene containing a red umbrella with a recognizable logo: whole-image search is distracted by the broad scene, whereas cropped search recovers the Ferrari entity. These cases are used to motivate crop-first, entity-level retrieval and multi-hop textual expansion rather than one-shot whole-image matching [2602.02185].

The benchmark’s anti-shortcut strategy is therefore architectural, not merely rhetorical. It relies on manual crop-based visual search during construction, human verification that entities are not trivially recoverable through full-image search alone, knowledge-graph expansion so that text search becomes multi-hop rather than direct lookup, and solvability checks that enforce visual necessity and answer recoverability. The resulting design attempts to make the image necessary, the visual search localized, and the subsequent text search nontrivial [2602.02185].

## 3. Dataset composition and construction pipeline

VDR-Bench contains **2,000 VQA instances** spanning **10 visual domains**. The domain distribution reported in the benchmark paper is shown below [2602.02185].

| Domain | Share |
|---|---:|
| Game | 15.65% |
| Sports | 15.35% |
| Sci & Tech | 12.1% |
| Art & Music | 10.95% |
| People | 9.05% |
| Object | 9.85% |
| Architecture | 8.5% |
| Nature | 7.9% |
| Movie | 7.95% |
| Others | 2.7% |

The instances are described as multi-hop VQA tasks varying in visual complexity, entity density, and reasoning depth. Example answer formats include named entities, numbers, years, descriptive factual phrases, and multi-part factual responses. The paper does not provide full train/dev/test split details in the main text; it notes a **testmini** set in appendix tables, and the later Vision-DeepResearch system paper confirms use of the **test-mini split**, but the exact split protocol is not otherwise specified [2602.02185][2601.22060].

The construction pipeline is explicitly multi-stage. It begins with multi-domain image pre-filtering, where low-resolution images are removed and **Qwen3-VL-235B-A22B-Instruct** is used as an image-filtering model to select high-quality, visually rich scenes containing multiple entities. Annotators then manually extract salient local regions such as objects, logos, landmarks, and individuals, and submit each crop to a web-scale image search engine. Candidate entity names are extracted from titles and captions of retrieved results, filtered by **Qwen3-VL-235B-A22B-Instruct**, and then confirmed by human validation. For each verified visual entity, **Gemini-2.5-Pro** synthesizes seed VQA pairs, which are manually reviewed for clarity, answer uniqueness, and dependence on both the visual entity and retrieved context. The pipeline then links each visual entity to an external knowledge graph and performs random walks over related entities—such as founders, cities, years, or affiliated organizations—to generate multi-hop questions. Finally, automatic solvability checks and human filtering remove examples with text-only shortcuts, prior-based guessing, ambiguous wording, or invalid reasoning paths. The paper summarizes the intended retained properties as **visual-centric**, **recoverable answer**, **image necessary**, **unambiguous**, and **not trivial retrieval** [2602.02185].

## 4. Task formulation and evaluation protocol

At inference time, the benchmark provides an **input image** and a **question**. Depending on the evaluation setting, the agent may answer directly or may use search tools. The main benchmark experiments use three settings: **Direct Answer**, **CIS+TS** (Cropped-Image Search plus Text Search), and **CIS+TS+MVF** (the same toolset augmented with Multi-turn Visual Forcing, the paper’s multi-round cropped-search workflow). In a separate benchmark-analysis section, the paper also reports **Whole-Image Search (WIS)**, **Text Search (TS)**, **WIS+TS**, and caption-based variants to study shortcut susceptibility [2602.02185].

The official metrics are **Answer Accuracy** and **Entity Recall (ER)**. Answer Accuracy is judged by **Qwen3-VL-30B-A3B-Instruct** using a judge prompt that follows the template from the Tongyi DeepResearch report. Entity Recall is designed to measure whether the search trajectory actually surfaced the task-relevant entities. For task \(i\), let \(\mathcal{E}_i = \{e_1, \ldots, e_{n_i}\}\) be the gold entity sequence and let \(S_i = \cup_{t=1}^{T} S_t\) be the set of entities extracted from the agent’s search trajectory across \(T\) time steps. Using the paper’s notation, trajectory success is defined as
$$
S_i = \mathbb{I}\big[\mathrm{LLM}_{\mathrm{eval}}(S_i,\mathcal{E}_i)=1\big].
$$
The benchmark explicitly states that ER is not based on exact string matching: the judge can accept semantic equivalence, synonyms, and alternative but valid retrieval paths [2602.02185].

Subsequent work uses VDR-Bench to probe the effect of search depth, crop locality, and training regime. On the **test-mini split**, a pipeline ablation in the Vision-DeepResearch system paper reports the following VDR scores: **Direct Answer 4.8**, **WIS 11.8**, **WIS+TS 16.0**, **CIS 15.4**, and **CIS+TS 37.8**. The same paper also reports a data-and-method ablation on VDR in which **Qwen3-VL-30B-Instruct** scores **20.2**, then improves to **24.4** after **+16K VQA traj. (SFT)**, **27.0** after **+8K QA traj. (SFT)**, **33.2** after **+6K fuzzy VQA traj. (SFT)**, and **37.8** after **+RL training**, indicating that the benchmark is highly sensitive to trajectory supervision, search behavior, and long-horizon optimization rather than to direct-answering ability alone [2601.22060].

## 5. Empirical findings and difficulty profile

The benchmark paper’s validation experiments show that direct answering is uniformly weak. Under **Direct Answer**, the reported overall scores are **8.2** for **Gemini 2.5 Pro**, **9.5** for **GPT-5**, **5.6** for **Claude-4-Sonnet**, **3.8** for **Qwen3-VL-30B-A3B-Instruct**, and **8.8** for **Qwen3-VL-235B-A22B-Instruct**. Under **CIS+TS**, these rise to **16.2**, **19.2**, **13.2**, **17.2**, and **21.2** respectively. Under **CIS+TS+MVF**, the scores become **30.0**, **26.6**, **20.6**, **21.2**, and **27.4**. The paper highlights the gain for Gemini 2.5 Pro from **16.2** to **30.0** as evidence that multi-round cropped search materially improves realistic visual retrieval [2602.02185].

The benchmark-analysis table further supports the anti-shortcut claim. On VDR-Bench, **Gemini 2.5 Pro** scores **8.2** in Direct Answer, **9.8** in WIS, **5.4** in TS, **10.6** in WIS+TS, **4.6** in Caption Direct Answer, and **5.2** in Caption+TS. **Qwen3-VL-30B** scores **3.8**, **5.2**, **5.8**, **6.0**, **4.6**, and **7.4** under the same settings. The paper contrasts these values with higher shortcut-prone scores on earlier benchmarks and argues that VDR-Bench is much harder to solve through caption-only or text-only routes [2602.02185].

Entity Recall is strongly associated with final answer correctness. The benchmark paper reports a strong positive relationship between answer accuracy and entity recall, and shows that MVF tends to improve both. In appendix **testmini** results, overall ER under **CIS+TS** is **13.0** for Gemini, **14.0** for GPT-5, **8.0** for Claude, **13.0** for Qwen3-VL-30B, **17.6** for Qwen3-VL-235B, **34.6** for **Vision-DeepResearch-8B**, and **45.0** for **Vision-DeepResearch-30B**; under **CIS+TS+MVF**, the corresponding ER values are **32.0**, **24.4**, **19.2**, **20.0**, and **28.4** for the first five models. In the later system paper, **Vision-DeepResearch-8B** scores **29.2** on VDR and **Vision-DeepResearch-30B-A3B** scores **37.8** under **CIS+TS**, further suggesting that specialized multimodal deep-research training yields large gains on this benchmark [2602.02185][2601.22060].

## 6. Position within the multimodal deep-research benchmark landscape

VDR-Bench occupies a specific niche among multimodal search and deep-research benchmarks. It shares with later comparisons the properties of **multimodal queries**, **multi-turn interaction**, **public search information**, **expert validation**, and **thinking with image**. However, in the comparison table of "VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agents" [2603.16289], VDR-Bench is marked **no** for **visual evidence** and **cross-image reasoning**, whereas VisBrowse-Bench is marked **yes** on both. In that framing, VDR-Bench emphasizes visual query understanding and joint visual-textual search, but does not explicitly require visual evidence gathering across multiple webpages or reasoning across multiple retrieved images in the search path [2603.16289].

This distinction is methodologically significant. According to the VisBrowse-Bench paper, some benchmarks such as **MMSearch-Plus** and **VDR-Bench** may require some initial visual understanding of the query image, but the subsequent search trajectory often becomes text-only after the image is identified. By contrast, VisBrowse-Bench is designed to force agents to seek out additional visual evidence on webpages and reason across multiple images. A plausible implication is that VDR-Bench represents an earlier but still important stage in the evolution of vision-centered deep-research evaluation: it makes visual search central at the entry point of the reasoning chain, while later benchmarks push visual evidence deeper into the trajectory itself [2603.16289].

Relative to report-level deep-research benchmarks, VDR-Bench also occupies a different part of the evaluation spectrum. "DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Report" [2601.08536] and "Understanding DeepResearch via Reports" [2510.07861] are report-centric and not presented as vision or multimodal benchmarks; the latter explicitly evaluates textual content only, with images and charts reserved for future work. A plausible implication is that VDR-Bench is best understood as a search-centric, short-answer benchmark for multimodal fact-finding, rather than a report-generation benchmark for end-to-end research systems [2601.08536][2510.07861].

## 7. Limitations, interpretive cautions, and benchmark significance

Several limitations are explicit or strongly implied in the benchmark paper. Both Answer Accuracy and Entity Recall depend on LLM judges, which introduces evaluator dependence and possible prompt sensitivity. The exact retrieval API details, crop policy, and fixed interaction budget are not fully specified in the provided benchmark text. At **2,000 instances**, the benchmark is substantial for expert-curated multimodal deep research, but still modest relative to web-scale static VQA corpora. The crop-based, knowledge-graph-expanded construction pipeline may also emphasize tasks that naturally fit entity identification followed by relational traversal [2602.02185].

External comparisons add a further caution. In the VisBrowse-Bench framing, VDR-Bench does not explicitly require **visual evidence** or **cross-image reasoning** within the search path, which limits its coverage of visual-native browsing behavior. Conversely, this limitation is also part of its historical significance: the benchmark exposed a gap between strong generic VQA or world knowledge and realistic multimodal retrieval, and it provided one of the clearest empirical arguments that entity-level cropped search is a major bottleneck in Vision-DeepResearch systems [2603.16289][2602.02185].

The benchmark’s main significance lies in three contributions. First, it reframed multimodal QA evaluation around **visual-search-centric** fact-finding rather than image-conditioned answering alone. Second, it introduced a construction pipeline that explicitly suppresses text-only shortcuts and trivial whole-image matching. Third, it yielded a practical design lesson: multi-round cropped-image search materially improves performance. Later training results on VDR-Bench suggest that gains come not simply from larger base models, but from better internalized deep-research behavior—trajectory supervision, fuzzy multi-hop VQA synthesis, and RL over long-horizon tool use. This suggests that VDR-Bench functions not only as a benchmark, but as a diagnostic instrument for whether multimodal agents have learned to search, localize, and reason rather than merely to guess [2602.02185][2601.22060].

Source: https://www.emergentmind.com/topics/vision-deepresearch-benchmark-vdr-bench