VDR-Bench: Vision-DeepResearch Benchmark

• VDR-Bench is a multimodal benchmark that defines fact-finding through combined image inspection, entity cropping, and iterative text search.
• It employs a multi-step construction pipeline with manual verification, knowledge graph expansion, and anti-shortcut strategies to minimize text-only guessing.
• Empirical findings show that iterative cropped-image search significantly boosts model performance in complex, real-world visual search scenarios.

Vision-DeepResearch Benchmark (VDR-Bench) is a benchmark for evaluating Vision-DeepResearch systems: multimodal LLM agents that answer complex questions by combining image understanding, visual search over the web, text search or browsing, and multi-step reasoning across both modalities. It is designed as a visual-search-centric benchmark for “joint visual and textual search for fact-finding,” and comprises 2,000 VQA instances constructed to reduce text-only shortcutting, prior-knowledge guessing, and overly idealized whole-image retrieval scenarios (Zeng et al., 2 Feb 2026).

1. Definition and benchmark scope

VDR-Bench targets a setting in which the model is not expected to answer from the image alone. Instead, the intended workflow is to inspect the image, identify salient visual entities, search for those entities visually, use retrieved evidence to launch textual search, aggregate evidence across multiple steps, and then produce a factually grounded answer. In this formulation, the benchmark is neither a conventional static VQA dataset nor a pure retrieval benchmark. It is an evaluation framework for multimodal deep-research behavior under open-web conditions, with image-grounded entity identification as the entry point to broader factual reasoning (Zeng et al., 2 Feb 2026).

A later system paper, "Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal LLMs" (Huang et al., 29 Jan 2026), treats VDR-Bench as a central external benchmark and evaluates on its test-mini split. That paper reinforces the same task interpretation: image-plus-question input, noisy and cluttered visual scenes, multi-turn visual and textual search, and long-horizon evidence aggregation. The benchmark is therefore used not merely as a perception test, but as an instrument for measuring whether a multimodal agent can sustain iterative, agentic fact-finding under realistic search noise (Huang et al., 29 Jan 2026).

2. Motivation and anti-shortcut design

The benchmark is motivated by two stated limitations in prior multimodal search evaluation. First, earlier benchmarks are described as insufficiently visual-search centric: answers may leak through cross-textual cues in the question, through parametric world knowledge already stored in the model, or through text-only retrieval that does not require genuine visual grounding. Second, prior evaluation scenarios are described as overly idealized: on the image-search side, the required information can often be obtained by near-exact matching against the full image, while on the text-search side the query phrasing is often so direct that shallow lookup suffices (Zeng et al., 2 Feb 2026).

The benchmark paper illustrates these problems with several failure modes. One example involves an image of the Norman Tower in Bury St Edmunds, where textual clues such as “early 12th-century gate,” “English town,” and later use as a “belfry” for a church that became a cathedral make the answer recoverable by cross-referencing text alone. Another example asks for the developer of the game shown in an image evidently depicting Grand Theft Auto, making “Rockstar Games” guessable from prior world knowledge. A third example contrasts whole-image search with cropped search on a crowd scene containing a red umbrella with a recognizable logo: whole-image search is distracted by the broad scene, whereas cropped search recovers the Ferrari entity. These cases are used to motivate crop-first, entity-level retrieval and multi-hop textual expansion rather than one-shot whole-image matching (Zeng et al., 2 Feb 2026).

The benchmark’s anti-shortcut strategy is therefore architectural, not merely rhetorical. It relies on manual crop-based visual search during construction, human verification that entities are not trivially recoverable through full-image search alone, knowledge-graph expansion so that text search becomes multi-hop rather than direct lookup, and solvability checks that enforce visual necessity and answer recoverability. The resulting design attempts to make the image necessary, the visual search localized, and the subsequent text search nontrivial (Zeng et al., 2 Feb 2026).

3. Dataset composition and construction pipeline

VDR-Bench contains 2,000 VQA instances spanning 10 visual domains. The domain distribution reported in the benchmark paper is shown below (Zeng et al., 2 Feb 2026).

Domain	Share
Game	15.65%
Sports	15.35%
Sci & Tech	12.1%
Art & Music	10.95%
People	9.05%
Object	9.85%
Architecture	8.5%
Nature	7.9%
Movie	7.95%
Others	2.7%

The instances are described as multi-hop VQA tasks varying in visual complexity, entity density, and reasoning depth. Example answer formats include named entities, numbers, years, descriptive factual phrases, and multi-part factual responses. The paper does not provide full train/dev/test split details in the main text; it notes a testmini set in appendix tables, and the later Vision-DeepResearch system paper confirms use of the test-mini split, but the exact split protocol is not otherwise specified (Zeng et al., 2 Feb 2026, Huang et al., 29 Jan 2026).

The construction pipeline is explicitly multi-stage. It begins with multi-domain image pre-filtering, where low-resolution images are removed and Qwen3-VL-235B-A22B-Instruct is used as an image-filtering model to select high-quality, visually rich scenes containing multiple entities. Annotators then manually extract salient local regions such as objects, logos, landmarks, and individuals, and submit each crop to a web-scale image search engine. Candidate entity names are extracted from titles and captions of retrieved results, filtered by Qwen3-VL-235B-A22B-Instruct, and then confirmed by human validation. For each verified visual entity, Gemini-2.5-Pro synthesizes seed VQA pairs, which are manually reviewed for clarity, answer uniqueness, and dependence on both the visual entity and retrieved context. The pipeline then links each visual entity to an external knowledge graph and performs random walks over related entities—such as founders, cities, years, or affiliated organizations—to generate multi-hop questions. Finally, automatic solvability checks and human filtering remove examples with text-only shortcuts, prior-based guessing, ambiguous wording, or invalid reasoning paths. The paper summarizes the intended retained properties as visual-centric, recoverable answer, image necessary, unambiguous, and not trivial retrieval (Zeng et al., 2 Feb 2026).

4. Task formulation and evaluation protocol

At inference time, the benchmark provides an input image and a question. Depending on the evaluation setting, the agent may answer directly or may use search tools. The main benchmark experiments use three settings: Direct Answer, CIS+TS (Cropped-Image Search plus Text Search), and CIS+TS+MVF (the same toolset augmented with Multi-turn Visual Forcing, the paper’s multi-round cropped-search workflow). In a separate benchmark-analysis section, the paper also reports Whole-Image Search (WIS), Text Search (TS), WIS+TS, and caption-based variants to study shortcut susceptibility (Zeng et al., 2 Feb 2026).

The official metrics are Answer Accuracy and Entity Recall (ER). Answer Accuracy is judged by Qwen3-VL-30B-A3B-Instruct using a judge prompt that follows the template from the Tongyi DeepResearch report. Entity Recall is designed to measure whether the search trajectory actually surfaced the task-relevant entities. For task $i$, let $\mathcal{E}_i = \{e_1, \ldots, e_{n_i}\}$ be the gold entity sequence and let $S_i = \cup_{t=1}^{T} S_t$ be the set of entities extracted from the agent’s search trajectory across $T$ time steps. Using the paper’s notation, trajectory success is defined as

$$S_i = \mathbb{I}\big[\mathrm{LLM}_{\mathrm{eval}}(S_i,\mathcal{E}_i)=1\big].$$

The benchmark explicitly states that ER is not based on exact string matching: the judge can accept semantic equivalence, synonyms, and alternative but valid retrieval paths (Zeng et al., 2 Feb 2026).

Subsequent work uses VDR-Bench to probe the effect of search depth, crop locality, and training regime. On the test-mini split, a pipeline ablation in the Vision-DeepResearch system paper reports the following VDR scores: Direct Answer 4.8, WIS 11.8, WIS+TS 16.0, CIS 15.4, and CIS+TS 37.8. The same paper also reports a data-and-method ablation on VDR in which Qwen3-VL-30B-Instruct scores 20.2, then improves to 24.4 after +16K VQA traj. (SFT), 27.0 after +8K QA traj. (SFT), 33.2 after +6K fuzzy VQA traj. (SFT), and 37.8 after +RL training, indicating that the benchmark is highly sensitive to trajectory supervision, search behavior, and long-horizon optimization rather than to direct-answering ability alone (Huang et al., 29 Jan 2026).

5. Empirical findings and difficulty profile

The benchmark paper’s validation experiments show that direct answering is uniformly weak. Under Direct Answer, the reported overall scores are 8.2 for Gemini 2.5 Pro, 9.5 for GPT-5, 5.6 for Claude-4-Sonnet, 3.8 for Qwen3-VL-30B-A3B-Instruct, and 8.8 for Qwen3-VL-235B-A22B-Instruct. Under CIS+TS, these rise to 16.2, 19.2, 13.2, 17.2, and 21.2 respectively. Under CIS+TS+MVF, the scores become 30.0, 26.6, 20.6, 21.2, and 27.4. The paper highlights the gain for Gemini 2.5 Pro from 16.2 to 30.0 as evidence that multi-round cropped search materially improves realistic visual retrieval (Zeng et al., 2 Feb 2026).

The benchmark-analysis table further supports the anti-shortcut claim. On VDR-Bench, Gemini 2.5 Pro scores 8.2 in Direct Answer, 9.8 in WIS, 5.4 in TS, 10.6 in WIS+TS, 4.6 in Caption Direct Answer, and 5.2 in Caption+TS. Qwen3-VL-30B scores 3.8, 5.2, 5.8, 6.0, 4.6, and 7.4 under the same settings. The paper contrasts these values with higher shortcut-prone scores on earlier benchmarks and argues that VDR-Bench is much harder to solve through caption-only or text-only routes (Zeng et al., 2 Feb 2026).

Entity Recall is strongly associated with final answer correctness. The benchmark paper reports a strong positive relationship between answer accuracy and entity recall, and shows that MVF tends to improve both. In appendix testmini results, overall ER under CIS+TS is 13.0 for Gemini, 14.0 for GPT-5, 8.0 for Claude, 13.0 for Qwen3-VL-30B, 17.6 for Qwen3-VL-235B, 34.6 for Vision-DeepResearch-8B, and 45.0 for Vision-DeepResearch-30B; under CIS+TS+MVF, the corresponding ER values are 32.0, 24.4, 19.2, 20.0, and 28.4 for the first five models. In the later system paper, Vision-DeepResearch-8B scores 29.2 on VDR and Vision-DeepResearch-30B-A3B scores 37.8 under CIS+TS, further suggesting that specialized multimodal deep-research training yields large gains on this benchmark (Zeng et al., 2 Feb 2026, Huang et al., 29 Jan 2026).

6. Position within the multimodal deep-research benchmark landscape

VDR-Bench occupies a specific niche among multimodal search and deep-research benchmarks. It shares with later comparisons the properties of multimodal queries, multi-turn interaction, public search information, expert validation, and thinking with image. However, in the comparison table of "VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agents" (Zhang et al., 17 Mar 2026), VDR-Bench is marked no for visual evidence and cross-image reasoning, whereas VisBrowse-Bench is marked yes on both. In that framing, VDR-Bench emphasizes visual query understanding and joint visual-textual search, but does not explicitly require visual evidence gathering across multiple webpages or reasoning across multiple retrieved images in the search path (Zhang et al., 17 Mar 2026).

This distinction is methodologically significant. According to the VisBrowse-Bench paper, some benchmarks such as MMSearch-Plus and VDR-Bench may require some initial visual understanding of the query image, but the subsequent search trajectory often becomes text-only after the image is identified. By contrast, VisBrowse-Bench is designed to force agents to seek out additional visual evidence on webpages and reason across multiple images. A plausible implication is that VDR-Bench represents an earlier but still important stage in the evolution of vision-centered deep-research evaluation: it makes visual search central at the entry point of the reasoning chain, while later benchmarks push visual evidence deeper into the trajectory itself (Zhang et al., 17 Mar 2026).

Relative to report-level deep-research benchmarks, VDR-Bench also occupies a different part of the evaluation spectrum. "DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Report" (Li et al., 13 Jan 2026) and "Understanding DeepResearch via Reports" (Fan et al., 9 Oct 2025) are report-centric and not presented as vision or multimodal benchmarks; the latter explicitly evaluates textual content only, with images and charts reserved for future work. A plausible implication is that VDR-Bench is best understood as a search-centric, short-answer benchmark for multimodal fact-finding, rather than a report-generation benchmark for end-to-end research systems (Li et al., 13 Jan 2026, Fan et al., 9 Oct 2025).

7. Limitations, interpretive cautions, and benchmark significance

Several limitations are explicit or strongly implied in the benchmark paper. Both Answer Accuracy and Entity Recall depend on LLM judges, which introduces evaluator dependence and possible prompt sensitivity. The exact retrieval API details, crop policy, and fixed interaction budget are not fully specified in the provided benchmark text. At 2,000 instances, the benchmark is substantial for expert-curated multimodal deep research, but still modest relative to web-scale static VQA corpora. The crop-based, knowledge-graph-expanded construction pipeline may also emphasize tasks that naturally fit entity identification followed by relational traversal (Zeng et al., 2 Feb 2026).

External comparisons add a further caution. In the VisBrowse-Bench framing, VDR-Bench does not explicitly require visual evidence or cross-image reasoning within the search path, which limits its coverage of visual-native browsing behavior. Conversely, this limitation is also part of its historical significance: the benchmark exposed a gap between strong generic VQA or world knowledge and realistic multimodal retrieval, and it provided one of the clearest empirical arguments that entity-level cropped search is a major bottleneck in Vision-DeepResearch systems (Zhang et al., 17 Mar 2026, Zeng et al., 2 Feb 2026).

The benchmark’s main significance lies in three contributions. First, it reframed multimodal QA evaluation around visual-search-centric fact-finding rather than image-conditioned answering alone. Second, it introduced a construction pipeline that explicitly suppresses text-only shortcuts and trivial whole-image matching. Third, it yielded a practical design lesson: multi-round cropped-image search materially improves performance. Later training results on VDR-Bench suggest that gains come not simply from larger base models, but from better internalized deep-research behavior—trajectory supervision, fuzzy multi-hop VQA synthesis, and RL over long-horizon tool use. This suggests that VDR-Bench functions not only as a benchmark, but as a diagnostic instrument for whether multimodal agents have learned to search, localize, and reason rather than merely to guess (Zeng et al., 2 Feb 2026, Huang et al., 29 Jan 2026).