VisBrowse-Bench: Visual-Native Search Benchmark

• VisBrowse-Bench is a benchmark for visual-native search that mandates continuous incorporation of visual evidence alongside text in mixed-modality browsing.
• It employs an expert-guided, multi-stage process with tools like image search, cropping, and reverse-image search to ensure robust multimodal evidence cross-validation.
• Empirical evaluations reveal that even state-of-the-art systems struggle to exceed 50% accuracy, highlighting the challenge of maintaining pixel-level and joint multimodal reasoning.

VisBrowse-Bench is a benchmark for visual-native search in multimodal browsing agents. It targets a failure mode in which browsing systems either reduce visual queries to black-box reverse-image searches or collapse into purely textual search once an entity name is identified. The benchmark therefore makes fine-grained visual evidence structurally indispensable throughout the search-and-reasoning chain. It contains 169 VQA instances covering multiple domains, evaluates visual reasoning capabilities during the search process through multimodal evidence cross-validation via text-image retrieval and joint reasoning, and is accompanied by an agent workflow intended to drive active collection and use of visual information during browsing (Zhang et al., 17 Mar 2026).

1. Definition and problem setting

The central object of evaluation in VisBrowse-Bench is not static image understanding and not webpage understanding in isolation, but a browsing trajectory in which textual and visual evidence must be interleaved. The motivating claim is that real-world web information seeking is inherently multimodal: critical clues often reside in charts, product photos, maps, or posters that cannot be fully described by surrounding text. The benchmark is designed around the stronger requirement that models continue to “think with images” after the search process begins, rather than using images only as an initial prompt and then reverting to text-only reasoning (Zhang et al., 17 Mar 2026).

This design choice distinguishes “visual-native search” from two weaker paradigms. In one, the agent treats image retrieval as a black box and outsources the substantive reasoning burden to reverse-image search. In the other, the agent uses an image only to identify an entity, then proceeds entirely through textual retrieval. VisBrowse-Bench rejects both patterns by requiring multimodal evidence cross-validation and joint reasoning over mixed-modality evidence. A plausible implication is that the benchmark measures not only answer correctness but also whether the evidence pathway remains genuinely multimodal.

The benchmark’s stated objectives are threefold: interleave text and image search, demand pixel-level understanding, and evaluate the agent’s ability to cross-validate and jointly reason over mixed-modality evidence. Examples of the targeted visual competencies include spatial grounding such as “upper left corner,” attribute perception such as colors and textures, and relational parsing such as “the person talking to the statue” (Zhang et al., 17 Mar 2026).

2. Dataset construction and annotation

VisBrowse-Bench was constructed through a multi-stage, expert-guided pipeline. Its instance-creation process begins with seed entity selection: domain experts, specifically two per category, chose seed entities such as films, landmarks, and products that have rich publicly accessible multimodal documentation. From each seed, the authors used recursive evidence chaining: experts located a real-world event or fact, retrieved a relevant image, identified a new entity in that image, and repeated the process. Each chain spans at least three hops and includes at least two distinct visual evidence blocks. Those chains were then fused into a single VQA instance consisting of a bundled text query and initial reference images, such that solving the instance requires visiting web pages, retrieving additional images, cropping regions of interest, and combining textual and visual findings (Zhang et al., 17 Mar 2026).

Quality control is explicit and multi-layered. At the instance level, two experts verify that no single-hop shortcut exists and that both visual and textual evidence are indispensable. For solvability and uniqueness, two additional experts must independently reproduce the correct answer with a documented reasoning trace; questions that fail consistency checks are revised or discarded. This verification regime is consistent with the benchmark’s aim of preventing accidental text-only shortcuts.

The released benchmark contains 169 VQA cases. These are distributed across 7 main categories—Media, Life, Art, Geography, Technology, Sport, and Finance—and 24 subcategories. Each example comprises a textual question with average length 47.7 words, one or more reference images, a concise ground-truth answer with average length 1.5 words, and full expert-authored reasoning traces and evidence locations (Zhang et al., 17 Mar 2026).

3. Formal task structure

The paper formalizes each instance as $Q=(T, I_0)$, where $T$ is the text query and $I_0$ is the set of initial reference images. The agent must produce an answer $\hat y$ by issuing a sequence of tool-invocation actions $a_1,\ldots,a_K$ under the constraint that it must gather at least two novel visual evidence blocks. This constraint is central: it encodes the benchmark’s view that browsing competence should include active acquisition of new visual evidence rather than passive interpretation of an initial image set (Zhang et al., 17 Mar 2026).

The benchmark decomposes this objective into three subtasks. The first is Visual-Native Search, which seeks relevant image-based evidence through sequences of text queries or image crops:

$\arg\max \sum_{t=1}^K \mathrm{Relevance}(I_t, Q).$

The second is Multimodal Evidence Cross-Validation. For each candidate image $i \in \{I_0 \cup I_1 \ldots I_K\}$ and each retrieved textual snippet $s$, the benchmark calls for bidirectional retrieval, summarized as verify_image(i) and verify_text(s). The third is Joint Reasoning, which integrates validated textual and image evidence:

$\hat y = \arg\max_y P(y \mid E_T, E_I),$

where $E_T$ is the set of textual evidence and $T$0 is the set of image evidence.

These definitions make the benchmark more restrictive than conventional multimodal QA. The answer must arise from a search process with explicit evidence acquisition, explicit visual expansion beyond the seed inputs, and explicit cross-modal consistency checks. This suggests that benchmark difficulty is tied not merely to latent knowledge or generic VQA skill, but to the orchestration of retrieval, grounding, and inference over a growing evidence state.

4. Agent workflow and tool interface

To operationalize the benchmark, the paper equips the MLLM with five tools in a closed-loop, LLM-orchestrated framework: text_search(query), image_search(query), reverse_image_search(image_URL), crop_image(image_URL, region), and webpage_visit(URL, subquery). The workflow is iterative. At each turn, the model summarizes current findings and open subquestions, decides which tool to invoke and why, issues the tool call, ingests the tool response, updates its internal state, and repeats until it has sufficient evidence to answer (Zhang et al., 17 Mar 2026).

The intended effect of this workflow is to prevent premature convergence to a text-only chain. Because image search, reverse image search, and region cropping remain available throughout the trajectory, visual operations are not confined to a preprocessing stage. Instead, they are first-class steps in the reasoning chain. The benchmark therefore evaluates whether an agent can sustain mixed-modality evidence gathering rather than merely consume a multimodal prompt.

The use of crop_image is particularly important for the benchmark’s notion of visual competency enforcement. The data were intentionally designed so that visual elements may be ambiguous, overlapping, or subtle, and therefore cannot be faithfully paraphrased at a coarse level. A plausible implication is that successful agents must perform region-sensitive inspection rather than global screenshot interpretation alone.

5. Evaluation protocol and empirical results

VisBrowse-Bench measures overall question-answering performance by accuracy:

$T$1

with $T$2. For retrieval sub-tasks, it also uses precision and recall:

$T$3

Here $T$4 denotes truly relevant retrieved items, $T$5 irrelevant retrieved items, and $T$6 relevant items that were missed (Zhang et al., 17 Mar 2026).

The evaluation covers twelve models, including closed-source systems, an open-source model, and the proprietary o3-Deep-Research system, under three regimes: Direct Answer, +TS using text_search and webpage_visit, and +IS using all five tools. Claude-4.6-Opus is reported as the strongest system, with accuracy improving from 27.2% in Direct Answer mode to 42.6% under +TS and 47.6% under +IS. Kimi-K2.5 reaches 41.4% with full tool access. Open-source Qwen3-VL-235B peaks at 14.2%. The proprietary o3-Deep-Research model, evaluated only in Direct Answer mode, achieves 41.1% (Zhang et al., 17 Mar 2026).

Two conclusions are explicit in the reported results. First, even the best-performing model remains below 50% accuracy. Second, purely textual retrieval yields only modest gains relative to full visual-tool access, which the authors interpret as evidence that many answers hinge on attributes visible only in images. The per-category analysis further reports continued difficulty in domains requiring very fine-grained visual discrimination, specifically Art and Finance. The paper also notes that current models often underuse or misuse cropping and reverse-image search and frequently revert to text-only chains once a single entity name is found.

6. Relation to neighboring benchmarks and terminological ambiguity

VisBrowse-Bench belongs to a larger family of multimodal evaluation suites for web and browsing systems, but its target differs from nearby benchmarks. VisualWebBench is a multimodal benchmark for realistic web pages organized into seven QA-style tasks—Captioning, WebQA, Heading OCR, Element OCR, Element Grounding, Action Prediction, and Action Grounding—and contains 1,534 instances from 139 sites across 12 high-level domains. Its focus is fine-grained webpage perception, comprehension, grounding, and action reasoning, rather than multi-hop evidence chains requiring at least two novel visual evidence blocks (Liu et al., 2024).

BrowseComp-$T$7 evaluates multimodal browsing and deep search in a different direction. It contains 300 questions, decomposes each into 3–6 atomic sub-goals, and introduces process-level assessment through Process Score in addition to final Success Rate. Its emphasis is “Visual + Vertical + Verifiable,” including publicly searchable evidence, multi-level cross-modal reasoning, and expert-validated sub-goals. Compared with VisBrowse-Bench, it places greater formal weight on trajectory-level decomposition and process evaluation (Zhang et al., 13 Feb 2026).

A source of confusion in the literature is nomenclature. One summary describes VisualWebBench as “VisBrowse-Bench (officially VisualWebBench),” and another describes BrowseComp-$T$8 under the label “VisBrowse-Bench.” The title “VisBrowse-Bench” most directly refers to the benchmark for visual-native search introduced in 2026 (Zhang et al., 17 Mar 2026), but adjacent summaries use similar shorthand for other web-focused multimodal benchmarks. This terminological overlap should not obscure the substantive distinctions among webpage understanding, multimodal browsing, and visual-native search.

Placed against broader VLM evaluation, VisBrowse-Bench can also be read as a specialization beyond general instruction-following benchmarks such as VisIT-Bench, which evaluates 70 instruction families and 592 test queries for real-world image-chat use (Bitton et al., 2023). This suggests a progression from general instruction-following, to web-page understanding, to open-world multimodal browsing, and finally to visual-native search in which continued image-grounded evidence acquisition is structurally required.