MMSearchVQA Benchmarks Overview

• MMSearchVQA Benchmarks are comprehensive testbeds that evaluate multimodal language models on integrating text and visual cues in multi-step reasoning tasks.
• They employ iterative tool use, standardized APIs, and self-refinement to overcome limitations of static, unimodal retrieval strategies.
• Key metrics such as accuracy, IoU, and checklist completion highlight both the progress and challenges in achieving robust, evidence-driven performance.

Multimodal Search VQA (MMSearchVQA) benchmarks constitute a rigorous testbed for the evaluation of large multimodal LLMs (MLLMs) as dynamic information-seeking agents capable of integrating textual and visual retrieval, fine-grained reasoning, iterative tool use, and evidence validation in real-world tasks. These benchmarks are engineered to transcend the limitations of prior vision-language evaluation, focusing on scenarios where neither unimodal nor static retrieval strategies suffice and where text and image cues must be synthesized across multiple steps and modalities. MMSearchVQA benchmarks include MMSearch (Jiang et al., 2024), MMSearch-Plus (Tao et al., 29 Aug 2025), MM-BrowseComp (Li et al., 14 Aug 2025), Vision-DeepResearch (VDR-Bench) (Zeng et al., 2 Feb 2026), and DeepMMSearchVQA (Narayan et al., 14 Oct 2025), among others. This entry surveys their core methodologies, dataset design, evaluation protocols, key findings, and the evolving research landscape.

1. Benchmark Motivation and Problem Setting

MMSearchVQA benchmarks arise from the observation that previous vision-language and web-browsing datasets are either predominantly textual or solvable via high-recall, single-step image or text queries, lacking the depth needed to probe genuine multimodal reasoning. While AI search engines and browsing agents have achieved notable results in text-only frameworks, modern web and knowledge domains are inherently multimodal: web pages contain interleaved text, images, diagrams, infographics, and video; many user queries require factual inference that traverses visual recognition and cross-source verification; and dynamic content change further complicates evidence assembly (Li et al., 14 Aug 2025, Tao et al., 29 Aug 2025, Zeng et al., 2 Feb 2026).

The MMSearchVQA setting formalizes an agent that, upon receiving a multimodal query (text + optional image), must iteratively interact with external tools (web/image search, cropping, retrieval APIs), extract and propagate intermediate clues (entities, regions), and output precise answers, often including reasoning trace or multi-step lookup chains (Tao et al., 29 Aug 2025, Narayan et al., 14 Oct 2025). Success on these benchmarks indicates an agent's ability to "think with images," decompose questions, plan under uncertainty, and maintain verifiable, cross-modal evidence chains.

2. Dataset Construction, Curation, and Taxonomy

MMSearchVQA benchmarks employ diverse, expert-informed construction pipelines designed to defeat shortcutting and force true multimodal reasoning:

• MMSearch (Jiang et al., 2024): 300 search-style questions (57% multimodal), balanced across 14 subfields, each requiring agents to requery, rerank, and summarize over text and images. Data collection spans recent news and rare-knowledge domains outside model pretraining, with strict answer verification.
• MMSearch-Plus (Tao et al., 29 Aug 2025): 311 tasks, explicitly crafted (via "Spatial-Temporal Extrapolation") so that the answer depends on multiple weak, localized visual cues (micro-text, part-level logos, broadcast overlays) and nontrivial cross-step inference. Annotators seed spatial and temporal clues and require off-image fact retrieval (e.g., event dates/venues).
• MM-BrowseComp (Li et al., 14 Aug 2025): 224 hand-crafted multimodal browsing queries spanning 22 subtasks; each includes a short answer and an "irreducible reasoning checklist" (3–5 modality-tagged steps), ensuring no purely textual solution is possible and requiring tool calls over images, text, and videos.
• VDR-Bench (Zeng et al., 2 Feb 2026): 2,000 VQA-style questions across 10 domains, systematically filtered and curated to preclude cross-textual leakage, enforce visual necessity (mandatory cropping), and introduce multi-hop knowledge graph trails.
• DeepMMSearchVQA (Narayan et al., 14 Oct 2025): 10,000 search-intensive conversations built automatically from InfoSeek pairs, filtered for answer reliability, and balanced for knowledge categories and search requirements. Tool calls (text search, image/crop search, self-refinement) are explicitly modeled and iteratively annotated.

Key design steps in these benchmarks include manual annotation or automated filtering for visual richness, salience of entities, forced multi-step dependencies, and the elimination of full-image shortcutting. Question styles range from short-answer (mean 1.9–3.8 tokens) to multi-turn dialog, with wide domain and modality (image, text, video) coverage.

3. Agent Frameworks, Tool Interfaces, and Workflow Protocols

All MMSearchVQA benchmarks implement agent frameworks supporting dynamic, multi-stage tool use:

• Standardized Tool APIs: Functions for text and image search, sub-image cropping, and retrieval of top-K summaries or URLs (e.g., SerpApi in MMSearch-Plus (Tao et al., 29 Aug 2025)).
• Iterative Search and Reasoning: Agents must chain search and refinement steps, using retrieved entities to enhance future queries, isolate visual regions (via bounding-boxes or attention maps), and validate provenance under noisy/near-duplicate retrieval (Zeng et al., 2 Feb 2026, Narayan et al., 14 Oct 2025).
• Cropping: Cropped image search is mandatory in VDR-Bench and DeepMMSearchVQA; models must localize salient cues and avoid relying on full-image retrieval alone.
• Self-Reflection and Correction: Adaptive query reformulation, multi-turn tool use, and self-correction are stressed in DeepMMSearchVQA and MMSearch-Plus (Narayan et al., 14 Oct 2025, Tao et al., 29 Aug 2025).
• Reproducibility: Benchmarks cache all search API outputs and extracted web contents, standardizing tool access for any model or agent to plug into the framework (Tao et al., 29 Aug 2025).
• Agent Context: Agents retain access to prior tool outputs, conversation, and chain-of-thought steps throughout each benchmark instance (Jiang et al., 2024).

A generic pseudocode abstraction of an MMSearchVQA agent involves looping over crop selection, search (text and visual), entity extraction, question augmentation, and answer derivation, adapting focus based on retrieved evidence and model attention maps (Zeng et al., 2 Feb 2026).

4. Evaluation Metrics, Protocols, and Baselines

All MMSearchVQA benchmarks prioritize strict, multi-dimensional evaluation to measure not just final answer accuracy but reasoning fidelity and retrieval quality.

Canonical metrics include:

• Accuracy ($\mathrm{Acc}$, OA):

$$\mathrm{Acc} = \frac{\#\text{correct predictions}}{\#\text{total questions}}$$

Computed by direct match or using an LLM-as-judge (paraphrase-robust) (Zeng et al., 2 Feb 2026, Narayan et al., 14 Oct 2025).

• Checklist Completion (MM-BrowseComp): Fraction of "irreducible" checklist steps correctly executed; separated by modality (text, image, video) (Li et al., 14 Aug 2025).
• Entity Recall (VDR-Bench): LLM-judged indicator that all gold intermediate entities were discovered during a search trajectory.
• Bounding-box Grounding (IoU):

$$\mathrm{IoU} = \frac{|B_{\mathrm{pred}} \cap B_{\mathrm{gt}}|}{|B_{\mathrm{pred}} \cup B_{\mathrm{gt}}|}$$

For evaluating cropped region proposals (Tao et al., 29 Aug 2025).

• Component F1, Rerank Accuracy, Requery Quality (MMSearch): Modular scoring for sub-tasks, including ROUGE/BLEU for query rewriting, rerank correctness, and answer F1 (Jiang et al., 2024).
• Top-K Accuracy: For multi-choice settings; $k$-guess reporting (Yang et al., 29 May 2025).

Baselines include human performance, random guessing, non-search (direct answer), single-round search, and full multi-turn agent rollouts. Closed-source models (e.g., GPT-4o, o3, Gemini-2.5-Pro) and open-source systems (e.g., Qwen, LLaVA, InternVL) are all evaluated.

Selected baseline benchmark results:

Benchmark	Best Model / Setting	Accuracy (or OA)	Notable Gaps
MMSearch-Plus	o3, full rollout	36.0%	Qwen: 6.9% (20 rounds), 0% (no search) (Tao et al., 29 Aug 2025)
MM-BrowseComp	o3 (tools)	29.02%	Image steps ~10–15 pp harder than text (Li et al., 14 Aug 2025)
VDR-Bench	Gemini-2.5-Pro (+MVF)	30.0%	<10% no-search; open-source: 21–27% (Zeng et al., 2 Feb 2026)
MMSearch	GPT-4o (end-to-end)	60.4% (Humans: 68.2%)	Open-source: up to 49.1% (Jiang et al., 2024)
MMSI-Bench	o3, multi-image spatial VQA	41% (Humans: 97%)	Open-source: 30.7%, random: 25% (Yang et al., 29 May 2025)
DeepMMSearchVQA	DeepMMSearch-R1-7B (RL)	57.13% (avg, 6 datasets)	SFT: 56.23%, non-search: 50.56% (Narayan et al., 14 Oct 2025)

This demonstrates the persistent gap between current frontier models and the upper bound, especially on tasks requiring robust localization, multi-step reasoning, and visual-textual integration.

5. Error Analysis and Model Failure Modes

Extensive error analysis reveals consistent patterns across all MMSearchVQA benchmarks:

• Retrieval Failures: Over-specific or ineffective queries lead to no relevant information retrieved (>50% of errors in MMSearch-Plus) (Tao et al., 29 Aug 2025).
• Reasoning Gaps: Hallucination (unjustified visual/textual attributions without evidence), incomplete extraction of necessary facts, or latching onto distractor entities.
• Verification/Provenance: Failure to cross-check retrieved information, leading to wrong-event or spurious-answer errors.
• Modality-specific Deficits: Image-based and video-based reasoning steps are systematically harder than text, with performance lagging by 10–15 percentage points (MM-BrowseComp) (Li et al., 14 Aug 2025).
• Workflow Pathologies: In unconstrained rollouts, overuse of multi-hop toolchains can introduce noise and distractors faster than they accumulate evidence; high tool-calling frequency is not consistently beneficial (Tao et al., 29 Aug 2025, Narayan et al., 14 Oct 2025).
• Grounding and Locality: Uncontrolled sub-image cropping may degrade performance; precise crop selection remains nontrivial (Zeng et al., 2 Feb 2026).
• Format and Instruction Following: Open-source systems show higher rates of format/instruction violations, impacting measurement (Jiang et al., 2024).

Error taxonomies such as in MMSearch-Plus cluster model failures into information not found, hallucination, non-extraction, non-verification, and other categories. Automated error analysis (MMSI-Bench) further decomposes into grounding, overlap-matching/scene-reconstruction, situation-transformation, and spatial-logic errors (Yang et al., 29 May 2025).

6. Emerging Principles and Future Research Trajectories

MMSearchVQA benchmarks have catalyzed several crucial methodological insights and pointed out open problems:

• Necessity of Cropped, Localized Search: Force models to ground queries and disambiguate entities with region proposals, in order to obviate trivial full-image recall (Zeng et al., 2 Feb 2026, Narayan et al., 14 Oct 2025).
• Integrated Reasoning and Tool Use: Agents must interleave retrieval, reasoning, and prompt/trajectory refinement, benefiting from explicit self-reflection and self-correction—and avoid rigid, rule-driven workflows (Narayan et al., 14 Oct 2025, Tao et al., 29 Aug 2025).
• Adaptive Rollout Policies: Future models should learn, possibly via reinforcement learning (e.g., GRPO), how many rounds to search, when to crop, and which search modality to prioritize (Tao et al., 29 Aug 2025, Narayan et al., 14 Oct 2025).
• Error Diagnosis and Training Robustness: Automated error pipelines surface recurring model weaknesses that can be directly targeted for model improvement and data augmentation (Yang et al., 29 May 2025).
• Scaling Test-time Computation vs. Model Size: Best-of-N or ensemble inference can yield gains exceeding those from increasing model parameter count, suggesting deployment efficiency tradeoffs (Jiang et al., 2024).
• Best Practices for Dataset/Workflow Extension: Modularizing search, entity extraction, and prompting components allows adaptation to new knowledge domains, integration of structured knowledge bases, and joint retrieval-generation for low-resource or dynamic settings (Zeng et al., 2 Feb 2026).

Open research directions include learning end-to-end crop selection, integrating explicit uncertainty and provenance signals, extending to video and 3D retrieval, and benchmarking over continuously updating news and hybrid modalities.

7. Relationship to Broader VQA and Multimodal Reasoning Benchmarks

Unlike traditional VQA datasets, which often feature single-image, closed-context, or synthetic questions, MMSearchVQA benchmarks require cross-source, multi-step, multimodal information seeking in open-world conditions. They thus complement spatial intelligence benchmarks like MMSI-Bench (Yang et al., 29 May 2025)—which probe multi-image spatial logic—and highlight that naively scaling model capacity or prompt engineering is insufficient for robust, evidence-driven multimodal search and reasoning. MMSearchVQA challenges have become the new standard for evaluating retrieval-augmented MLLMs in real-world, high-variance, evidence-grounded question answering.