MMSearch-Plus: Multimodal Browsing Benchmark

• MMSearch-Plus is a provenance-aware benchmark comprising 311 multimodal tasks that demand extraction of weak visual cues and cross-modal reasoning.
• The dataset employs Spatial-Temporal Extrapolation, leveraging detailed visual signals and temporal traces to infer out-of-image facts.
• Empirical results show strong agents score moderately, revealing challenges in iterative search control, evidence verification, and long-horizon planning.

Searching arXiv for MMSearch-Plus and closely related multimodal search-agent papers to ground the article in the current literature. MMSearch-Plus is a benchmark of 311 tasks for multimodal browsing agents. It was introduced as a provenance-aware benchmark that highly demand multimodal understanding while preserving the difficulty profile of strong text-only browsing suites. Each item is constructed to contain multiple weak, localized visual signals that must be extracted, propagated through iterative text-image search, and cross-validated under retrieval noise before answering. In this formulation, the target is not merely image-conditioned retrieval, but long-horizon, provenance-sensitive reasoning over multimodal webpages, with answers that are out-of-image facts such as events, dates, venues, schedules, and identities (Tao et al., 29 Aug 2025).

1. Definition and relation to earlier multimodal search benchmarks

MMSearch-Plus was proposed in response to a specific limitation in prior multimodal browsing settings: many tasks could be solved by shallow, fixed workflows that lean on high-recall image search and nearby text, masking the genuinely multimodal challenges of fine-grained visual reasoning, provenance verification, and long-horizon tool use. The benchmark therefore shifts the problem from single-entity anchoring toward iterative extraction and propagation of sparse visual evidence. It stresses open-domain search and browsing where the agent must collect and reason over both textual and visual evidence to reach verifiable answers (Tao et al., 29 Aug 2025).

This design is best understood against the earlier MMSearch benchmark. MMSearch introduced a general pipeline, MMSearch-Engine, and a benchmark of 300 manually collected instances spanning 14 subfields. Its evaluation decomposed multimodal search into requery, rerank, summarization, and an end-to-end task, with a final aggregate score

$$S_{\mathrm{final}} = 0.75\cdot S_{\mathrm{e2e}} + 0.05\cdot S_{\mathrm{req}} + 0.10\cdot S_{\mathrm{rer}} + 0.10\cdot S_{\mathrm{sum}}.$$

MMSearch-Plus retains the open-world search setting but is explicitly designed to avoid single-entity anchoring and nearby-text shortcuts by requiring multiple weak, localized cues and out-of-image extrapolation (Jiang et al., 2024).

A central distinction is that MMSearch-Plus compels the agent to think with images. The benchmark description emphasizes fine-grained visual reasoning over micro-text, part-level appearance, layouts, signage, and temporal traces, followed by cross-validation under noisy retrieval. This is a stricter requirement than simply using an image to trigger a high-recall search hit and then reading surrounding text (Tao et al., 29 Aug 2025).

2. Dataset design and curation methodology

The dataset contains 311 tasks spanning eight primary categories. It is balanced across domains such as Geography, Sports, Academic research, Film & TV, Technology, Video games, Vlog, and Music. Each item includes at least one image, with average images per entry approximately 1.418, average words in the question approximately 12.273, and average words in the answer approximately 3.840. Most answers are short-form (Tao et al., 29 Aug 2025).

Category	Count	Share
Geography	64	20.6%
Sports	54	17.4%
Academic research	50	16.1%
Film & TV	40	12.9%
Technology	36	11.6%
Video games	31	10.0%
Vlog	19	6.1%
Music	17	5.5%

Its curation procedure is called Spatial-Temporal Extrapolation. The goal is to seed questions whose answers require extrapolating from spatial cues and temporal traces to out-of-image facts. Spatial cues include micro-text, part-level appearance, layouts, and signage or branding. Temporal traces include broadcast overlays, seasonal context, and historical rosters or schedules. The intended outputs include event identity, date or time, venue, match score, episode timestamp, and paper metadata (Tao et al., 29 Aug 2025).

The construction process proceeds through source collection, keyframe or figure selection, seeding questions with spatial or temporal relations, extrapolation to event metadata, and adversarial filtering. Sources include YouTube, Bilibili, and arXiv, with both videos and papers treated as events. For papers, figures or tables are extracted by screenshot when they challenge perception or depict less familiar entities requiring external retrieval. Adversarial filtering then removes items that remain solvable without external retrieval, including cases eliminated after testing on at least two closed-source MLLMs (Tao et al., 29 Aug 2025).

Difficulty is labeled by whether an item is solvable without search or with only image search by strong closed models such as o3 or Gemini. If not, it is categorized as Difficult or Hard. This preserves a BrowseComp-like difficulty profile for many items. A plausible implication is that MMSearch-Plus is intended not just as a multimodal benchmark, but as a benchmark in which multimodality does not relax the search burden (Tao et al., 29 Aug 2025).

3. Agent environment, rollout protocol, and evaluation

MMSearch-Plus is accompanied by a model-agnostic agent framework. The environment provides SerpApi-backed text search and image search, caches ranked image-search URLs and thumbnails per benchmark image, retrieves top-5 websites for text queries, and summarizes retrieved pages into web_info and related_info. Boilerplate HTML is discarded to save tokens, and agents receive the full history and prior reasoning steps (Tao et al., 29 Aug 2025).

The benchmark defines three evaluation modes. In Without Search, the model answers from question and images only. In Image Search, the agent is given top-10 image-search summaries per image. In Full Rollout, the agent interacts iteratively with the environment. The default budget is 20 search rounds; Qwen is capped at 10 because pilot experiments found more than 10 unhelpful. In each text-search round, the model generates 3–5 refined queries, while each image-search round issues exactly one query (Tao et al., 29 Aug 2025).

The primary metric is answer accuracy. If $N$ is the number of tasks and $\mathbb{1}[\cdot]$ is the indicator of correctness under the judge, then

$$\mathrm{Accuracy} = \frac{1}{N}\sum_{i=1}^{N}\mathbb{1}[p_i \in A_i],$$

where $p_i$ is the prediction and $A_i$ is the acceptable answer set. The benchmark uses GPT-4o as LLM-as-a-judge, and manual validation found complete agreement with human judgment. An optional rule-based verifier is also provided to aid reproducibility (Tao et al., 29 Aug 2025).

Beyond answer accuracy, MMSearch-Plus assesses bounding-box production and cropped-image search. Bounding-box scoring follows the standard Intersection-over-Union definition

$\mathrm{IoU} = \frac{|A \cap B|}{|A \cup B|},$

with a typical acceptance criterion of $\mathrm{IoU} \ge 0.5$ if bounding boxes are part of the task. However, the benchmark focuses primarily on answer accuracy and qualitative analysis for grounding because current model limitations make full-scale cropping experiments unstable (Tao et al., 29 Aug 2025).

4. Empirical difficulty and baseline performance

The reported results establish MMSearch-Plus as a difficult benchmark even for frontier multimodal systems. The strongest agent in the benchmark paper, o3, attains 15.1% without search and 36.0% accuracy with rollout under the provided framework. Gemini-2.5-Pro reaches 10.6% without search and 23.8% with full rollout. A strong open-source model, Qwen-2.5-VL-72B-Instruct, achieves 0.0% without search and 6.9% after 20 rounds of search in the abstract; in the main experimental table, its full-rollout average is 6.1 under the paper’s 10-round controller (Tao et al., 29 Aug 2025).

Model	Without Search	Full Rollout
o3 (2025-04-16)	15.1	36.0
Gemini-2.5-Pro	10.6	23.8
Qwen-2.5-VL-72B-Instruct	0.0	6.1

Image search provides consistent but modest gains for closed models and a larger one-shot lift for Qwen. The benchmark paper reports 19.3 for o3 and 16.4 for Gemini-2.5-Pro in Image Search mode, while Qwen rises from 0.0 without search to 13.5 with image search. At the same time, Qwen regresses under iterative rollout in the paper’s controller, from 13.5 to 6.1, indicating that long-horizon search is not plug-and-play and depends on rollout policies such as hop budgeting, snippet ranking, stopping, and web-reading robustness (Tao et al., 29 Aug 2025).

Trajectory analysis further clarifies the failure profile. Incorrect trajectories use more text-search calls. Gemini under-explores, typically issuing fewer than 5 search calls and leaning on parametric knowledge. Qwen exhibits tool-use failures, including 421 invalid tool calls across 45 tasks, with cascaded retries inflating call counts without better evidence. The manual error taxonomy for Gemini full rollout identifies “No relevant information found” as the dominant error at 51.1%, followed by hallucination at 11.5%, with additional categories including key information not extracted, relevance not verified, failed to understand image, refused to answer, multiple answers, question misunderstood, search not invoked, possible requirement for video understanding, and summarization error (Tao et al., 29 Aug 2025).

These results support the benchmark’s central claim: MMSearch-Plus restores genuinely multimodal pressure by requiring extraction of weak, localized cues and their propagation through noisy retrieval to out-of-image facts. A plausible implication is that benchmark progress depends not only on stronger perception, but also on stronger search control, evidence verification, and long-horizon planning (Tao et al., 29 Aug 2025).

5. Methods evaluated on or aligned with MMSearch-Plus

Among later multimodal search agents, LMM-Searcher is the clearest system directly evaluated on MMSearch-Plus. It introduces a Long-horizon MultiModal deep search framework centered on a file-based visual representation mechanism: images retrieved during search are stored externally and replaced in context by lightweight textual identifiers (UIDs), with a fetch_image tool used for progressive, on-demand visual loading. The default comparison uses a 30-turn cap and 128K context length, while long-horizon evaluation scales to 100 turns with a context management strategy that keeps only the recent 5 tool call results. On the single-image subset of MMSearch-Plus, LMM-Searcher-30B reports 32.9 success rate at 30 turns and 34.8 at 100 turns, outperforming open-source comparators such as Vision-DeepResearch-30B at 28.5 and REDSearcher-MM-30B at 26.6. Within the same framework, Seed-1.8 reaches 46.7, and removal of fetch_image lowers that result to 43.7, a 3.0-point drop (Du et al., 14 Apr 2026).

The paper “DR-MMSearchAgent: Deepening Reasoning in Multimodal Search Agents” does not name a “MMSearch-Plus” system explicitly. Conceptually and empirically, however, it extends MMSearch-R1 along three axes that would constitute a “Plus” upgrade: Structural Proximity-weighted Advantage Injection (SPAI), differentiated Gaussian rewards or BGAS, and a multi-turn deep-reasoning infrastructure built around BridgeVQA and a refining agent. The reported gains are not benchmarked on MMSearch-Plus itself in the provided summary, but they are presented as directly informative for how to build an MMSearch-Plus agent, including an 8.4% absolute improvement over MMSearch-R1 on FVQA-test and state-of-the-art performance among trained agents (Wang et al., 21 Apr 2026).

A similar conceptual usage appears in ProMMSearchAgent. That paper also does not explicitly define “MMSearch-Plus,” but interprets it as an enhanced MMSearch system. Its main contribution is a Sim-to-Real training paradigm combined with an introspective process-oriented reward that teaches the agent to initiate multimodal or text search only when visually or factually uncertain. Training occurs in a deterministic local sandbox and transfers zero-shot to live Google Search. Reported gains over MMSearch-R1 include +5.1 on FVQA-test, +6.3 on InfoSeek, and +11.3 on MMSearch, with the method positioned as a stronger, more general multimodal search capability than prior MMSearch agents (Yan et al., 22 Apr 2026).

Taken together, these works show two distinct uses of the term. In the strict sense, MMSearch-Plus is a benchmark introduced in 2025. In a looser systems sense, later agent papers use “MMSearch-Plus” or “Plus” language to denote a stronger multimodal search capability characterized by deeper tool-augmented reasoning, improved reward design, and more reliable long-horizon interaction. This suggests that the benchmark has also become a shorthand for a target capability profile, not just a dataset.

6. Limitations, controversies, and research directions

The benchmark paper identifies several limitations. Source bias remains significant because the dataset is drawn from public web sources such as YouTube, Bilibili, and arXiv, and under-represents non-English or low-visual-density pages. The framework targets image-text browsing; video ingestion and interactive UIs remain limited, and some tasks may require video understanding. Grounding is also a practical bottleneck: current MLLMs struggle with cropping and bounding boxes, which is why full-scale cropping experiments were avoided in the main evaluation (Tao et al., 29 Aug 2025).

The systems literature evaluated against or conceptually aligned with MMSearch-Plus exposes additional constraints. LMM-Searcher reports only the single-image subset of MMSearch-Plus for fair comparison, which narrows the tested regime. DR-MMSearchAgent notes sensitivity to batch composition in SPAI, the need for careful Gaussian parameter tuning in BGAS, and possible topical skews in BridgeVQA. ProMMSearchAgent highlights residual sandbox-to-web domain shift, coarse binary uncertainty labels, judge dependence in outcome rewards, and the brittleness introduced by tool constraints such as a single image-search call (Du et al., 14 Apr 2026).

A recurrent misconception is that multimodal browsing reduces to adding image search to a text-centric web agent. MMSearch-Plus was explicitly designed to counter that view. Its construction principles require multiple weak, localized visual signals per item, and its error analyses emphasize source verification failures, part-based reasoning failures, and long-horizon planning failures rather than purely perceptual misses. This suggests that success depends on the joint optimization of perception, retrieval, search control, and provenance checking rather than any single component (Tao et al., 29 Aug 2025).

The benchmark and the follow-on agent papers point toward several concrete research directions. The benchmark paper calls for richer visual toolchains, tighter cropping and grounding loops, video-native browsing, adaptive rollout policies, provenance-first training, and broader domain coverage. The agent papers add complementary directions: long-horizon context management through file-based visual representations and active perception; process-oriented or structure-aware reward shaping for multi-turn search; deterministic training environments for stable RL; and explicit mechanisms for calibrating when search should be invoked at all. A plausible implication is that future progress on MMSearch-Plus will be measured less by isolated perception improvements than by the quality of integrated browsing policies operating under noisy, multimodal, provenance-sensitive conditions (Tao et al., 29 Aug 2025).