MMSearch-R1: Adaptive Multimodal Search with RL

Updated 30 June 2025

MMSearch-R1 is a reinforcement learning framework that empowers large multimodal models to conduct adaptive, on-demand multi-turn searches using external text and image tools.
The paper demonstrates that RL-driven, on-demand search policies drastically reduce unnecessary search calls while enhancing accuracy on information-seeking VQA tasks.
Extensive experiments reveal that MMSearch-R1 balances internal knowledge with external retrieval, outperforming traditional RAG systems in real-world multimodal applications.

MMSearch-R1 is an end-to-end reinforcement learning (RL) framework that enables Large Multimodal Models (LMMs) to perform adaptive, on-demand, and multi-turn search in real-world Internet environments. Designed to address the inefficiencies of rigid retrieval-augmented generation (RAG) pipelines and prompt-engineered agents, MMSearch-R1 tightly integrates external text and image search tools and optimizes decision-making about tool use directly within the LMM’s agentic reasoning loop. The result is a paradigm in which multimodal models reason about when, how, and whether to issue search queries while aiming for minimal unnecessary search actions. This framework supports training on knowledge-intensive, information-seeking vision-language question-answering (VQA) tasks, demonstrating superior performance and search efficiency compared to equally sized and even larger RAG-based baselines.

1. Architectural Principles and Workflow

MMSearch-R1 is structured around an LMM (e.g., Qwen2.5-VL-7B) equipped with real-world search integrations and multi-step reasoning capabilities. The key components are:

Multimodal Input Handling: The model takes as input images and text-based queries relevant to visual VQA tasks.
Integrated Search Tools:
- Image Search (via SerpAPI): On encountering unfamiliar visual entities, the model can invoke this tool to retrieve the top-5 most visually similar web pages, supplying thumbnail-title pairs for further reasoning.
- Text Search (via SerpAPI + Jina Reader + Qwen3-32B): The model can generate and issue language-based web queries, receive fetched webpage content, and process informative summaries via the summarizer.
Multi-Turn, Multi-Tool Reasoning: At each turn, the model chooses between: continuing reasoning, invoking an image search, invoking a text search, or providing a final answer. Each round’s output is labeled by markup tags (e.g., <reason>, <search><img>...</search>, <search><text_search>...</search>, <answer>) for clear parsing and action tracking.
External Tool Abstraction with Data Masking: External tool returns are masked during the loss computation in training, ensuring that the model must learn to compose its own internal knowledge with fresh external evidence, rather than merely copying verbatim.

This architecture allows the LMM to adaptively invoke search only as needed, maximizing both accuracy and efficiency by balancing internal knowledge with external retrieval.

2. Reinforcement Learning Approach and Incentive Design

MMSearch-R1’s core training methodology is reinforcement learning using outcome-based rewards, specifically adapted Group Relative Policy Optimization (GRPO, a PPO variant). The key elements are:

Policy Gradient with Group-Normalized Advantage: GRPO updates the policy $\pi_\theta$ by evaluating a batch of G rollouts per prompt, normalizing rewards to reduce variance and stabilize learning.

$\mathcal{J}_{GRPO}(\theta) = \mathbb{E}[q\sim \mathcal{D}, \{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{old}}]\, \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\bigg\{ \min[R_{i,t}\hat{A}_{i,t}, \operatorname{clip}(R_{i,t},1-\varepsilon,1+\varepsilon)\hat{A}_{i,t}] - \beta\mathbb{D}_{KL}\left[\pi_{\theta}\|\pi_{ref}\right]\bigg\}$

where

$\hat{A}_{i,t} = \tilde{r}_i = \frac{r_i - \operatorname{mean}(\mathbf{r})}{\operatorname{std}(\mathbf{r})}$

Reward Function with Search Penalty:

$\text{reward} = (1 - \alpha) \cdot Acc\_Score \cdot Search\_Penalty + \alpha \cdot Format\_Score$

Acc_Score: 1 if the answer exactly matches ground truth; 0 otherwise.
Search_Penalty: [0, 1] multiplicative penalty if a search tool is used for a correct answer, discouraging unnecessary search calls when the LMM's intrinsic parametric knowledge suffices.
Format_Score: Enforces strict output formatting (1 if all required tags are used, 0 otherwise).
α: Weighting coefficient for format compliance (experimentally set to 0.1).

Through this outcome-based incentivization, MMSearch-R1 encourages the LMM to learn nuanced policies for when to rely on its own knowledge versus when to seek external evidence.

3. Dataset Construction and Search-Balanced Curation

To train and robustly evaluate search policy, MMSearch-R1 introduces a systematic pipeline for multimodal search VQA data collection:

Balanced Search Requirement: The final training set was curated to include search-required (e.g., rare facts, unfamiliar images) and search-free questions (those solvable without external retrieval). Effective search learning depends on the presence of both types to penalize overuse of external search.
Automated Generation and Categorization:
- Visual Knowledge-Required Samples: Sampled from the MetaCLIP metadata and web images paired with factual VQA generated by GPT-4o.
- Textual Knowledge-Required Samples: Drawn from the InfoSeek dataset, with similar GPT-4o curation.
- Manual Annotation: Additional human-written samples to maximize topic diversity and real-world coverage.
- Search-Type Labeling: A classifier (trained on model rollouts) automatically tags each instance as image-search, text-search, mixed, or search-free.
Final FVQA Dataset: 5,000 training (≈3400 search-required, 1600 search-free) and 1,800 test examples; each question includes images, concise answers, and web pages/facts as references. Only questions with high-quality rollouts (sufficient supporting search traces, minimal ambiguity) are retained.

This careful balancing and labeling are critical to incentivize efficient, on-demand search in RL.

4. Performance Benchmarks and Empirical Results

MMSearch-R1 is evaluated against direct generation (no search), RAG-based pipelines (retrieve-then-generate with fixed search calls), and ablations:

Accuracy Metrics: Judged by LLM-as-Judge (GPT-4o), which provides reliable, human-aligned correctness evaluation for textual and visual answers.
Search Ratio: Measures what fraction of questions the model actually invokes a search tool—critical for quantifying efficiency.
Key Results (see Table 1 in the paper):
- MMSearch-R1-7B achieves 54.6% accuracy—outperforming RAG-based Qwen2.5-VL-7B (51.6%) and approaching the accuracy of a much larger (32B) RAG model (55.1%).
- Search ratio is 67.1% for MMSearch-R1, representing a >32% reduction in search calls compared to RAG (always 100%).
- On classical VQA benchmarks (AI2D, ChartQA, etc.), the model matches or surpasses the base model, indicating there is no regression in general multimodal reasoning.
Ablation Insights: When data balancing or the search penalty is removed, unnecessary search calls rise toward 100%, even if task accuracy remains high. Both are necessary for efficient, real-world-ready behavior.

5. Empirical Findings and Methodological Insights

Several actionable insights and research implications are highlighted:

RL-Driven On-Demand Search is Effective: Outcome-oriented RL enables superior accuracy and significant search call reduction compared to rigid RAG workflows, even with equivalent or smaller parameter sizes. RL-trained models exhibit better knowledge utilization, answering more questions from parametric memory.
Balanced Training is Crucial: The presence of both search-required and search-free samples, coupled with an explicit search penalty, is essential for fostering adaptive search behavior.
Supervised Fine-Tuning is Insufficient: SFT, even with large data, is less effective than outcome-based RL for multi-modal tool-use policy. RL provides higher efficiency, greater search selectivity, and better task adaption.
Search Decision Generalization: The model adapts search frequency naturally to task demands (e.g., high for OOD and info-seeking tasks, low for internal-knowledge questions), indicating robust policy generalization.

6. Mathematical Expressions and Algorithmic Summary

Policy Update (GRPO):

$\mathcal{J}_{GRPO}(\theta) = \mathbb{E} \left[ \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|} \sum_{t=1}^{|o_{i}|} \left\{ \min[R_{i,t}\hat{A}_{i,t}, \operatorname{clip}(R_{i,t},1-\varepsilon,1+\varepsilon)\hat{A}_{i,t}] - \beta\mathbb{D}_{KL}[\pi_{\theta}\|\pi_{ref}] \right\}\right]$

Reward with Search Penalty:

$\text{reward} = (1 - \alpha) \cdot Acc\_Score \cdot Search\_Penalty + \alpha \cdot Format\_Score$

Advantage Normalization:

$\hat{A}_{i,t} = \frac{r_i - \operatorname{mean}(\mathbf{r})}{\operatorname{std}(\mathbf{r})}$

7. Impact and Applications

The MMSearch-R1 framework provides an efficient and robust approach for multimodal information-seeking agents in settings where real-time, on-demand search is necessary yet wasteful or excessive tool use must be avoided. The released code and dataset [https://github.com/EvolvingLMMs-Lab/multimodal-search-r1] facilitate reproducible research and further development of LMMs equipped for cost-sensitive, dynamic search workflows, with immediate relevance to AI assistants, knowledge-based VQA, and agent-based systems requiring balanced integration of internal and external knowledge sources.

PDF Markdown Chat (Pro)