OpenSearch-VL: Deep Multimodal Search

• OpenSearch-VL is a multimodal deep search framework that fuses visual perception, evidence retrieval, and multi-step reasoning using agentic reinforcement learning.
• It integrates a unified tool suite for active visual processing (e.g., cropping, sharpening, OCR) alongside text-based search to enhance evidence acquisition.
• The system demonstrates over 10-point average improvements on seven benchmarks while employing advanced reward strategies to manage tool failures.

Searching arXiv for OpenSearch-VL and closely related multimodal search-agent papers. OpenSearch-VL is a fully open-source recipe for training frontier multimodal deep search agents with agentic reinforcement learning. It is designed for settings in which a model must interleave visual perception, external evidence retrieval, evidence verification, and multi-step reasoning, rather than rely on passive image understanding alone. The recipe centers on three released components: an image-grounded multi-hop data construction pipeline, a unified tool environment spanning both retrieval and active visual pre-processing, and a multi-turn fatal-aware GRPO training algorithm for multimodal tool use. Built on Qwen3-VL backbones, it reports over 10-point average improvements across seven benchmarks and reaches 61.6 average at 30B-A3B and 63.7 at 32B (Chen et al., 6 May 2026).

1. Problem setting and conceptual scope

OpenSearch-VL defines deep search for multimodal agents as moving beyond passive vision-language inference into active evidence seeking. In this setting, the agent does not merely inspect an image and answer; it may crop or enhance visual evidence, invoke text or image search, verify retrieved facts, and assemble a multi-hop reasoning chain before producing a final response. The agent therefore operates in a multimodal environment of tools and produces multi-turn trajectories grounded in both visual observations and external knowledge (Chen et al., 6 May 2026).

The work is motivated by a reproducibility gap in frontier multimodal search agents. The reported obstacles are the lack of open high-quality data, opaque trajectory synthesis pipelines, and missing training recipes. OpenSearch-VL addresses these by releasing a data pipeline based on Wikipedia graph path construction, fuzzy entity rewriting, and source-anchor visual grounding; by synthesizing expert trajectories in a real tool environment with rejection sampling; and by specifying a detailed reinforcement learning recipe with explicit handling of cascading tool failures (Chen et al., 6 May 2026).

Formally, at step $l$, the accumulated history is

$$h_l = \bigl(\mathcal{I}_l,\;\, q,\;\, \mathbf{a}_{<l},\;\, \mathbf{o}_{<l}\bigr),$$

and the multi-turn trajectory is

$$\tau = \bigl\{(h_0, a_0, o_0),\; (h_1, a_1, o_1),\; \dots,\; (h_{L-1}, a_{L-1}, o_{L-1}),\; (h_L, a_L)\bigr\}.$$

Each action is decomposed in ReAct style as $a_l=[z_l,c_l]$, where $z_l$ is a reasoning trace and $c_l$ is either a tool call or the final response. The trajectory likelihood is factorized as

$$\pi_\theta(\tau \mid I_0, q) \;=\; \prod_{l=0}^{L} P_\theta(a_l \mid h_l) \;=\; \prod_{l=0}^{L} P_\theta(z_l \mid h_l)\, P_\theta(c_l \mid h_l, z_l).$$

This formalization places reasoning traces, tool invocations, and multimodal observations inside a single sequential policy over the environment (Chen et al., 6 May 2026).

A plausible implication is that OpenSearch-VL is not only a model family but also a reproducible systems recipe: it specifies how to construct the task distribution, how to expose tools to the policy, and how to optimize the policy when failures in tool execution are frequent and non-terminal.

2. Data construction pipeline and released datasets

The data pipeline begins by casting Wikipedia as a directed graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ and sampling constrained random walks of length $h\in\{2,3,4\}$:

$$P \;=\; \bigl(v_0 \xrightarrow{\rho_1} v_1 \xrightarrow{\rho_2} \cdots \xrightarrow{\rho_h} v_h\bigr),$$

with $$h_l = \bigl(\mathcal{I}_l,\;\, q,\;\, \mathbf{a}_{<l},\;\, \mathbf{o}_{<l}\bigr),$$0 as the visual anchor, $$h_l = \bigl(\mathcal{I}_l,\;\, q,\;\, \mathbf{a}_{<l},\;\, \mathbf{o}_{<l}\bigr),$$1 as bridge nodes, and $$h_l = \bigl(\mathcal{I}_l,\;\, q,\;\, \mathbf{a}_{<l},\;\, \mathbf{o}_{<l}\bigr),$$2 as the answer node. The path sampler excludes disambiguation pages, list pages, cycles, hubs with in-degree above $$h_l = \bigl(\mathcal{I}_l,\;\, q,\;\, \mathbf{a}_{<l},\;\, \mathbf{o}_{<l}\bigr),$$3, non-article namespaces, and resolves redirects via dereferencing. Path lengths are sampled with probabilities $$h_l = \bigl(\mathcal{I}_l,\;\, q,\;\, \mathbf{a}_{<l},\;\, \mathbf{o}_{<l}\bigr),$$4 for $$h_l = \bigl(\mathcal{I}_l,\;\, q,\;\, \mathbf{a}_{<l},\;\, \mathbf{o}_{<l}\bigr),$$5 (Chen et al., 6 May 2026).

The second stage is fuzzy entity rewriting, which transforms a canonical text question into a fuzzy question that suppresses one-step name-based shortcuts. The rewriting proceeds progressively from the farthest bridge node to the source anchor, under three invariants:

$$h_l = \bigl(\mathcal{I}_l,\;\, q,\;\, \mathbf{a}_{<l},\;\, \mathbf{o}_{<l}\bigr),$$6

These enforce answer invariance, uniqueness, and non-leakage of entity aliases. The stated goal is to prevent shortcut retrieval collapse while preserving answerability (Chen et al., 6 May 2026).

The third stage is source-anchor visual grounding. Representative images of the anchor node $$h_l = \bigl(\mathcal{I}_l,\;\, q,\;\, \mathbf{a}_{<l},\;\, \mathbf{o}_{<l}\bigr),$$7 are retrieved from Wikimedia or Wikipedia infobox sources and filtered by CLIP similarity to a short textual anchor description. The anchor mention in the rewritten question is then replaced by a visual referring expression such as “the person in the image.” According to the paper, grounding at the source node rather than near the answer entity forces the agent to identify the visual anchor and traverse textual relations to the answer (Chen et al., 6 May 2026).

Quality control is applied in several layers. The authors consolidate Wikipedia-derived VQA instances with LiveVQA, FVQA, and WebQA. A frozen Qwen3-VL-32B is then used in a two-stage difficulty filter to remove examples answerable without tools and examples solvable with a single ImageSearch call. In addition, 10% of retained images are randomly degraded by blur, downsampling, or skew and paired with enhancement tools to teach “fix-before-search” behavior (Chen et al., 6 May 2026).

Two datasets are produced.

Dataset	Size	Purpose
SearchVL-SFT-36k	36,592 trajectories	SFT
SearchVL-RL-8k	8,000 prompts	On-policy RL

SearchVL-SFT-36k consists of 36,592 high-quality multi-turn expert trajectories with an average of 6.3 tool calls per trajectory. SearchVL-RL-8k contains 8,000 disjoint, tool-demanding VQA prompts for reinforcement learning. Expert trajectories are rolled out with $$h_l = \bigl(\mathcal{I}_l,\;\, q,\;\, \mathbf{a}_{<l},\;\, \mathbf{o}_{<l}\bigr),$$8 candidates per instance using Claude Opus 4.6 in a real tool environment, then filtered by answer correctness with a GPT-4o judge and process-level vetting with GPT-5.4 for tool use, logical consistency, and avoidance of ineffective repetition (Chen et al., 6 May 2026).

A representative example in the released corpus is a bridge-opening question whose trajectory is: crop the sign, identify the entity with image search as “Kessock Bridge,” verify the opening year with text search, and then answer “1982.” This example is used in the paper to illustrate how active perception and external retrieval are interleaved in a single trajectory (Chen et al., 6 May 2026).

3. Tool environment and multimodal interaction

OpenSearch-VL provides a unified tool suite with seven tools: TextSearch, ImageSearch, OCR, Crop, Sharpen, SuperResolution, and PerspectiveCorrect. Tools are injected through an OpenAI-style JSON schema under <tools>, and the agent emits calls in <tool_call>{"name":..., "arguments":...}</tool_call> format. This design makes retrieval and visual pre-processing first-class actions of the policy rather than fixed external preprocessing steps (Chen et al., 6 May 2026).

The retrieval tools are specialized by evidence type. TextSearch takes a query string, language, and top-$$h_l = \bigl(\mathcal{I}_l,\;\, q,\;\, \mathbf{a}_{<l},\;\, \mathbf{o}_{<l}\bigr),$$9 and returns passage summaries produced from a Serper search pipeline, JINA Reader extraction, and Qwen3-32B summarization. ImageSearch takes an image URL and top-$$\tau = \bigl\{(h_0, a_0, o_0),\; (h_1, a_1, o_1),\; \dots,\; (h_{L-1}, a_{L-1}, o_{L-1}),\; (h_L, a_L)\bigr\}.$$0 and returns visually similar images, recognized entities, related webpages, and captions. The paper states that ImageSearch is typically used to identify the visual entity and is then followed by TextSearch for factual verification (Chen et al., 6 May 2026).

The visual tools implement active perception. OCR extracts text blocks with labels, reading order, and formatted text. Crop isolates a region of interest using rectangular coordinates. Sharpen applies unsharp masking,

$$\tau = \bigl\{(h_0, a_0, o_0),\; (h_1, a_1, o_1),\; \dots,\; (h_{L-1}, a_{L-1}, o_{L-1}),\; (h_L, a_L)\bigr\}.$$1

to aid OCR or matching. SuperResolution uses EDSR, with a fallback if weights are unavailable. PerspectiveCorrect rectifies skewed images through edge detection, quadrilateral detection, and a perspective transform. The paper’s framing is that retrieval alone cannot fix degraded or badly localized visual evidence, so tools that restore or isolate evidence are part of the environment itself (Chen et al., 6 May 2026).

Observations are multimodal and depend on the invoked tool:

$$\tau = \bigl\{(h_0, a_0, o_0),\; (h_1, a_1, o_1),\; \dots,\; (h_{L-1}, a_{L-1}, o_{L-1}),\; (h_L, a_L)\bigr\}.$$2

The active visual context $$\tau = \bigl\{(h_0, a_0, o_0),\; (h_1, a_1, o_1),\; \dots,\; (h_{L-1}, a_{L-1}, o_{L-1}),\; (h_L, a_L)\bigr\}.$$3 grows monotonically as image-valued observations are accumulated, so enhanced, cropped, and rectified variants remain available for later cross-reference (Chen et al., 6 May 2026).

The environment explicitly models execution failures. Tool-execution errors such as timeouts, malformed arguments, or parsing failures are flagged by the sandbox. A fatal state begins at the earliest step $$\tau = \bigl\{(h_0, a_0, o_0),\; (h_1, a_1, o_1),\; \dots,\; (h_{L-1}, a_{L-1}, o_{L-1}),\; (h_L, a_L)\bigr\}.$$4 where $$\tau = \bigl\{(h_0, a_0, o_0),\; (h_1, a_1, o_1),\; \dots,\; (h_{L-1}, a_{L-1}, o_{L-1}),\; (h_L, a_L)\bigr\}.$$5 consecutive tool-execution errors start. This fatal-state notion is central to the reinforcement learning algorithm described later. A common misconception is that OpenSearch-VL is only a retrieval benchmark with tool wrappers; the released recipe instead treats tool interaction as a stochastic sequential environment in which execution failures alter the optimization target (Chen et al., 6 May 2026).

4. Model architecture and training recipe

OpenSearch-VL is instantiated on Qwen3-VL-8B-Instruct, Qwen3-VL-30B-A3B-Instruct, and Qwen3-VL-32B-Instruct. The recipe uses the Qwen3-VL vision tower and multimodal projector, with full finetuning rather than freezing the vision tower. The reasoning format is ReAct-style: a > block precedes each tool call, and the final answer appears in <response> once evidence is judged sufficient (Chen et al., 6 May 2026).

The supervised objective trains both reasoning traces and tool or response actions while masking tool observations out of the loss:

$$\tau = \bigl\{(h_0, a_0, o_0),\; (h_1, a_1, o_1),\; \dots,\; (h_{L-1}, a_{L-1}, o_{L-1}),\; (h_L, a_L)\bigr\}.$$6

The generation mask $$\tau = \bigl\{(h_0, a_0, o_0),\; (h_1, a_1, o_1),\; \dots,\; (h_{L-1}, a_{L-1}, o_{L-1}),\; (h_L, a_L)\bigr\}.$$7 is $$\tau = \bigl\{(h_0, a_0, o_0),\; (h_1, a_1, o_1),\; \dots,\; (h_{L-1}, a_{L-1}, o_{L-1}),\; (h_L, a_L)\bigr\}.$$8 for generated actions such as think, tool call, or response tokens and $$\tau = \bigl\{(h_0, a_0, o_0),\; (h_1, a_1, o_1),\; \dots,\; (h_{L-1}, a_{L-1}, o_{L-1}),\; (h_L, a_L)\bigr\}.$$9 for textual observations. Image observations bypass token loss through the vision tower. Reported SFT hyperparameters include full finetuning with bfloat16, learning rate $a_l=[z_l,c_l]$0, 8 epochs, DeepSpeed ZeRO-3 over 256 H20 GPUs, effective batch size 256, maximum sequence length about 32k tokens, and images capped at about 262k pixels (Chen et al., 6 May 2026).

The reinforcement learning stage introduces multi-turn fatal-aware GRPO. The rollout reward is

$a_l=[z_l,c_l]$1

Here $a_l=[z_l,c_l]$2 is a deterministic format-compliance reward, $a_l=[z_l,c_l]$3 is a binary answer-accuracy reward from a GPT-4o judge, and $a_l=[z_l,c_l]$4 is a GPT-5.4 process-level reward that scores search quality over the valid prefix when the answer is wrong or the trajectory terminates fatally (Chen et al., 6 May 2026).

Fatal-aware masking zeroes post-failure tokens:

$a_l=[z_l,c_l]$5

where $a_l=[z_l,c_l]$6 maps token index $a_l=[z_l,c_l]$7 to step index $a_l=[z_l,c_l]$8. One-sided advantage clamping then prevents valid prefixes from being penalized by later failure:

$a_l=[z_l,c_l]$9

The resulting GRPO objective is written over masked token positions and clipped importance ratios, with standard KL regularization (Chen et al., 6 May 2026).

The RL implementation uses actor learning rate $z_l$0, PPO clip 0.28, RLOO advantage, bfloat16, SGLang asynchronous rollout, and group size $z_l$1 for 8B or $z_l$2 for 30B-A3B. The reported RL run uses 64 H20 GPUs for about 200 steps over about 10 days (Chen et al., 6 May 2026).

This training design is distinct from training-free alternatives such as cross-modal model merging with Optimal Brain Merging, which composes search behavior into a base VLM without additional multimodal training data (Wang et al., 2 Mar 2026). That contrast suggests two different open research directions: OpenSearch-VL emphasizes transparent data, tools, and agentic RL, whereas merging-based paradigms emphasize zero-shot composition and warm-start initialization.

5. Empirical results, ablations, and benchmark behavior

Evaluation is reported on seven benchmarks: SimpleVQA, VDR, MMSearch, LiveVQA, BrowseComp-VL, FVQA, and InfoSeek, using Pass@1 with a GPT-4o judge aligned with the VDR protocol. The three principal OpenSearch-VL checkpoints reach the following averages (Chen et al., 6 May 2026).

Model	Average	Selected detail
OpenSearch-VL-8B	56.6	MMSearch 64.5
OpenSearch-VL-30B-A3B	61.6	VDR 33.5
OpenSearch-VL-32B	63.7	MMSearch 72.3

For OpenSearch-VL-8B, the reported per-benchmark scores are 71.6 on SimpleVQA, 20.8 on VDR, 64.5 on MMSearch, 59.6 on LiveVQA, 37.6 on BrowseComp-VL, 71.5 on FVQA, and 70.2 on InfoSeek. OpenSearch-VL-30B-A3B reaches 74.9, 33.5, 68.7, 67.4, 41.1, 73.2, and 72.4 respectively, improving the baseline Qwen3-VL-30B-A3B agent average from 47.8 to 61.6, with reported gains of +13.3 on VDR, +24.5 on MMSearch, +10.2 on FVQA, and +16.2 on InfoSeek. OpenSearch-VL-32B reaches 76.2 on SimpleVQA, 33.8 on VDR, 72.3 on MMSearch, 70.5 on LiveVQA, 43.8 on BrowseComp-VL, 74.7 on FVQA, and 74.8 on InfoSeek (Chen et al., 6 May 2026).

The data-pipeline ablation quantifies the importance of the three main curation ingredients. On the average over SimpleVQA, InfoSeek, and FVQA, the full pipeline scores 64.6. Removing source-anchor grounding reduces this to 53.1, removing fuzzy entity rewriting reduces it to 54.3, removing staged filtering reduces it to 56.4, and removing the enhancement subset yields 63.3. These results support the claim that source-anchor grounding and fuzzy rewriting are not decorative preprocessing steps but the main mechanisms preventing shortcut retrieval and enforcing tool demand (Chen et al., 6 May 2026).

The RL-recipe ablation isolates the effect of fatal-aware optimization. On the same subset, the base Qwen3-VL-8B agent scores 53.7, SFT-only reaches 64.6, vanilla GRPO reaches 67.6, hard masking gives 67.7, fatal masking alone gives 69.1, and fatal masking plus one-sided clamp reaches 71.8. The reported gain is +4.2 over vanilla GRPO. The paper interprets this as evidence that preserving useful pre-failure reasoning yields longer and more productive tool-use trajectories (Chen et al., 6 May 2026).

The benchmark profile also situates OpenSearch-VL against prior agentic systems. The baselines include direct reasoning, RAG workflows, and agentic workflows such as MMSearch-R1-7B, WebWatcher-7B/32B, and SenseNova-MARS-8B. The authors report that OpenSearch-VL consistently outperforms open baselines and approaches proprietary systems on several tasks (Chen et al., 6 May 2026). A related but distinct result from a merging-based line of work is that training-free agents can secure a reasonable zero-shot floor, while warm-start merging improves RL convergence and peak search accuracy (Wang et al., 2 Mar 2026). Taken together, these results suggest a methodological divide between fully trained multimodal search agents and search-capability composition into pretrained backbones.

6. Positioning, limitations, and broader research context

OpenSearch-VL positions itself against earlier multimodal search-agent work by emphasizing three differences: an open data pipeline that suppresses one-step retrieval collapse, a unified tool suite that includes active perception rather than retrieval alone, and a multi-turn RL objective that prunes post-failure tokens while preserving valid prefixes through one-sided advantage clamping (Chen et al., 6 May 2026). The paper explicitly contrasts this with less transparent systems in which data sources, trajectory synthesis, and reward-design details remain proprietary.

Its emphasis on active perception connects it to a broader observation in multimodal search research: search often requires fixing or localizing visual evidence before external knowledge acquisition. This is also a recurring theme in multimodal retrieval-and-ranking systems, where document images, screenshots, or videos must be represented jointly with text in a unified retrieval space (Li et al., 8 Jan 2026). OpenSearch-VL differs in that its core unit is not a static query-document pair but a multi-turn policy that decides when to crop, sharpen, run OCR, search images, and search text (Chen et al., 6 May 2026).

The work’s limitations are explicit. External tools introduce instability through search-ranking drift, fetch failures, and occasional summarization hallucinations. The reward depends on GPT-4o and GPT-5.4 judges, making it costly and version-dependent. Query-quality reward currently scores textual queries and does not cover intermediate visual operations. Exact reproducibility is complicated by reliance on external APIs such as Serper and PaddleX OCR, even though code, checkpoints, and datasets are released to mitigate this (Chen et al., 6 May 2026).

A plausible implication is that future extensions may replace proprietary judges with open process reward models that score both textual and visual tool use. Another plausible implication is that the released recipe can serve as a substrate for studying tool-failure robustness, multimodal process supervision, and agentic search generalization under changing web environments.

More broadly, OpenSearch-VL should not be conflated with embodied or navigation-based open-vocabulary search frameworks. For example, OVAMOS formulates open-vocabulary multi-object search in unknown indoor environments with frontier-based exploration, VLM reasoning, and a POMDP to manage uncertainty (Wang et al., 3 Mar 2025), while FlySearch shows that current VLMs struggle with long-horizon exploration in photorealistic 3D outdoor search tasks (Pardyl et al., 3 Jun 2025). Those systems study physical exploration and search under partial observability; OpenSearch-VL studies multimodal deep search over tools, web evidence, and visual manipulations. The shared principle is active search, but the operational environment and optimization target differ substantially.