SearchVL-SFT-36k Dataset Overview

• SearchVL-SFT-36k is a supervised fine-tuning dataset that trains multimodal agents to verify and repair visual evidence through structured, multi-turn trajectories.
• The dataset employs source-anchor grounding, fuzzy entity rewriting, and curated tool calls to facilitate multi-hop, tool-augmented reasoning.
• Its design minimizes shortcut solutions and prepares agents for a reinforcement learning stage, ensuring robust, procedural competence.

Searching arXiv for the specified paper to ground the article in the cited source. SearchVL-SFT-36k is the supervised fine-tuning dataset introduced with OpenSearch-VL for training multimodal agents in deep search and tool-augmented reasoning. It comprises 36,592 high-quality, multi-turn trajectories in which an expert agent interleaves internal reasoning with calls to a unified tool suite—text search, image search, OCR, crop, sharpen, super-resolution, and perspective correction—to solve image-grounded, knowledge-intensive questions (Chen et al., 6 May 2026). Within the OpenSearch-VL recipe, the dataset serves as the supervised component of a two-stage curriculum: SearchVL-SFT-36k instills structured ReAct-style think-then-act behavior and tool routing, while SearchVL-RL-8k supplies prompts for a subsequent reinforcement-learning stage based on fatal-aware GRPO (Chen et al., 6 May 2026).

1. Definition and role within OpenSearch-VL

SearchVL-SFT-36k is intended to train multimodal agents to “verify, don’t guess” by demonstrating how to read or repair images, identify entities, query external sources, and verify evidence before answering (Chen et al., 6 May 2026). The target questions are explicitly multi-hop and cross-modal rather than single-shot visual recognition problems. Supported tasks include deep multimodal search, long-horizon tool-augmented reasoning, evidence verification, image pre-processing, OCR-driven reading, and multi-hop navigation from visual anchors into factual sources (Chen et al., 6 May 2026).

Its place in the broader training recipe is tightly specified. SearchVL-SFT-36k provides offline expert demonstrations for supervised training, and SearchVL-RL-8k provides prompts for on-policy reinforcement learning under a fatal-aware GRPO objective (Chen et al., 6 May 2026). The trajectories in the SFT set explicitly follow the ReAct convention and the OpenSearch-VL tool suite, making them directly compatible with the RL environment and rewards. This coupling is important because the later RL stage assumes a clean decomposition between reasoning traces, tool calls, and terminal responses.

A central design premise is that multimodal search agents should not treat vision merely as a recognition frontend. Instead, SearchVL-SFT-36k encodes procedures in which image understanding, image repair, retrieval, and evidence validation are interleaved. This suggests a dataset design oriented not just toward answer accuracy, but toward procedural competence in multimodal environments.

2. Formal trajectory model and supervision target

The dataset is defined over multi-turn interaction histories and trajectories. The interaction history at step $l$ is

$h_l = (\mathcal{I}_l, q, a_{<l}, o_{<l}),$

and a trajectory is

$$\tau = \{(h_0, a_0, o_0), \ldots, (h_{L-1}, a_{L-1}, o_{L-1}), (h_L, a_L)\}.$$

Each action is decomposed as

$a_l = [z_l, c_l],$

where $z_l$ is a reasoning trace and $c_l$ is either a tool call or the final response (Chen et al., 6 May 2026).

Observation routing is multimodal and depends on the tool invoked. The environment output is

$$o_l = \mathcal{E}(c_l, h_l) \in \begin{cases} \mathcal{O}^{img}, & \text{if } c_l \text{ invokes a visual tool in } \mathcal{T}_v \setminus \{\text{OCR}\} \ \mathcal{O}^{txt}, & \text{if } c_l \text{ invokes a retrieval tool in } \mathcal{T}_s \text{ or OCR} \end{cases}$$

and the active visual context grows monotonically:

$$\mathcal{I}_l = \{I_0\} \cup \{o_k : k < l,\, o_k \in \mathcal{O}^{img}\}.$$

This makes the trajectory explicitly stateful with respect to image-valued intermediate outputs such as crops, sharpened views, or perspective-corrected images (Chen et al., 6 May 2026).

The SFT likelihood factorizes over turns:

$$\pi_\theta(\tau \mid I_0, q) = \prod_{l=0}^{L} P_\theta(a_l \mid h_l) = \prod_{l=0}^{L} P_\theta(z_l \mid h_l)\, P_\theta(c_l \mid h_l, z_l).$$

The paper notes that observations $o_l$ are exogenous outputs of the environment and are not part of the generative mass (Chen et al., 6 May 2026). This distinction is operationalized through a token-level generation mask $h_l = (\mathcal{I}_l, q, a_{<l}, o_{<l}),$0, which marks only policy-generated tokens. Textual observations such as search results and OCR outputs are excluded from the loss.

The supervised objective is given in two equivalent forms. At the trajectory level, the model maximizes

$h_l = (\mathcal{I}_l, q, a_{<l}, o_{<l}),$1

In next-token form,

$h_l = (\mathcal{I}_l, q, a_{<l}, o_{<l}),$2

This masking scheme is a defining property of the dataset’s training semantics: the model is supervised on reasoning, tool invocation, and final responses, but not on environment-returned text (Chen et al., 6 May 2026).

3. Data construction pipeline

The synthesis pipeline is explicitly designed to reduce shortcut solutions and collapse into one-step retrieval. It has three stages (Chen et al., 6 May 2026).

3.1 High-quality VQA construction via Wikipedia paths

The first stage treats English Wikipedia as a directed graph $h_l = (\mathcal{I}_l, q, a_{<l}, o_{<l}),$3 with hyperlinks as edges. From a seed $h_l = (\mathcal{I}_l, q, a_{<l}, o_{<l}),$4, the pipeline samples a path length $h_l = (\mathcal{I}_l, q, a_{<l}, o_{<l}),$5 with categorical probabilities $h_l = (\mathcal{I}_l, q, a_{<l}, o_{<l}),$6, yielding

$h_l = (\mathcal{I}_l, q, a_{<l}, o_{<l}),$7

In this path, $h_l = (\mathcal{I}_l, q, a_{<l}, o_{<l}),$8 is the anchor, $h_l = (\mathcal{I}_l, q, a_{<l}, o_{<l}),$9 are bridges, and $$\tau = \{(h_0, a_0, o_0), \ldots, (h_{L-1}, a_{L-1}, o_{L-1}), (h_L, a_L)\}.$$0 is the answer node. The walk skips disambiguation and list pages, cycles, hub nodes with in-degree exceeding $$\tau = \{(h_0, a_0, o_0), \ldots, (h_{L-1}, a_{L-1}, o_{L-1}), (h_L, a_L)\}.$$1, and non-article namespaces; redirects are dereferenced. Seeds are stratified across Person, Building/Place, Location, Organism, and Artifact, and must have infoboxes and Wikimedia images of at least $$\tau = \{(h_0, a_0, o_0), \ldots, (h_{L-1}, a_{L-1}, o_{L-1}), (h_L, a_L)\}.$$2 pixels (Chen et al., 6 May 2026).

A canonical question $$\tau = \{(h_0, a_0, o_0), \ldots, (h_{L-1}, a_{L-1}, o_{L-1}), (h_L, a_L)\}.$$3 verbalizes the path and queries an attribute from $$\tau = \{(h_0, a_0, o_0), \ldots, (h_{L-1}, a_{L-1}, o_{L-1}), (h_L, a_L)\}.$$4 while extracting a short answer $$\tau = \{(h_0, a_0, o_0), \ldots, (h_{L-1}, a_{L-1}, o_{L-1}), (h_L, a_L)\}.$$5. Entity names are then iteratively rewritten into relational or attribute descriptors from Wikipedia context, moving from $$\tau = \{(h_0, a_0, o_0), \ldots, (h_{L-1}, a_{L-1}, o_{L-1}), (h_L, a_L)\}.$$6 toward $$\tau = \{(h_0, a_0, o_0), \ldots, (h_{L-1}, a_{L-1}, o_{L-1}), (h_L, a_L)\}.$$7, to produce a fuzzy question $$\tau = \{(h_0, a_0, o_0), \ldots, (h_{L-1}, a_{L-1}, o_{L-1}), (h_L, a_L)\}.$$8 that preserves the answer but removes explicit names. An LLM evaluator enforces three invariants:

$$\tau = \{(h_0, a_0, o_0), \ldots, (h_{L-1}, a_{L-1}, o_{L-1}), (h_L, a_L)\}.$$9

These impose answer invariance, uniqueness, and non-leakage of entity names (Chen et al., 6 May 2026).

The final step in this stage is source-anchor visual grounding. A representative image $a_l = [z_l, c_l],$0 is retrieved for $a_l = [z_l, c_l],$1 from Wikimedia Commons or the infobox and filtered by CLIP similarity to a short description. The anchor mention in $a_l = [z_l, c_l],$2 is then replaced with a visual reference such as “the place/person in the image,” producing the final VQA tuple $a_l = [z_l, c_l],$3. The paper states that anchoring at $a_l = [z_l, c_l],$4 rather than $a_l = [z_l, c_l],$5 prevents trivial reverse-image lookup of the answer entity and forces multi-hop reasoning from the anchor into the answer (Chen et al., 6 May 2026).

Automatic gating checks in a selector/examiner style verify masking, uniqueness, and visual relevance. The paper notes that a representative CLIP similarity threshold of 0.28 is shown in the running example, while exact thresholds may vary per instance (Chen et al., 6 May 2026).

3.2 Filtering and enhancement for tool demand

The Wikipedia-derived VQA pool is merged with three open-source corpora—LiveVQA, FVQA, and WebQA—to broaden topical coverage (Chen et al., 6 May 2026). A two-stage difficulty filter using a frozen Qwen3-VL-32B is then applied. Stage 1 discards samples answerable without tools; Stage 2 discards samples solvable with a single ImageSearch call. The stated effect is to remove parametric-knowledge cases, perceptual shortcuts, answer-coincident anchors, and one-hop leakage cases, thereby concentrating the retained data on tool-demanding queries (Chen et al., 6 May 2026).

An enhancement subset comprising 10% of the filtered VQA pool is created by randomly degrading images through blur, downsampling, and perspective skew, then pairing them with Sharpen, SuperResolution, and PerspectiveCorrect. This explicitly teaches agents to repair visual evidence before retrieval (Chen et al., 6 May 2026).

3.3 Multi-turn trajectory synthesis

For each filtered instance $a_l = [z_l, c_l],$6, Claude Opus 4.6 is rolled out against the production tool suite to collect $a_l = [z_l, c_l],$7 ReAct-style trajectories. These trajectories are then subjected to a two-stage rejection cascade. First, a GPT-4o judge checks whether the final answer matches $a_l = [z_l, c_l],$8. Second, a GPT-5.4 process-level judge evaluates tool-use structure, consistency between reasoning and observations, and the absence of ineffective repetition (Chen et al., 6 May 2026).

The final SFT corpus contains 36,592 accepted expert trajectories, with an average of 6.3 tool-invocation turns per trajectory (Chen et al., 6 May 2026).

4. Dataset composition, schema, and representative structure

SearchVL-SFT-36k is image-grounded by construction and spans seeds stratified over Person, Building/Place, Location, Organism, and Artifact, supplemented by LiveVQA, FVQA, and WebQA for live entities, commonsense fact lookup, and open-web multi-hop reasoning (Chen et al., 6 May 2026). The path-length distribution prior to rewriting and grounding follows the Wikipedia path sampling schedule $a_l = [z_l, c_l],$9 with probabilities $z_l$0. The paper does not report exact numerical breakdowns for pure text-only versus text-plus-image trajectories, OCR frequency, or per-tool usage counts.

Each example has a strict multi-turn ReAct schema. The core fields are summarized below.

Field	Content
id	Unique identifier
image_assets	Initial anchor image and subsequent image-valued observations
source_urls	Provenance for the anchor image
question	Fuzzed, visual-anchored VQA prompt
answer	Short ground-truth string
steps	Ordered turns with think, tool_call, and observation
final_response	Terminal answer text
schema flags	Formatting indicators and optional judge scores

Within steps, tool_call arguments reference images by ID, and Crop uses pixel coordinates in the form $z_l$1 (Chen et al., 6 May 2026). Observations are typed: TextSearch returns title/url/summary objects; ImageSearch returns similar image matches, entity candidates, and related URLs; OCR returns structured text blocks with labels and reading order. Image references are incremental: img_1 is the initial image, and subsequent visual-tool outputs become img_2, img_3, and so on.

The dataset also uses explicit formatting tokens > ..., <tool_call>...</tool_call>, and <response>...</response>, with supervision restricted to agent-generated spans (Chen et al., 6 May 2026). This formatting discipline aligns the SFT data with the RL stage’s masking and failure-handling assumptions.

The paper provides a representative example with identifier svl-sft-000123, anchored by an image of Australia Zoo. The question asks: “On what date did the wife of the man who took over management of the zoo in the image in 1991 become an Australian citizen?” The trajectory first crops the entrance text, then uses image search on the cropped sign to identify “Australia Zoo,” and finally uses text search to verify from the Terri Irwin page that she became an Australian citizen on “20 November 2009,” which is also the final response (Chen et al., 6 May 2026). The example illustrates the intended sequence: local visual inspection, visual retrieval for entity identification, and textual verification for the target fact.

5. Anti-shortcut design and quality control

The dataset’s main methodological claim is that its construction suppresses common shortcut regimes in multimodal QA. The paper identifies four principal mechanisms (Chen et al., 6 May 2026).

First, source-anchor grounding ensures that the image corresponds to the source node $z_l$2 rather than the answer node $z_l$3. This makes the image informative but not answer-coincident, so the agent must traverse bridge relations via text search instead of recovering the answer entity from reverse-image lookup.

Second, fuzzy entity rewriting removes explicit names from the question while enforcing uniqueness. The stated effect is to force relational identification rather than direct entity-name matching (Chen et al., 6 May 2026).

Third, difficulty filtering removes cases solvable by parametric knowledge or by a single ImageSearch call. This concentrates the training signal on genuinely tool-dependent, multi-hop trajectories.

Fourth, the enhancement subset requires agents to repair degraded visual evidence before downstream retrieval, avoiding exclusive reliance on an idealized vision encoder.

Quality control is correspondingly multi-layered. The VQA instances undergo automated gating for masking, uniqueness, and visual relevance; anchor images are CLIP-filtered; and instances are resampled if uniqueness fails (Chen et al., 6 May 2026). The trajectory synthesis stage adds answer-level and process-level judging. By contrast, deduplication and human verification are not explicitly described; the paper emphasizes automated LLM-as-judge filtering and environment consistency checks.

A likely misconception is that SearchVL-SFT-36k is merely a collection of multimodal question-answer pairs with tool traces attached. The paper instead presents it as a deliberately constrained corpus in which the question formation process, anchor selection, filtering stages, and trajectory judges are all coordinated to induce search behavior rather than shortcut retrieval (Chen et al., 6 May 2026).

6. Training usage, RL interface, and reported effects

In the OpenSearch-VL recipe, SearchVL-SFT-36k is used for full-parameter SFT on Qwen3-VL-8B-Instruct, Qwen3-VL-30B-A3B-Instruct, and Qwen3-VL-32B-Instruct (Chen et al., 6 May 2026). The representative SFT configuration uses bfloat16, DeepSpeed ZeRO-3, a context length of 32,000 tokens, image max pixels of approximately $z_l$4, videos disabled, an effective batch size of $z_l$5 GPUs times $z_l$6 sample per GPU, learning rate $z_l$7, cosine scheduler, warmup ratio $z_l$8, and $z_l$9 epochs. Infrastructure is listed as 256 H20 GPUs across 32 nodes with orchestration via Ray and DeepSpeed, and implementation is based on LlamaFactory extended with multi-turn tool-interleaved collators and Qwen3-VL-aware vision/text packing (Chen et al., 6 May 2026).

The SFT stage initializes the policy for subsequent reinforcement learning on SearchVL-RL-8k under a fatal-aware GRPO objective. Although RL uses a separate prompt pool, its design influences the SFT data structure. The composite RL reward is

$c_l$0

with $c_l$1 (Chen et al., 6 May 2026). Fatal-aware masking is defined as

$c_l$2

where $c_l$3 is the earliest step where $c_l$4 consecutive tool-execution errors begin. One-sided advantage clamping is

$c_l$5

and the final GRPO objective is

$c_l$6

These RL definitions explain why the SFT trajectories uniformly enforce clean formatting, one tool call per turn, and masking of exogenous observations (Chen et al., 6 May 2026).

The paper attributes substantial benchmark gains to the overall recipe using SearchVL-SFT-36k followed by RL. Across seven multimodal deep-search benchmarks—SimpleVQA, VDR, MMSearch, LiveVQA, BrowseComp-VL, FVQA, and InfoSeek—OpenSearch-VL reports over 10-point average improvements, and OpenSearch-VL-30B-A3B improves the Qwen3-VL-30B-A3B agentic baseline average from 47.8 to 61.6, with gains of +24.5 on MMSearch, +13.3 on VDR, +10.2 on FVQA, and +16.2 on InfoSeek (Chen et al., 6 May 2026). Pipeline ablations isolate the contribution of the curated SFT data: removing source-anchor grounding costs 11.5 average points, removing fuzzy entity rewriting costs 10.3, removing staged filtering costs 8.2, and removing the enhancement subset costs 1.3 (Chen et al., 6 May 2026). The paper interprets these results as evidence that source anchoring, fuzzing, and filtering are central to preventing shortcut retrieval, while enhancement mainly improves robustness.

7. Relation to prior datasets, limitations, and availability

The paper contrasts SearchVL-SFT-36k with prior multimodal search datasets or synthesis pipelines such as WebWatcher, MMSearch-R1, and Skywork-R1V4 (Chen et al., 6 May 2026). According to the comparison, earlier resources often ground near the answer, retain entity names, or assume pristine images, which makes single-hop retrieval or parametric answering viable. SearchVL-SFT-36k instead decouples visual anchoring from the answer entity, enforces name fuzzing with uniqueness checks, and includes image degradation paired with enhancement tools. The stated consequence is joint suppression of shortcuts together with explicit training for active perception (Chen et al., 6 May 2026).

Several limitations are also specified. The environment depends on third-party APIs, including search ranking and content-fetch systems, so trajectories may inherit ranking drift, fetch failures, or summarization hallucinations in observations (Chen et al., 6 May 2026). The synthesis pipeline uses proprietary GPT-4o and GPT-5.4 judges, which may be version-dependent and expensive to reproduce exactly. Coverage is biased by the English-language Wikipedia snapshot and Wikimedia imagery, although the inclusion of LiveVQA, FVQA, and WebQA is intended to mitigate this bias rather than eliminate it.

The recommended usage guidelines follow directly from these constraints. Training should use retrieved-token masking so that observation tokens are excluded from the loss. Formatting should preserve <think> first, one tool call per turn, and a final <response> only after sufficient evidence has been gathered. Extensions to the dataset should preserve the source-anchor grounding and fuzzy rewriting invariants, and changes to the tool environment should be accompanied by trajectory re-validation or prompt adaptation (Chen et al., 6 May 2026).

The authors state that they will release all data, code, and models under permissive licenses to support open research, with repositories at https://github.com/shawn0728/OpenSearch-VL and https://huggingface.co/OpenSearch-VL (Chen et al., 6 May 2026). The exact dataset license is not specified in the paper, so definitive license terms are deferred to the repository release. Provenance includes Wikipedia, Wikimedia Commons, infoboxes, LiveVQA, FVQA, and WebQA, while textual observations in trajectories may reflect outputs of third-party services such as Serper and JINA Reader as well as model-generated summaries (Chen et al., 6 May 2026).

In sum, SearchVL-SFT-36k is a supervised trajectory corpus engineered not simply for multimodal question answering, but for multimodal search behavior under tool use. Its defining characteristics are source-anchor visual grounding, fuzzy entity rewriting with invariants, explicit exclusion of observation tokens from the training loss, and multi-turn expert traces aligned with a downstream fatal-aware GRPO regime (Chen et al., 6 May 2026).