VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement

Abstract: As video corpora continue to expand in both scale and task complexity, there is increasing demand for approaches that retrieve relevant videos from large-scale corpora (inter-video reasoning) and subsequently perform fine-grained, query-conditioned tasks (intra-video reasoning) within the retrieved content, such as temporal grounding. However, existing approaches typically treat retrieval as a preprocessing step, and consequently, when the initial retrieval fails, there is no mechanism to refine the search, leading to the failure of subsequent fine-grained intra-video reasoning. Moreover, while recent agentic frameworks have advanced video understanding, they typically assume that the query-relevant video is already given, focusing exclusively on intra-video reasoning tasks. To address these limitations, we propose VideoSearch-R1, an agentic framework for iterative video retrieval and reasoning through multi-turn interaction with a video search engine. Specifically, we introduce Soft Query Refinement (SQR) to refine search query tokens in a continuous latent space rather than rewriting queries in the discrete text space, enabling more efficient and fine-grained adjustments. SQR and its reasoning process are trained using Group Relative Policy Optimization (GRPO), guided by task-level reward signals derived from retrieval and downstream tasks. Building upon this, VideoSearch-R1 achieves state-of-the-art performance across three datasets on Video Corpus Moment Retrieval (VCMR), iteratively retrieving videos from large-scale corpora, refining search queries, and performing precise query-conditioned temporal grounding within the retrieved content. Our analyses show that SQR effectively refines the original query, requiring significantly fewer generated tokens than explicit text-level query refinement. Code and model checkpoints are publicly available at mlvlab.github.io/VideoSearch-R1.

• The paper introduces a novel Soft Query Refinement module that iteratively improves video retrieval using latent-space adjustments and reinforcement learning.
• It unifies corpus-level search with intra-video temporal grounding to achieve state-of-the-art performance on challenging VCMR benchmarks.
• Extensive experiments demonstrate efficiency gains with fewer tokens and enhanced semantic matching over traditional hard query refinement methods.

VideoSearch-R1: An Agentic Framework for Iterative Video Retrieval and Reasoning via Soft Query Refinement

Overview

"VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement" (2607.00446) introduces an agentic AI system for joint retrieval and reasoning in large-scale video corpora. Unlike traditional pipelines that treat retrieval as a preprocessing stage and fail to recover from initial errors, VideoSearch-R1 implements multi-turn, interactive retrieval integrated with both symbolic and latent-space query refinement. The core innovation is the Soft Query Refinement (SQR) module, which generates continuous query tokens, optimizing retrieval via reinforcement learning with Group Relative Policy Optimization (GRPO). This latent-space update mechanism, in conjunction with a unified policy for both video-level search and intra-video (temporal grounding) reasoning, realizes state-of-the-art performance on challenging Video Corpus Moment Retrieval (VCMR) benchmarks.

Figure 1: An illustrative example of VideoSearch-R1's iterative video retrieval and reasoning, highlighting multi-turn interaction with an external video search engine, unifying corpus-level retrieval and intra-video temporal grounding.

Motivation

In practice, real-world multimodal retrieval tasks increasingly require not only robust inter-video identification, but also high-fidelity intra-video reasoning—such as temporal localization—conditioned on user queries. Prevailing systems adopt rigid pipelines, decoupling retrieval and temporal reasoning stages and thus accumulating errors through non-interactive retrieval. Recent advances in Retrieval-Augmented Generation (RAG) for text (e.g., Search-R1, Self-RAG) leverage interaction with search engines to correct errors and iteratively refine inference [jin2025search, asai2023self], but video agent frameworks have not explicitly integrated this multi-turn interaction, often assuming the correct video has already been retrieved [luo2024video, ren2025videorag]. VideoSearch-R1 fills this gap by introducing an interactive, RL-trained agentic system for end-to-end video corpus reasoning and retrieval.

System Architecture

VideoSearch-R1 operates in a multi-turn agentic loop. At each turn, given a query:

1. Video Retrieval: Uses the current query representation to retrieve a candidate video via a cross-modal embedding search engine (Qwen3-VL-Embedding-2B).
2. Verification: Evaluates semantic match between the query and retrieved video content; produces a stepwise reasoning trace and a binary (match/not match) decision.
3. Soft Query Refinement (SQR): For negative matches, SQR autoregressively generates a fixed-length sequence of soft query tokens in continuous latent space, appended to the original query to steer retrieval.
4. Iteration: The refined query is resubmitted to the search engine, and the process repeats for up to $T$ turns or until a matching video is found.
5. Temporal Grounding: After a positive match, the model predicts fine-grained temporal boundaries within the retrieved video segment most relevant to the user query.

   Figure 3: VideoSearch-R1's iterative retrieval and reasoning loop, showing verification, soft query refinement, and query updating across interaction turns.

Unlike hard query refinement (HQR), which re-writes queries in discrete token space, SQR’s latent-space updates enable compact and fine-grained representational adjustments, minimizing semantic noise and token bloat. SQR is jointly trained with the reasoning process via GRPO, optimizing policy trajectories that maximize both retrieval and downstream task rewards.

Training Paradigm

The VideoSearch-R1 training pipeline consists of two stages:

• Supervised Fine-Tuning (SFT): The model is initialized via SFT, leveraging structured reasoning traces (chain-of-thought) and ground-truth labels for query-video alignment, temporal tagging, and contrastive (InfoNCE-based) optimization of soft query tokens.
• Reinforcement Learning via GRPO: A custom RL framework with multiple reward channels (format, verification, retrieval, and temporal grounding IoU) is applied to holistically optimize both retrieval and reasoning policies.

Tabular ablations confirm SFT yields structural benefits, but joint optimization with GRPO substantially boosts moment localization and verification performance.

Soft Query Refinement: Effectiveness and Analysis

A major contribution is the demonstration that SQR, operating in the continuous latent embedding space, is more compact and effective than explicit textual (hard) query rewriting.

Figure 2: Visual comparison of hard query refinement (left) versus SQR (right), illustrating fine-grained, efficient continuous-space query adjustments.

Figure 4: R@1 performance increases as more soft query tokens are appended, confirming the incremental value of SQR.

Figure 5: The search rank of the ground-truth video improves as soft tokens are appended, demonstrating the effectiveness of fine-grained semantic adjustment.

Comparative results on ActivityNet-FIG show SQR outperforms HQR in video retrieval accuracy (R@1: 61.1 vs. 57.6) while using only 8 latent tokens versus HQR’s average of 26.8 tokens. Qualitative analysis suggests that HQR’s verbose rewrites introduce semantic confusion into embedding-based retrieval, whereas SQR’s token-efficient refinements directly optimize for cross-modal alignment.

Experimental Evaluation

VideoSearch-R1 is evaluated on ActivityNet-FIG, DiDeMo-FIG, and Charades-FIG under VCMR and video retrieval (VR) metrics (e.g., Recall@$K$, IoU/R@1). Baselines include zero-shot and fine-tuned Qwen3-VL-2B models. VideoSearch-R1 achieves clear gains in both single-video retrieval and cross-video moment localization. For example, on DiDeMo-FIG, VideoSearch-R1 delivers a +9.7 improvement in 0.3/R@1 compared to the fine-tuned baseline.

Ablation studies verify the individual contributions of each RL reward: retrieval rewards ($R^\text{ret}$) enhance VR, verification rewards ($R^\text{verif}$) boost semantic matching accuracy, and temporal grounding rewards ($R^\text{time}$) refine moment localization with acceptable trade-offs.

Figure 6: Qualitative comparison—SQR enables more precise query-targeted retrieval using a small latent token budget compared to hard query refinement.

Further, the system converges in 2-3 refinement turns, illustrating the efficiency of the interactive process.

Implications and Future Directions

VideoSearch-R1 establishes a unified agentic framework for corpus-level video understanding, demonstrating the viability of continuous query optimization for retrieval tasks. From a practical standpoint, its compact query refinements translate to reduced computational overhead and lower traffic between modules. Its ability to self-correct initial retrieval failures addresses persistent weaknesses in traditional one-shot retrieval/grounding pipelines.

Theoretically, the success of latent-space query refinement for multi-modal retrieval suggests promising avenues for RL-optimized latent control in broader vision-language and multi-modal applications. Open research questions include adaptive inference with dynamic turn allocation, hierarchical reasoning across nested multi-modal corpora, and abstraction of SQR-like mechanisms to other media (e.g., audio, multimodal documents).

Conclusion

VideoSearch-R1 introduces an interactive, RL-optimized agentic system that unifies inter-video retrieval and intra-video reasoning through soft query refinement in latent space. This paradigm achieves state-of-the-art performance on VCMR, demonstrating that compact, policy-driven query optimization in the continuous embedding space offers significant advantages for scalable, fine-grained video corpus reasoning (2607.00446).