VideoSearch-R1: Agentic Video Retrieval & Reasoning

• The paper introduces a novel agentic framework that jointly optimizes inter-video retrieval and intra-video temporal grounding using soft query refinement.
• VideoSearch-R1 refines queries in a continuous latent space, enabling iterative correction and improved localization of relevant video segments.
• The system leverages reinforcement learning with Group Relative Policy Optimization to align retrieval, verification, and temporal grounding for state-of-the-art performance.

VideoSearch-R1 is an agentic framework for iterative video retrieval and reasoning through multi-turn interaction with a video search engine. It is designed for settings in which a system must retrieve relevant videos from large-scale corpora and then perform fine-grained, query-conditioned tasks within the retrieved content, such as temporal grounding. The framework addresses two limitations identified in prior work: retrieval is often treated as a preprocessing step with no mechanism to refine failed initial search, and recent agentic video-understanding systems typically assume that the query-relevant video is already given. VideoSearch-R1 responds by jointly optimizing inter-video retrieval and intra-video temporal grounding, using Soft Query Refinement (SQR) to refine search query tokens in a continuous latent space rather than rewriting queries in the discrete text space (Lee et al., 1 Jul 2026).

1. Problem setting and system scope

VideoSearch-R1 is framed around Video Corpus Moment Retrieval (VCMR), where the objective is not only to identify the correct video in a corpus but also to localize the relevant temporal segment within that video. In the formulation reported for the method, the system iteratively retrieves videos from large-scale corpora, refines search queries, and performs precise query-conditioned temporal grounding within the retrieved content (Lee et al., 1 Jul 2026).

The motivating diagnosis is explicit. Existing approaches typically treat retrieval as a preprocessing step, and consequently, when the initial retrieval fails, there is no mechanism to refine the search. At the same time, recent agentic frameworks have advanced video understanding but typically assume that the query-relevant video is already given, focusing exclusively on intra-video reasoning tasks. VideoSearch-R1 reorganizes this pipeline into a coupled loop in which retrieval failure is not terminal but triggers a learned refinement step (Lee et al., 1 Jul 2026).

This places the framework in a lineage of interactive retrieval systems, but with a different optimization target. Earlier interactive VCMR work modeled known-item search as a Markov Decision Process over a graph of clips and learned navigation with user feedback and actor-critic reinforcement learning (Ma et al., 2023). VideoSearch-R1 instead refines the query representation itself in latent space, then re-invokes a pre-trained cross-modal dense retriever, coupling retrieval, verification, and temporal grounding in a single agentic loop (Lee et al., 1 Jul 2026).

2. Iterative architecture and inference loop

At inference time, VideoSearch-R1 operates in a loop of at most $T = 2$ retrieval turns. At turn $t$, the input is the original text query $q_1$. The system invokes a pre-trained cross-modal dense retriever $\mathcal{R}$, specifically Qwen3-VL-Embedding-2B, to obtain the top-1 video

$v_t = \arg \max_{v\in V} f(q_t)^\top f(v).$

The retrieved video is then passed to a vision-LLM, Qwen3-VL-2B-Instruct, which compares $q_t$ against $v_t$ and produces a reasoning trace $r_t$ together with a binary match indication $$y_t^{\mathrm{ret}}\in\{\mathrm{match},\mathrm{not\ match}\}$$ (Lee et al., 1 Jul 2026).

If $y_t^{\mathrm{ret}}=\mathrm{match}$, the system proceeds to temporal grounding and predicts start and end timestamps $t$0. If the retrieved video is judged to be a mismatch, the model generates $t$1 soft tokens $t$2, appends them to the original query embedding, forms $t$3, and repeats the retrieval step. The loop terminates either when a match is confirmed or after $t$4 turns (Lee et al., 1 Jul 2026).

The architecture therefore decomposes each turn into three operational stages: retrieval, verification-and-reasoning, and either temporal grounding or latent query refinement. This suggests a tighter dependency between inter-video and intra-video reasoning than in cascaded systems where retrieval and grounding are optimized separately. A related contrast is visible in systems such as ForeSea, which also use a staged design—tracking/filtering, multimodal embedding and indexing, then retrieval plus VideoLLM inference—but are organized around multimodal surveillance queries rather than iterative latent query refinement (Park et al., 24 Mar 2026).

3. Soft Query Refinement in latent space

The central mechanism of VideoSearch-R1 is Soft Query Refinement. Rather than rewriting the query in discrete text space, SQR adds a small continuous update to the original query representation. Let

$t$5

be the embedding sequence of the original query of length $t$6. At a mismatch turn $t$7, the model autoregressively produces $t$8 soft embeddings

$t$9

by letting the hidden state $q_1$0 project to the next embedding $q_1$1. The refined query embedding is

$q_1$2

This construction preserves the original query while appending a learned continuous correction (Lee et al., 1 Jul 2026).

During Supervised Fine-Tuning, these soft tokens are trained with an InfoNCE contrastive loss that encourages the refined query to be close to the ground-truth video embedding $q_1$3 and far from negative videos $q_1$4:

$q_1$5

The one-turn SFT objective is

$q_1$6

where $q_1$7 and $q_1$8 (Lee et al., 1 Jul 2026).

A direct empirical comparison is reported between SQR and hard query refinement. On ActivityNet-FIG, “VideoSearch-R1 + HQR” attains VR R@1 of $q_1$9 with $\mathcal{R}$0 generated tokens, whereas “VideoSearch-R1 + SQR” attains VR R@1 of $\mathcal{R}$1 with $\mathcal{R}$2 generated tokens. The analysis in the paper states that SQR outperforms hard text-level rewriting despite generating $\mathcal{R}$3 fewer tokens (Lee et al., 1 Jul 2026).

4. Reinforcement learning with Group Relative Policy Optimization

After SFT, VideoSearch-R1 is further optimized with reinforcement learning using Group Relative Policy Optimization (GRPO). The aim is to align retrieval/refinement behavior and temporal grounding through task-level reward signals derived from retrieval and downstream tasks. Four reward signals per sample $\mathcal{R}$4 are defined: a format reward, a verification reward, a retrieval reward, and a temporal grounding reward (Lee et al., 1 Jul 2026).

The rewards are specified as follows. The format reward is

$\mathcal{R}$5

if the model output strictly follows the > …, <answer>…</answer>, <start>…</start>, <end>…</end> template, else $\mathcal{R}$6. The verification reward is

$\mathcal{R}$7

if $\mathcal{R}$8 is correct, else $\mathcal{R}$9. The retrieval reward is

$v_t = \arg \max_{v\in V} f(q_t)^\top f(v).$0

The temporal grounding reward is

$v_t = \arg \max_{v\in V} f(q_t)^\top f(v).$1

if $v_t = \arg \max_{v\in V} f(q_t)^\top f(v).$2, else $v_t = \arg \max_{v\in V} f(q_t)^\top f(v).$3. The total reward is

$v_t = \arg \max_{v\in V} f(q_t)^\top f(v).$4

Advantages are normalized within each mini-batch:

$v_t = \arg \max_{v\in V} f(q_t)^\top f(v).$5

GRPO then updates the policy $v_t = \arg \max_{v\in V} f(q_t)^\top f(v).$6 with a clipped surrogate objective and a KL penalty:

$v_t = \arg \max_{v\in V} f(q_t)^\top f(v).$7

Here $v_t = \arg \max_{v\in V} f(q_t)^\top f(v).$8, $v_t = \arg \max_{v\in V} f(q_t)^\top f(v).$9 is the PPO clip, and $q_t$0 is a small weight (Lee et al., 1 Jul 2026).

In comparative terms, this RL formulation differs from prior interactive VCMR work that used advantage actor-critic over a graph neighborhood of candidate moments, with rewards defined by shortest-path distance to the target moment (Ma et al., 2023). It also differs from TimeSearch-R, which interleaves text reasoning and tool-based temporal search and augments GRPO with Completeness Self-Verification to supervise the adequacy of searched frames (Pan et al., 7 Nov 2025). VideoSearch-R1’s distinctive feature is that the RL loop optimizes latent query refinement itself, rather than only navigation policy or temporal search policy (Lee et al., 1 Jul 2026).

5. Experimental configuration and benchmark results

The reported evaluation covers three VCMR benchmarks: ActivityNet-FIG, DiDeMo-FIG, and Charades-FIG. ActivityNet-FIG contains $q_t$1 K videos with $q_t$2 train, $q_t$3 val, and $q_t$4 test, and average duration $q_t$5 s. DiDeMo-FIG contains $q_t$6 K videos with $q_t$7 train, $q_t$8 val, and $q_t$9 test, and average duration $v_t$0 s. Charades-FIG contains $v_t$1 K videos with $v_t$2 train, $v_t$3 val, and $v_t$4 test, and average duration $v_t$5 s. Metrics are Video Retrieval Recall@K for $v_t$6, end-to-end IoU/R@1 thresholds $v_t$7, and verification accuracy (Lee et al., 1 Jul 2026).

Implementation details are similarly specific. The search engine is Qwen3-VL-Embedding-2B, the agent model uses a Qwen3-VL-2B-Instruct backbone, the number of soft tokens is $v_t$8, and the visual token budget is $v_t$9 tokens using $r_t$0 FPS up to $r_t$1 frames. SFT is run for $r_t$2 K steps on $r_t$3 K samples per dataset with learning rate $r_t$4. RL with GRPO uses learning rate $r_t$5, KL $r_t$6, rollout $r_t$7, and max gradient norm $r_t$8. Inference uses $r_t$9 turns (Lee et al., 1 Jul 2026).

The main reported results on IoU/R@1 at threshold $$y_t^{\mathrm{ret}}\in\{\mathrm{match},\mathrm{not\ match}\}$$0, verification accuracy, and VR R@1 are as follows:

Dataset	Method	VCMR(.5/R@1)	VER	VR R@1
Charades-FIG	Qwen3-VL-2B ZS	7.2	30.0	21.6
Charades-FIG	Qwen3-VL-2B FT	10.4	74.7	21.6
Charades-FIG	VideoSearch-R1	13.4	75.7	24.6
DiDeMo-FIG	Qwen3-VL-2B FT	22.1	73.1	54.8
DiDeMo-FIG	VideoSearch-R1	30.2	74.6	59.0
ActivityNet-FIG	Qwen3-VL-2B FT	19.2	83.1	55.1
ActivityNet-FIG	VideoSearch-R1	22.3	83.3	61.1

These results are reported as state-of-the-art performance across the three datasets (Lee et al., 1 Jul 2026).

The ablations are equally central to interpreting the model. On DiDeMo-FIG, Stage1 (SFT only) yields $$y_t^{\mathrm{ret}}\in\{\mathrm{match},\mathrm{not\ match}\}$$1/R@1 of $$y_t^{\mathrm{ret}}\in\{\mathrm{match},\mathrm{not\ match}\}$$2 and VR R@1 of $$y_t^{\mathrm{ret}}\in\{\mathrm{match},\mathrm{not\ match}\}$$3, while adding Stage2 (GRPO) raises these to $$y_t^{\mathrm{ret}}\in\{\mathrm{match},\mathrm{not\ match}\}$$4/R@1 of $$y_t^{\mathrm{ret}}\in\{\mathrm{match},\mathrm{not\ match}\}$$5 and VR R@1 of $$y_t^{\mathrm{ret}}\in\{\mathrm{match},\mathrm{not\ match}\}$$6. Reward ablation on the same dataset shows that using only $$y_t^{\mathrm{ret}}\in\{\mathrm{match},\mathrm{not\ match}\}$$7 drops $$y_t^{\mathrm{ret}}\in\{\mathrm{match},\mathrm{not\ match}\}$$8/R@1 to $$y_t^{\mathrm{ret}}\in\{\mathrm{match},\mathrm{not\ match}\}$$9; adding $y_t^{\mathrm{ret}}=\mathrm{match}$0 keeps $y_t^{\mathrm{ret}}=\mathrm{match}$1/R@1 at $y_t^{\mathrm{ret}}=\mathrm{match}$2 but raises VER to $y_t^{\mathrm{ret}}=\mathrm{match}$3; adding $y_t^{\mathrm{ret}}=\mathrm{match}$4 raises $y_t^{\mathrm{ret}}=\mathrm{match}$5/R@1$y_t^{\mathrm{ret}}=\mathrm{match}$630.2%%%%97$t$98%%%%\approx 55%%%%99$$y_t^{\mathrm{ret}}\in\{\mathrm{match},\mathrm{not\ match}\}$$100%%%%\approx 60$t$01N=8$$(<a href="/papers/2607.00446" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Lee et al., 1 Jul 2026</a>).</p> <h2 class='paper-heading' id='interpretation-qualitative-behavior-and-relation-to-adjacent-systems'>6. Interpretation, qualitative behavior, and relation to adjacent systems</h2> <p>The qualitative examples included with VideoSearch-R1 illustrate the intended function of SQR. In one case, an initial retrieval returns a video with “man brushing shoe outdoors”; the verification trace notes missing “dark-gray T-shirt” and “indoor background”; mismatch then triggers soft-embedding generation that steers the retriever to the correct video, followed by final grounding with IoU$$t$02. Another example, “woman brushing hair,” shows that as $t$03 grows, the rank of the ground-truth video improves from no soft tokens to $t$04 (Lee et al., 1 Jul 2026).

Within the broader literature of video search, VideoSearch-R1 occupies a specific position. Interactive Video Corpus Moment Retrieval using Reinforcement Learning addressed known-item VCMR by planning navigation paths from user feedback over a clip graph, and showed that targets hidden deep in static ranks could be surfaced in a handful of feedback-driven steps (Ma et al., 2023). TimeSearch-R reformulated temporal search as interleaved text-video thinking with tool calls and GRPO-CSV, focusing on adaptive frame search inside long videos rather than retrieval refinement across a corpus (Pan et al., 7 Nov 2025). ForeSea, in turn, paired a tracking frontend, a multimodal embedding index, and a VideoLLM backend for surveillance QA with image-and-text queries and temporal grounding (Park et al., 24 Mar 2026). VideoSearch-R1 differs from each of these by treating failed corpus-level retrieval as a first-class reasoning event and by refining the query in latent space rather than navigating only the result graph, only the timeline, or only a multimodal index (Lee et al., 1 Jul 2026).

A plausible implication is that VideoSearch-R1 narrows the historical separation between search and reasoning in video systems. Earlier pipelines frequently placed retrieval, reranking, and downstream reasoning in disjoint modules; VideoSearch-R1 instead treats retrieval failure as an input to learned reasoning and uses that reasoning to modify the representation queried against the retriever. The reported gains over fine-tuned baselines, the reward ablations, and the SQR-versus-HQR comparison all point in that direction, while the hard cap of $t$05 turns shows that the method remains intentionally lightweight rather than open-ended (Lee et al., 1 Jul 2026).