TempSamp-R1: RL for Video Temporal Grounding

• The paper introduces a reinforcement learning framework that combines on-policy sampling with off-policy guidance to precisely localize video segments.
• It integrates a non-linear soft advantage computation and hybrid Chain-of-Thought training to mitigate sparse rewards and refine temporal decisions.
• State-of-the-art results on Charades-STA, ActivityNet Captions, and QVHighlights validate its effectiveness in handling large temporal search spaces.

TempSamp-R1, also written TeSamp-R1, is a reinforcement fine-tuning framework for adapting multimodal LLMs to video temporal grounding, the task of grounding a natural-language query to a precise time interval in a long, untrimmed video. It is designed for settings in which the temporal search space is large, rewards are sparse, and standard on-policy reinforcement learning methods such as Group Relative Policy Optimization (GRPO) often fail to identify temporally accurate solutions. The framework combines on-policy reinforcement learning with off-policy supervision from ground-truth temporal annotations, introduces a non-linear soft advantage computation method with an asymmetric transformation, and trains a single unified model to support both Chain-of-Thought (CoT) and non-CoT inference modes. Reported results establish new state-of-the-art performance on Charades-STA, ActivityNet Captions, and QVHighlights, and also indicate robust few-shot generalization under limited data (Li et al., 22 Sep 2025).

1. Problem setting and motivation

TempSamp-R1 is formulated for temporal video grounding, also called moment retrieval. The input is a long, untrimmed video $V$ and a natural-language query $q$; the output is a temporal segment $[t_s, t_e]$ such that $0 \le t_s < t_e \le T$, where $T$ is the video duration. Evaluation is based on overlap with the ground-truth segment $[t_s^\*, t_e^\*]$, typically measured by intersection over union (IoU):

$\text{IoU}([t_s,t_e],[t_s^\*,t_e^\*]) = \frac{\min(t_e,t_e^\*) - \max(t_s,t_s^\*)} {\max(t_e,t_e^\*) - \min(t_s,t_s^\*)}.$

The central difficulty is the size of the temporal search space. In long videos, the number of possible intervals is enormous, and most intervals are irrelevant. This produces sparse rewards, because only segments near the ground truth yield high IoU, and it also creates an exploration problem: when a generic MLLM is fine-tuned by purely on-policy RL, sampled intervals are frequently far from the correct location, especially early in training. In this setting, relative comparisons among low-quality samples provide weak information about where the correct interval lies.

The framework is motivated by specific shortcomings attributed to GRPO-style training in temporal grounding. GRPO samples candidate outputs from the current policy, computes rewards, derives group-relative advantages, and updates the policy on the basis of those on-policy samples. In large temporal spaces, this strategy is described as inefficient and limited in performance, because it often fails to identify temporally precise solutions. The resulting advantages are noisy, the gradients provide little signal toward precise timestamps, and the model may learn coarse localization without systematically refining temporal boundaries. The problem is especially acute under steep evaluation criteria such as [email protected], where small boundary errors can change the reward from success to failure (Li et al., 22 Sep 2025).

2. Architecture and representation of temporal decisions

TempSamp-R1 is built on Qwen2.5-VL-7B-Instruct. The model uses a video encoder that maps input frames or clips $\{f_i\}$ to visual tokens $\{v_1, v_2, \dots, v_M\}$, and a transformer LLM that consumes both the query text and the video tokens and autoregressively generates an output sequence $\mathbf{y} = (y_1,\dots,y_L)$.

A defining design choice is to cast temporal grounding as structured language output. Rather than predicting timestamps through a separate regression head, the model emits timestamps or segment indices in a fixed textual format. In non-CoT mode, the output can be a direct answer such as Start: 32.5s, End: 40.0s. In CoT mode, reasoning is generated inside > ... and the final timestamps are produced inside <answer>...</answer>. The model prediction is then parsed into $q$0, while the ground-truth segment remains $q$1.

This formulation permits a unified policy over multimodal reasoning and temporal decision making. The task reward is computed from the agreement between the predicted and ground-truth segments, using IoU or a metric derived from IoU:

$q$2

The reported objective is to maximize expected reward over prompts and videos:

$q$3

Within this setup, timestamps are not ancillary metadata but the primary task decision encoded in the generated text. This makes the temporal grounding problem directly compatible with RL-style sequence optimization, while preserving compatibility with CoT prompting and standard next-token supervision (Li et al., 22 Sep 2025).

3. Reinforcement fine-tuning formulation

TempSamp-R1 treats each full generation as an episodic RL trajectory. The state $q$4 consists of the video, the query, and the partial decoded sequence. Each generated token is an action, while the final decoded timestamps $q$5 constitute the task decision. The policy is the MLLM conditional distribution $q$6. Rewards are computed from the predicted temporal segment relative to the ground truth, with either thresholded forms such as [email protected] or continuous IoU-based variants.

The central modification to GRPO-style fine-tuning is the introduction of off-policy supervision from ground-truth timestamps. Ground-truth segments are converted into supervised target sequences $q$7 in the required output format, and the model is trained with next-token cross-entropy on those targets. This supervised term is explicitly off-policy, because the sequences are not sampled from the current policy. Its role is to provide temporally precise guidance in cases where on-policy exploration fails to discover accurate segments.

The framework therefore optimizes a mixed objective that combines supervised guidance and RL:

$q$8

Here, $q$9 is the standard supervised loss on ground-truth temporal sequences, and $[t_s, t_e]$0 is a policy-gradient loss weighted by an advantage term. In conceptual terms, the supervised component fills the gaps created by sparse and misaligned on-policy samples, while the RL component preserves the ability to explore and improve beyond direct imitation.

The paper describes this combination as a response to a task-specific failure mode of on-policy learning in long videos. On-policy samples may hover near approximate regions without converging to accurate boundaries, whereas off-policy ground-truth sequences directly push the model toward exact timestamps. This suggests a hybrid regime in which RL provides exploration and reward optimization, while ground-truth annotations provide a stable attractor for temporally precise behavior (Li et al., 22 Sep 2025).

4. Soft advantage shaping and hybrid CoT training

A second core component is the replacement of standard linear group-relative advantages with a non-linear soft advantage computation method that dynamically reshapes reward feedback via an asymmetric transformation. In GRPO or PPO-like methods, advantages are often computed as reward minus a group mean or other baseline. In temporal grounding, where most sampled intervals may be poor and rewards can be sparse, this linear form has high variance and supplies weak learning signals.

TempSamp-R1 instead uses a soft advantage that is described as asymmetric: it suppresses large negative signals, preventing the model from overreacting to moderate errors, and amplifies reliably good outcomes without exploding gradients. The exact formula is not printed in the provided excerpt, but the function is characterized as non-linear and asymmetric, with the stated purpose of stabilizing training and reducing variance in reward-based updates. The intended effect is variance reduction and more effective reward shaping for precise temporal localization.

The third major component is a hybrid CoT training paradigm. TempSamp-R1 trains one unified model to support both CoT and non-CoT modes. In CoT mode, the model is prompted to think step-by-step and produce intermediate reasoning before the final temporal answer. In non-CoT mode, it is prompted to output timestamps directly with minimal reasoning. During training, both formats are presented, and the model is exposed to both reasoning-rich and direct-answer patterns.

This unified training regime is meant to support different inference-time requirements. CoT mode is associated with complex queries, multi-event reasoning, or ambiguity resolution, whereas non-CoT mode is suited to simpler queries and lower latency. The framework therefore does not treat reasoning traces as a separate specialist capability; instead, it optimizes a single policy that can respond in either mode depending on the prompt. A plausible implication is that the method seeks to preserve the performance benefits of reasoning while avoiding a compulsory CoT overhead for all queries (Li et al., 22 Sep 2025).

5. Training procedure, benchmarks, and empirical findings

The high-level training loop alternates between an off-policy supervised step and an on-policy sampling step. Ground-truth sequences are built in both CoT and non-CoT formats. The current policy then samples outputs under mixed prompt modes, predicted timestamps are parsed, IoU is computed against the ground truth, rewards are assigned, a baseline is computed over the sampled group, soft advantages are derived, and the supervised and RL losses are combined for a joint parameter update.

The reported training configuration is specific. The base model is Qwen2.5-VL-7B-Instruct; fine-tuning is full-parameter rather than LoRA-based; the per-device batch size is 1; gradient accumulation is 2, giving an effective batch size of 2; training lasts 2 epochs; the optimizer is AdamW; Deepspeed ZeRO-3 Offload is used; the learning rate is $[t_s, t_e]$1; the maximum generated sequence length is 2048 tokens; and training is run on 4× A100 (80 GB).

The evaluation uses [email protected], [email protected], and mAP, depending on the dataset. Charades-STA is described as containing indoor videos and testing fine-grained grounding in relatively short but event-dense videos, with approximately 12.4k training and 3.7k validation examples. ActivityNet Captions consists of diverse, long untrimmed videos with multiple event captions per video, with 37,421 training, 17,505 validation, and 17,031 test samples. QVHighlights is a query-conditioned highlight detection benchmark with 10,148 videos of fixed 150s duration, 10,310 queries, and 18,367 highlights.

The main reported results are as follows:

Dataset	Metric	TempSamp-R1
Charades-STA	[email protected]	52.9% (+2.7%)
ActivityNet Captions	[email protected]	56.0% (+5.3%)
QVHighlights	mAP	30.0% (+3.0%)

These results are reported as new state-of-the-art performance over GRPO-based baselines. The gains are interpreted in the source as evidence of better precise localization, especially at high IoU thresholds, robust improvement across both grounding and highlight detection tasks, and particularly strong benefit on ActivityNet Captions, where long videos and diverse events enlarge the temporal search space.

The paper also reports ablations on the principal design components. Removing the ground-truth-based supervised term leads to a notable performance drop, especially on high-IoU metrics such as [email protected], supporting the claim that off-policy supervision is crucial for boundary refinement. Replacing the soft advantage with a standard linear advantage makes training less stable, with higher variance in training curves and more frequent reward plateaus or collapses. Training only CoT or only non-CoT is reported to be inferior to hybrid training: CoT-only is slower at inference and may degrade on simple queries, whereas non-CoT-only is faster but less robust on complex multi-event queries and ambiguous scenarios.

Qualitative analysis identifies both success and failure modes. Success cases include precise localization on Charades-STA and ActivityNet Captions, accompanied by CoT explanations that align visual events to timestamps. Failure cases often involve ambiguous visual content, repeated events, or loosely defined ground-truth boundaries. The appendix-level interpretation is that some errors reflect intrinsic annotation noise and ambiguity in temporal grounding benchmarks rather than only model deficiencies. The experiments further indicate robust few-shot generalization, with consistent improvements over GRPO baselines in few-shot settings and generalization to highlight detection without task-specific architectural changes (Li et al., 22 Sep 2025).

6. Limitations and place within the R1-style RL literature

The reported limitations are tied directly to the mechanism of the method. First, TempSamp-R1 depends on high-quality off-policy supervision, because it uses accurate ground-truth timestamps as explicit guidance. In weakly labeled or noisy settings, this supervised signal may mislead the model. Second, the empirical scope is concentrated on temporal grounding and highlight detection, so effectiveness on other temporal reasoning tasks such as multi-event tracking or video question answering with complex temporal dependencies remains untested. Third, the method is computationally demanding: it requires full-parameter fine-tuning of a 7B video-LLM with RL, alongside ZeRO-3 offloading and 4× A100 hardware.

Within the broader R1-style RL landscape, TempSamp-R1 is presented as a domain-specific extension of GRPO-based post-training rather than a generic preference-alignment method. Like other R1-style systems, it uses RL post-training, group-comparative or policy-gradient updates, and reward shaping. Unlike canonical RLHF or RLAIF setups, its rewards are task-intrinsic, arising from temporal overlap metrics rather than human or AI preference judgments. The combination of task-specific rule-based rewards and explicit off-policy supervision places it near a line of work that uses environment-grounded or structurally verified feedback rather than subjective preference models.

Two related 2025 systems illustrate this broader context. "Time-R1" develops comprehensive temporal reasoning in a 3B LLM through a multi-stage RL curriculum with dynamic rule-based rewards and GRPO, showing that R1-style training can target temporal skills without architectural time modules (Liu et al., 16 May 2025). "Repair-R1" applies GRPO to automated program repair by making test generation precede repair and by co-optimizing both behaviors through rule-based execution feedback, thereby exemplifying a non-preference, environment-driven RL design (Hu et al., 30 Jul 2025). Against that background, TempSamp-R1 can be understood as a video-temporal analogue: it addresses a large structured search space in which pure on-policy sampling is weak, then compensates with off-policy supervision, soft advantage shaping, and a unified CoT/non-CoT policy.

Possible extensions listed in the source include action detection, event segmentation, video question answering with temporal grounding, multi-event tracking, audio-video grounding, and weakly supervised or self-supervised variants that rely on pseudo-labels or human/AI feedback. These are presented as compatible directions rather than validated outcomes. A plausible implication is that the main contribution of TempSamp-R1 is not a new video backbone, but a training recipe for making MLLMs better at precise temporal decision-making when the reward landscape is sparse, the search space is large, and exact ground-truth segments are available (Li et al., 22 Sep 2025).