Tempo-R0: Multimodal Temporal Video Grounding

• The paper shows that Tempo-R0, a multimodal video grounding model, effectively employs adaptive attention and explicit timestamp alignment to localize query-relevant segments.
• It combines reinforcement-learning fine-tuning for refusal of irrelevant pairs with self-adaptive visual token allocation to address redundancy and subtle event boundaries.
• Empirical results demonstrate a roughly 3.5% improvement over previous state-of-the-art benchmarks while efficiently managing limited token budgets.

Searching arXiv for the relevant papers on Tempo-R0 and its related TEMPO framework. Tempo-R0 is a Video Multimodal LLM for temporal video grounding (TVG) that is designed to locate the start and end timestamps of query-relevant events in long, redundant videos. Its pipeline combines three elements: Self-adaptive Attention Allocation (SAA) for allocating visual capacity to informative frames, Explicit Timestamp-modal Aligned (ETA) timestamp tokenization for stronger temporal boundary perception, and Partial Irrelevance Refusing-based Group Relative Policy Optimization (PIR-GRPO) for reinforcement-learning fine-tuning that teaches both acceptance of relevant video–query pairs and refusal of irrelevant ones. The model is built on Qwen2-VL-7B and is reported to improve performance on both the original QVHighlights benchmark and a corrected version, cQvH, by around $3.5\%$ relative to prior state of the art (Yue et al., 7 Jul 2025).

1. Definition, scope, and nomenclature

In TVG, the input is a video and a natural-language query, and the output is a temporal segment represented as a start and end timestamp, written as $[St_i, Ed_i]$. Tempo-R0 frames this as a multimodal temporal sensing problem: the system must determine which frames matter, where event boundaries lie, and how temporal positions align with linguistic content. The paper identifies four specific obstacles: videos contain substantial redundancy, relevant events may occupy only a small fraction of the sequence, boundaries are often subtle, and Video-MLLMs operate under limited context or token budgets (Yue et al., 7 Jul 2025).

This formulation is narrower than generic video retrieval. The model is not described as retrieving semantically similar clips from a corpus; it is designed to ground a query inside a single video by producing timestamps. A plausible implication is that Tempo-R0 belongs most directly to the line of work on language-conditioned temporal localization rather than to broader multimodal retrieval systems.

The name also requires disambiguation. In later work on test-time training for large reasoning models, the phrase “Tempo-R0”-style test-time training is used in a different sense: it refers to a self-training regime that TEMPO later generalizes by alternating policy refinement on unlabeled test questions with critic recalibration on labeled data. That later usage is analytically distinct from the Video-MLLM system described here, even though the string “Tempo-R0” is shared across both contexts (Zhang et al., 21 Apr 2026).

2. Temporal video grounding as multimodal temporal sensing

Tempo-R0 is built on Qwen2-VL-7B. Its inference pipeline separates the input into three modalities—visual frames, timestamps, and the language query or instruction—preprocesses them independently, interleaves the resulting tokens, and feeds them into the Video-MLLM. The reported inference flow is: sample frames from the video; use SAA to determine how much visual resolution or token budget each frame should receive; convert each sampled frame timestamp into aligned text tokens through ETA; interleave visual and timestamp tokens; provide them with a TVG prompt to Qwen2-VL-7B; and output the matched start and end timestamps (Yue et al., 7 Jul 2025).

SAA is motivated by the observation that if a video has $T$ seconds and is sampled at $f$ fps, then the number of sampled frames is

$F=\lfloor T/f \rfloor.$

A model with limited visual context $L$ cannot encode every frame at high resolution. Tempo-R0 therefore estimates frame-to-frame variation and uses that signal to allocate more effective visual capacity to frames judged more informative. The method converts frames to HSL color space, uses only the hue channel, and computes an Optimal Transport-based score between adjacent frames. The visual-token construction is written as

$$Token_{v_i}=Visual\_Encode(Resize(F_i,R)), \qquad F_i \in F,$$

with the resize factor selected by

$$R= \begin{cases} R_l, & \text{if } OT(H(F_i))<T_{Key} \ R_s, & \text{otherwise} \end{cases} \qquad \text{with } R_l>R_s.$$

The paper states that OT is more sensitive than PHash and Optical Flow to new objects or exteriors appearing in a frame sequence, and it explicitly argues that merely increasing the total number of visual tokens does not guarantee better performance (Yue et al., 7 Jul 2025).

ETA addresses a different failure mode: MLLMs often encode time only implicitly through positional structure. Tempo-R0 instead treats timestamps as an explicit text modality. Each timestamp is converted into text and aligned so that timestamps with different digit lengths share a consistent token format by zero-padding or formatting alignment: $Token_{ts_i}=Text\_Encode(Align(I_{ts_i})).$ The paper’s claim is that explicit, aligned timestamp tokens are easier for the model to parse and associate with visual content than relying only on latent positional signals (Yue et al., 7 Jul 2025).

3. Architecture-level components

The three named components of Tempo-R0 are structurally complementary rather than redundant. SAA addresses visual redundancy and token-budget allocation; ETA addresses explicit temporal representation; PIR-GRPO addresses decision quality during fine-tuning by distinguishing between answerable and unanswerable video–query pairs.

Component	Operational role	Reported emphasis
SAA	Allocates visual capacity according to frame content variation	Efficient use of limited attention
ETA	Encodes timestamps as aligned text tokens	Stronger perception of event boundaries
PIR-GRPO	RL fine-tuning for grounding and refusal	Major gain driver in ablations

This decomposition matters because the paper does not attribute the model’s gains to a single intervention. SAA and ETA modify the input representation and tokenization pathway, whereas PIR-GRPO changes the training objective. This suggests a division between temporal sensing before the backbone and temporal decision-making during fine-tuning.

The architecture is still fundamentally a Video-MLLM system, not a bespoke temporal localization network with separate proposal-generation and ranking stages. A plausible implication is that the method aims to obtain temporal grounding behavior from a general multimodal foundation model by restructuring what information it sees and how it is rewarded, rather than by replacing the backbone with a specialized localizer.

4. Fine-tuning with refusal-aware reinforcement learning

Tempo-R0 uses a two-stage optimization strategy. In Stage 1, the pretrained Qwen2-VL-7B is supervised-fine-tuned on in-domain TVG data plus some out-of-domain data with similar content. The reported supervised fine-tuning sources are the QvHighlights training split, Charades-STA, and some similar out-of-domain data from InternVid. In Stage 2, the model is optimized with a reinforcement-learning-with-verifiable-reward setup using GRPO-style optimization, modified to handle irrelevant pairs. For this second stage, the training mix includes in-domain data plus about $10\%$ irrelevant video–query pairs (Yue et al., 7 Jul 2025).

The paper’s key claim is that TVG models should learn not only to accept relevant video–query pairs but also to refuse irrelevant ones. Without explicit refusal training, the model may hallucinate intervals for queries that do not match any valid moment in the video. PIR-GRPO operationalizes this through a reward with two parts: a format reward and a precision reward. The format reward is

$[St_i, Ed_i]$0

and the precision reward is

$[St_i, Ed_i]$1

where $[St_i, Ed_i]$2 rewards overlap with ground truth for relevant pairs and $[St_i, Ed_i]$3 rewards correct refusal on irrelevant pairs. The total reward is

$[St_i, Ed_i]$4

The paper describes $[St_i, Ed_i]$5 as the relevant-pair set, $[St_i, Ed_i]$6 as the irrelevant-pair set, and uses the indicator $[St_i, Ed_i]$7 for empty prediction or correct rejection (Yue et al., 7 Jul 2025).

This training design is presented as a way to reduce reward hacking in which the model emits plausible-looking timestamps even when no segment should be returned. The authors emphasize that this refusal capability is introduced through data composition and reward design rather than through architectural modification. That positioning distinguishes Tempo-R0 from methods that attempt to solve irrelevance handling by adding separate detection heads or calibration modules.

5. Empirical results, benchmarks, and ablations

Tempo-R0 is evaluated on QvHighlights, Charades-STA, and ActivityNet, and the paper also reports results on a corrected QvHighlights testbench, cQvH, produced because some original test annotations were judged inconsistent with human interpretation. The standard reported metrics are $[St_i, Ed_i]$8, $[St_i, Ed_i]$9, and $T$0. On QvHighlights, Tempo-R0 reports

$T$1

On cQvH, it reports

$T$2

The paper summarizes these as around $T$3 improvements over prior state of the art on both the original and corrected testbenches. On transfer from QvHighlights to ActivityNet, it reports

$T$4

and describes these numbers as outperforming BLIP-Chrono and slightly improving over VideoChat-R1 on the reported comparison (Yue et al., 7 Jul 2025).

The ablation study assigns the largest performance gain to PIR-GRPO. The baseline supervised-fine-tuned system gives $T$5 for $T$6; adding out-of-domain data raises this to $T$7; and adding PIR-GRPO yields $T$8. SAA provides a smaller but consistent benefit: with SAA, the model reaches $T$9, compared with $f$0 without SAA. ETA also helps, especially in $f$1: with ETA, the reported values are $f$2, versus $f$3 without it (Yue et al., 7 Jul 2025).

Two further ablation findings are central to the paper’s interpretation. First, increasing visual tokens from $f$4 to $f$5, $f$6, or $f$7 does not monotonically improve performance, which is presented as evidence that adaptive token placement matters more than brute-force token expansion. Second, the best negative-sample ratio is reported to be around $f$8: too little irrelevance weakens refusal learning, while too much harms positive grounding performance. These results support the paper’s argument that TVG in a Video-MLLM regime is constrained not only by model size but also by how temporal evidence and irrelevance are encoded during training.

6. Interpretation, limitations, and later reuse of the name

Tempo-R0’s main technical claim is that temporal video grounding is not only a retrieval problem but also a temporal sensing and refusal problem. In the paper’s framing, SAA determines where limited visual capacity should be spent, ETA determines how timestamps should be represented so that the model can read them explicitly, and PIR-GRPO determines when the model should answer and when it should refuse (Yue et al., 7 Jul 2025).

The limitations identified in the paper are practical rather than purely theoretical. Performance depends on the quality of sampling and on the OT-based key-frame detector. The balance of irrelevant training pairs is important, with the ablation indicating that the proportion cannot be increased arbitrarily. The release of cQvH also implies that evaluation is sensitive to annotation quality: small boundary inconsistencies can materially affect measured TVG performance. The reported results are tied to the Qwen2-VL-7B backbone, and portability to other Video-MLLMs is not established in the paper.

A further source of confusion is terminological. The later paper “TEMPO: Scaling Test-time Training for Large Reasoning Models” uses the expression “Tempo-R0”-style test-time training to refer to an earlier self-training regime in which the model improves on unlabeled test questions using its own evolving reward estimates, and positions TEMPO as the scalable version that adds periodic critic recalibration on labeled data through an EM-style actor–critic procedure. In that work, the defining distinction is not video grounding but alternating policy refinement on $f$9 with critic recalibration on $F=\lfloor T/f \rfloor.$0, which is reported to improve OLMO3-7B on AIME 2024 from $F=\lfloor T/f \rfloor.$1 to $F=\lfloor T/f \rfloor.$2 and Qwen3-14B from $F=\lfloor T/f \rfloor.$3 to $F=\lfloor T/f \rfloor.$4 while maintaining diversity (Zhang et al., 21 Apr 2026). This later reuse does not describe the Video-MLLM system itself; rather, it shows that “Tempo-R0” acquired a second, method-family meaning in adjacent arXiv discourse.

Within the TVG literature, however, Tempo-R0 is most precisely understood as the Qwen2-VL-7B-based Video-MLLM that combines adaptive visual token allocation, explicit timestamp alignment, and refusal-aware RL fine-tuning for temporal grounding. Its reported contribution is not a new generic theory of temporal reasoning, but a concrete systems design for long-video localization under token-budget constraints.