ReWatch-R1: Advanced Video Reasoning Model

• The paper introduces ReWatch-R1, a model that leverages a three-part ReWatch dataset and a multi-agent re-watching framework to improve complex video reasoning.
• It employs a structured reinforcement learning reward (O&R reward) to ensure that both final answers and intermediate observations are well grounded in video content.
• Empirical results show state-of-the-art performance on challenging video reasoning benchmarks while mitigating the data bottlenecks in long-video scenarios.

ReWatch-R1 is a post-trained large vision-LLM for complex video reasoning, built by training Qwen2.5-VL-7B-Instruct on the ReWatch dataset and then refining it with Reinforcement Learning with Verifiable Reward (RLVR). Its central claim is that the limited transfer of SFT + RLVR from image reasoning to video reasoning is primarily a data bottleneck: existing video data lack the challenging, multi-hop questions and high-quality, video-grounded Chain-of-Thought (CoT) data needed to bootstrap RLVR effectively. ReWatch-R1 addresses that bottleneck through a three-part dataset—ReWatch-Caption, ReWatch-QA, and ReWatch-CoT—a Multi-Agent ReAct framework that simulates a human-like “re-watching” process during CoT synthesis, and an Observation & Reasoning (O&R) reward that evaluates both final-answer correctness and the alignment of reasoning with video content (Zhang et al., 28 Sep 2025).

1. Problem formulation and conceptual scope

ReWatch-R1 is defined around complex video reasoning rather than generic video understanding. The paper identifies a mismatch between current RLVR practice and the structure of video tasks. In its account, existing video datasets are inadequate in three specific ways: captions are holistic and untimestamped, QA pairs are too simple and perception-oriented, and CoT traces are weakly grounded, often relying on commonsense or elimination rather than on evidence explicitly retrieved from video content (Zhang et al., 28 Sep 2025).

This diagnosis is tied to a particular failure mode of video post-training. If the reward checks mainly the final answer, then a model can obtain a correct answer with hallucinated or weakly grounded intermediate reasoning. In the paper’s formulation, video reasoning should instead require linking events across distant timestamps, retrieving evidence from multiple parts of a video, and composing observations into a final answer. The motivating contrast is between a weak rationale that uses generic commonsense and a stronger one that explicitly localizes events in time and verifies visual details before answering.

The term “re-watching” in ReWatch-R1 does not denote mere longer context windows. It denotes an explicit reasoning pattern: identify what evidence is needed, jump to a relevant temporal region, inspect it, possibly inspect another region, and then synthesize the result. A common misconception is to equate the method with simple long-video scaling or with generic CoT prompting. The paper instead frames it as a data-and-reward design for grounded temporal reasoning.

2. ReWatch dataset and synthetic supervision pipeline

The ReWatch dataset has three components. Each component is designed to supply one of the supervision types that the paper argues are missing in prior video-R1 pipelines.

Component	Size	Function
ReWatch-Caption-10k	10k captioned videos	Temporally dense video descriptions
ReWatch-QA-170k	170k QA pairs	High-difficulty QA supervision
ReWatch-CoT-135k	135k CoTs	Video-grounded reasoning traces

The videos are drawn from five public datasets: MiraData, VideoEspresso, VideoMarathon, Video-R1, and Vript. Table 2 in the paper reports the source breakdown as 1748 videos from MiraData (15.9%), 1977 from VideoEspresso (18.0%), 3291 from VideoMarathon (30.0%), 1982 from Video-R1 (18.0%), and 1991 from Vript (18.1%), for a total pool of about 11k videos (Zhang et al., 28 Sep 2025).

ReWatch-Caption-10k provides temporally dense captions intended to preserve long-range structure. The reported statistics are 4370.0 avg / 68279 max caption tokens and 504.3 avg / 16370 max summary tokens. The captioning pipeline is hierarchical. A long video $V$ is first partitioned into segments,

$S = \{s_1, \ldots, s_k\} = M_{\text{seg}}(V),$

each segment is densely described with relative timestamps,

$$D_{\text{rel}} = \{(c_{ij}, T_{ij})\}_{j=1}^{m_i} = M_{\text{cap}}(s_i),$$

those timestamps are realigned to absolute time,

$$t_{ij} = P(T_{ij}, t_{\text{start}}) = t_{\text{start}} + T_{ij},$$

and the final detailed caption is formed by union:

$$C_{\text{detail}}(V) = \bigcup_{i=1}^{k} \{(c_{ij}, t_{ij})\}_{j=1}^{m_i}.$$

ReWatch-QA-170k is generated contrastively. The pipeline first compresses the detailed caption into a summary $C_{\text{sum}}$, then asks a QA generator to create questions that are answerable from the detailed caption but not from the summary alone:

$$(Q, A)_{\text{raw}} = M_{\text{ga}}(C_{\text{detail}}, C_{\text{sum}}).$$

The paper then applies three filters: Answer Verification, Text Bias Elimination, and Summary Bias Elimination. The thresholds are reported as $\theta_{\text{text}} = 1$ and $\theta_{\text{sum}} = 1$. After filtering, 85k QA pairs survive and are rewritten into multiple-choice form, yielding 170k total QA pairs. The dataset contains 85,792 multiple-choice questions (50.2%) and 85,070 open-ended questions (49.8%), with 70.5 avg / 256 max question tokens and 6.2 avg / 256 max answer tokens. The paper further partitions them into 10 reasoning types, including Event Localization, Temporal Localization, Counting, Cause and Effect, Reading, Spatial Perception, Object Recognition, State Changes, Numerical Reasoning, and Counterfactual Reasoning (Zhang et al., 28 Sep 2025).

ReWatch-CoT-135k contains reasoning traces with 2.3 avg / 11 max reasoning steps and 332.5 avg / 2045 max reasoning tokens. These traces are not plain rationales: they are structured with explicit action and observation tags. This distinction is central. The paper is not merely synthesizing verbose explanations; it is synthesizing traces that represent retrieval and verification operations.

3. Multi-Agent ReAct and the simulated “re-watching” process

The paper’s core synthesis innovation is a Multi-Agent ReAct framework that separates planning from evidence retrieval. It uses two agents: a Reasoner Agent $A_R$, which produces thoughts $S = \{s_1, \ldots, s_k\} = M_{\text{seg}}(V),$0 and actions $S = \{s_1, \ldots, s_k\} = M_{\text{seg}}(V),$1, and an Observer Agent $S = \{s_1, \ldots, s_k\} = M_{\text{seg}}(V),$2, which executes those actions against the detailed caption and returns observations $S = \{s_1, \ldots, s_k\} = M_{\text{seg}}(V),$3 (Zhang et al., 28 Sep 2025).

For a question $S = \{s_1, \ldots, s_k\} = M_{\text{seg}}(V),$4, the interaction history is

$S = \{s_1, \ldots, s_k\} = M_{\text{seg}}(V),$5

and the next step is generated as

$S = \{s_1, \ldots, s_k\} = M_{\text{seg}}(V),$6

followed by observation retrieval,

$S = \{s_1, \ldots, s_k\} = M_{\text{seg}}(V),$7

The action space is intentionally small. The paper lists two core actions: segment_retrieval(query), which finds the timestamp of an event from a natural-language query, and segment_query(timestamp), which retrieves the detailed description of an event at a timestamp. This makes the “re-watching” metaphor concrete. The model is not supposed to reason from a monolithic summary; it is supposed to retrieve, inspect, and connect temporally localized evidence.

The resulting execution trajectory is then converted into a natural-language CoT string with explicit tags such as <action> ... </action> and <observation> ... </observation>. Appendix A.2 specifies the models used in synthesis: Gemini2.5-Flash (Non-Thinking) for $S = \{s_1, \ldots, s_k\} = M_{\text{seg}}(V),$8 and $S = \{s_1, \ldots, s_k\} = M_{\text{seg}}(V),$9, Gemini2.5-Flash-Lite (Non-Thinking) for $$D_{\text{rel}} = \{(c_{ij}, T_{ij})\}_{j=1}^{m_i} = M_{\text{cap}}(s_i),$$0 and $$D_{\text{rel}} = \{(c_{ij}, T_{ij})\}_{j=1}^{m_i} = M_{\text{cap}}(s_i),$$1, Gemini2.5-Flash (Thinking) for $$D_{\text{rel}} = \{(c_{ij}, T_{ij})\}_{j=1}^{m_i} = M_{\text{cap}}(s_i),$$2 and the Reasoner $$D_{\text{rel}} = \{(c_{ij}, T_{ij})\}_{j=1}^{m_i} = M_{\text{cap}}(s_i),$$3, GPT-4.1 for $$D_{\text{rel}} = \{(c_{ij}, T_{ij})\}_{j=1}^{m_i} = M_{\text{cap}}(s_i),$$4 and the Observer $$D_{\text{rel}} = \{(c_{ij}, T_{ij})\}_{j=1}^{m_i} = M_{\text{cap}}(s_i),$$5, and Qwen3-235B-A22B-Instruct together with Qwen2.5-VL-72B-Instruct as probe models.

A key clarification concerns how this mechanism appears during RL. The paper states that rollouts produce a purely textual CoT. During RL, the policy generates thought, action, observation, and answer as text segments. The “re-watch” loop is therefore not implemented as live external tool calls during policy rollout; it is a textual simulation of a Thought–Action–Observation process that is subsequently scored by reward models. This is one of the most important architectural distinctions of ReWatch-R1. A plausible implication is that the method gains training-time process supervision without incurring live tool-usage complexity in the RL inner loop.

4. Post-training: supervised warm start, RLVR, and the O&R reward

ReWatch-R1 is trained in two stages: Supervised Fine-Tuning (SFT) to obtain ReWatch-R1-SFT, followed by RLVR to obtain ReWatch-R1. The paper states explicitly that it uses the SFT+RL paradigm to train Qwen2.5-VL (Zhang et al., 28 Sep 2025).

The SFT stage jointly optimizes three losses:

$$D_{\text{rel}} = \{(c_{ij}, T_{ij})\}_{j=1}^{m_i} = M_{\text{cap}}(s_i),$$6

with

$$D_{\text{rel}} = \{(c_{ij}, T_{ij})\}_{j=1}^{m_i} = M_{\text{cap}}(s_i),$$7

$$D_{\text{rel}} = \{(c_{ij}, T_{ij})\}_{j=1}^{m_i} = M_{\text{cap}}(s_i),$$8

$$D_{\text{rel}} = \{(c_{ij}, T_{ij})\}_{j=1}^{m_i} = M_{\text{cap}}(s_i),$$9

The “thinking mode” prompt instructs the model to answer step by step, “re-watch” relevant clips when needed, mark actions with <action>, observations with <observation>, and the final answer with <answer>. SFT implementation details are reported as context length 16k, default fps 2.0, maximum frames 192, max resolution per frame 128 × 28 × 28, train batch size per device 1, gradient accumulation 4, learning rate 1e-6, max grad norm 1.0, optimizer AdamW, epochs 10, on 16 H800 GPUs.

The RL stage uses GRPO and 40k QA pairs sampled from ReWatch-QA (20k), Video-R1-QA (10k), and Long VideoReason-QA (10k). Reported RL settings are number of rollouts 8, sampling temperature 0.8, top-p 0.9, train batch size 14, PPO mini-batch size 14, PPO micro-batch size per GPU 1, learning rate 1e-5, max grad norm 5.0, optimizer AdamW, epochs 1, also on 16 H800 GPUs.

The principal RL innovation is the Observation & Reasoning (O&R) reward. A baseline answer-only reward is defined as

$$t_{ij} = P(T_{ij}, t_{\text{start}}) = t_{\text{start}} + T_{ij},$$0

The model output $$t_{ij} = P(T_{ij}, t_{\text{start}}) = t_{\text{start}} + T_{ij},$$1 is parsed into action-observation pairs,

$$t_{ij} = P(T_{ij}, t_{\text{start}}) = t_{\text{start}} + T_{ij},$$2

and an observation-grounding reward is computed as

$$t_{ij} = P(T_{ij}, t_{\text{start}}) = t_{\text{start}} + T_{ij},$$3

The reasoning sufficiency reward asks whether those actions and observations are enough to answer the question:

$$t_{ij} = P(T_{ij}, t_{\text{start}}) = t_{\text{start}} + T_{ij},$$4

$$t_{ij} = P(T_{ij}, t_{\text{start}}) = t_{\text{start}} + T_{ij},$$5

The combined reward is

$$t_{ij} = P(T_{ij}, t_{\text{start}}) = t_{\text{start}} + T_{ij},$$6

with format reward

$$t_{ij} = P(T_{ij}, t_{\text{start}}) = t_{\text{start}} + T_{ij},$$7

The reward models $$t_{ij} = P(T_{ij}, t_{\text{start}}) = t_{\text{start}} + T_{ij},$$8 and $$t_{ij} = P(T_{ij}, t_{\text{start}}) = t_{\text{start}} + T_{ij},$$9 are both Qwen3-30B-A3B-Instruct. The paper’s interpretation is explicit: unsupported intermediate observations reduce $$C_{\text{detail}}(V) = \bigcup_{i=1}^{k} \{(c_{ij}, t_{ij})\}_{j=1}^{m_i}.$$0, and insufficient or irrelevant observation sequences reduce $$C_{\text{detail}}(V) = \bigcup_{i=1}^{k} \{(c_{ij}, t_{ij})\}_{j=1}^{m_i}.$$1, thereby penalizing hallucinated reasoning. A common misconception is that ReWatch-R1 only rewards final-answer correctness; the paper’s novelty is precisely that it also evaluates whether the generated observations are grounded and sufficient.

At inference time, the paper evaluates both thinking mode and non-thinking mode, but positions the final ReWatch-R1 primarily for thinking mode.

5. Empirical results, ablations, and long-video behavior

The main reported claim is that ReWatch-R1 achieves state-of-the-art average performance on five challenging video reasoning benchmarks among comparably sized models (Zhang et al., 28 Sep 2025). At 192 frames, average reasoning accuracy is 35.51 for ReWatch-R1 + O&R, 34.86 for ReWatch-R1, and 33.25 for ReWatch-R1-SFT. Strong baselines at the same setting include Long VideoReason-RL (31.58), GLM4.1V-9B (31.19), Video-R1 (31.00), VideoRFT (30.83), and Video-R1-RL (30.28). The paper reports a +3.93 margin for ReWatch-R1 + O&R over the next-best comparable model at 192 frames. At 384 frames, average reasoning accuracy is 35.78 for ReWatch-R1 + O&R, 35.59 for ReWatch-R1, and 32.80 for ReWatch-R1-SFT, with a reported +1.30 margin over GLM4.1V-9B (34.48).

The per-benchmark reasoning scores for ReWatch-R1 + O&R are also given. At 192 frames: 40.43 on VCR Bench, 36.05 on MINERVA, 43.88 on Video Holmes, 31.67 on Video MathQA, and 25.51 on CG-AV Counting. At 384 frames: 38.78, 36.54, 44.26, 32.62, and 26.68 respectively. On understanding benchmarks, the model remains competitive. Average understanding accuracy at 192 frames is 54.15 for ReWatch-R1 + O&R, compared with 53.84 for GLM4.1V-9B and 53.12 for VideoRFT. At 384 frames, ReWatch-R1 + O&R reaches 54.60, while GLM4.1V-9B reports 56.43.

The ablation results identify the dataset components and reward design as the main drivers. SFT is indispensable: removing it causes catastrophic degradation. Replacing ReWatch-CoT with Video-R1-CoT degrades performance significantly, which the authors interpret as evidence that the multi-agent synthesis pipeline produces higher-quality reasoning supervision. RL also depends strongly on QA quality. The paper reports that baseline-QA-only RL reaches 42.0% all, 34.3% reasoning, and 51.7% understanding, whereas ReWatch-QA yields better results.

The paper also argues that ReWatch-QA is harder to shortcut than prior datasets. In a comparison using 5,000 correctly answered samples, ReWatch-QA has 3.31 average reasoning steps versus 1.82 for Video-R1-QA, 398.75 average response length versus 205.74, and 29.4% text-only accuracy versus 68.9%, against a 25% random-guess baseline. This suggests that the filtering pipeline effectively removes much of the text-only shortcut signal.

Training dynamics further support the paper’s interpretation of SFT and RL roles. During SFT, non-thinking mode improves faster and thinking mode converges more slowly; after RL, thinking mode experiences a large jump and eventually surpasses non-thinking mode. The authors summarize this as SFT teaching the format of reasoning and RL teaching the logic/function of reasoning.

Long-video robustness remains a central concern. Figure 1 reports a general “Long Video Tax.” ReWatch-R1 still degrades as video duration increases, but less severely than its comparators. On long reasoning videos (>20 minutes), ReWatch-R1 scores 27.46, versus 24.03 for Long VideoReason-RL, a +3.43 margin. The paper treats this as evidence of stronger robustness to long-range temporal dependencies, while also stating plainly that long-video reasoning remains unsolved.

6. Position within the video-R1 literature, misconceptions, and limitations

ReWatch-R1 belongs to a rapidly expanding family of video-R1 methods, but its mechanism is distinctive. VideoChat-R1 emphasizes spatio-temporal perception through GRPO-based reinforcement fine-tuning with rule-based rewards, reporting large gains in temporal grounding (+31.8) and object tracking (+31.2) over Qwen2.5-VL-7B while preserving general capability (Li et al., 9 Apr 2025). VersaVid-R1 frames progress as a matter of video-native reasoning tasks, notably DarkEventInfer and MixVidQA, and trains Qwen2.5-VL-7B-Instruct with direct GRPO and explicit > /<answer> outputs across MCQA, open-ended QA, and captioning (Chen et al., 10 Jun 2025). ViSS-R1 is explicitly relevant as a counterpoint: it also targets video-R1 failure modes, but it does so by forcing models to solve self-supervised visual transformation tasks during RL; it is therefore a close conceptual neighbor but not a re-watching or iterative observation method (Fang et al., 17 Nov 2025). Reflect-R1, a later system for long-video self-correction, is conceptually closer to the re-inspection idea because it uses an intuition–verification–arbitration pipeline with iterative temporal retrieval and SD-GRPO (Chen et al., 26 Jun 2026). This suggests that the video-R1 field has already diversified into at least three strategies: process-grounded data synthesis and reward shaping, visual-centric auxiliary tasks, and evidence-driven self-correction.

Several misconceptions are therefore best dispelled explicitly. First, ReWatch-R1 is not merely “more CoT.” Its distinctive claim is that RLVR for video requires process supervision that is video-grounded and retrieval-structured, not just longer answers. Second, the “re-watching” in the paper is not implemented as live external tool use during RL rollout; it is synthesized into textual Thought–Action–Observation trajectories and judged post hoc. Third, the state-of-the-art claim is qualified: the paper states that ReWatch-R1 achieves state of the art among comparably sized models. Fourth, the O&R reward is not a generic stylistic preference model; it is a structured reward that checks answer correctness, observation grounding, reasoning sufficiency, and output format.

The limitations are equally concrete. The pipeline is computationally heavy: data synthesis uses multiple frontier models such as Gemini2.5-Flash, GPT-4.1, and large Qwen variants; both SFT and RL use 16 H800 GPUs; and captions can be extremely long. The entire dataset is synthetic, which makes the system dependent on segmentation quality, caption faithfulness, QA generation quality, verifier reliability, and conversion quality. The O&R reward depends on $$C_{\text{detail}}(V) = \bigcup_{i=1}^{k} \{(c_{ij}, t_{ij})\}_{j=1}^{m_i}.$$2 and $$C_{\text{detail}}(V) = \bigcup_{i=1}^{k} \{(c_{ij}, t_{ij})\}_{j=1}^{m_i}.$$3, so reward noise remains possible, and the paper does not present a deep analysis of reward hacking or judge failure modes. Finally, although ReWatch-R1 improves long-video reasoning, the paper explicitly notes that all models, including its own, degrade as duration increases. A plausible implication is that future work will need to combine stronger evidence retrieval and self-correction mechanisms with the data-and-reward innovations introduced here.

In the broader history of video-R1 research, ReWatch-R1 is best understood as a system that reframes progress as a joint problem of dataset construction, reasoning-trace synthesis, and process-sensitive RL reward design. Its defining contribution is not a new visual encoder or a larger backbone, but a training regime in which video reasoning is made more learnable and more verifiable by simulating a human-like “re-watching” process inside the supervision and reward pipeline (Zhang et al., 28 Sep 2025).