---
title: 'ReWatch-R1: Advanced Video Reasoning Model'
url: https://www.emergentmind.com/topics/rewatch-r1
type: topic
---
# ReWatch-R1: Advanced Video Reasoning Model
ReWatch-R1 is a post-trained large vision-language model for complex video reasoning, built by training **Qwen2.5-VL-7B-Instruct** on the **ReWatch** dataset and then refining it with **Reinforcement Learning with Verifiable Reward (RLVR)**. Its central claim is that the limited transfer of **SFT + RLVR** from image reasoning to video reasoning is primarily a **data bottleneck**: existing video data lack the **challenging, multi-hop questions** and **high-quality, video-grounded Chain-of-Thought (CoT) data** needed to bootstrap RLVR effectively. ReWatch-R1 addresses that bottleneck through a three-part dataset—**ReWatch-Caption**, **ReWatch-QA**, and **ReWatch-CoT**—a **Multi-Agent ReAct framework** that simulates a human-like “re-watching” process during CoT synthesis, and an **Observation & Reasoning (O&R) reward** that evaluates both final-answer correctness and the alignment of reasoning with video content [2509.23652].
## 1. Problem formulation and conceptual scope
ReWatch-R1 is defined around **complex video reasoning** rather than generic video understanding. The paper identifies a mismatch between current RLVR practice and the structure of video tasks. In its account, existing video datasets are inadequate in three specific ways: **captions are holistic and untimestamped**, **QA pairs are too simple and perception-oriented**, and **CoT traces are weakly grounded**, often relying on commonsense or elimination rather than on evidence explicitly retrieved from video content [2509.23652].
This diagnosis is tied to a particular failure mode of video post-training. If the reward checks mainly the final answer, then a model can obtain a correct answer with hallucinated or weakly grounded intermediate reasoning. In the paper’s formulation, video reasoning should instead require **linking events across distant timestamps**, **retrieving evidence from multiple parts of a video**, and **composing observations into a final answer**. The motivating contrast is between a weak rationale that uses generic commonsense and a stronger one that explicitly localizes events in time and verifies visual details before answering.
The term “re-watching” in ReWatch-R1 does not denote mere longer context windows. It denotes an explicit reasoning pattern: identify what evidence is needed, jump to a relevant temporal region, inspect it, possibly inspect another region, and then synthesize the result. A common misconception is to equate the method with simple long-video scaling or with generic CoT prompting. The paper instead frames it as a **data-and-reward design** for grounded temporal reasoning.
## 2. ReWatch dataset and synthetic supervision pipeline
The **ReWatch** dataset has three components. Each component is designed to supply one of the supervision types that the paper argues are missing in prior video-R1 pipelines.
| Component | Size | Function |
|---|---:|---|
| ReWatch-Caption-10k | 10k captioned videos | Temporally dense video descriptions |
| ReWatch-QA-170k | 170k QA pairs | High-difficulty QA supervision |
| ReWatch-CoT-135k | 135k CoTs | Video-grounded reasoning traces |
The videos are drawn from **five public datasets**: **MiraData**, **VideoEspresso**, **VideoMarathon**, **Video-R1**, and **Vript**. Table 2 in the paper reports the source breakdown as **1748** videos from MiraData (**15.9%**), **1977** from VideoEspresso (**18.0%**), **3291** from VideoMarathon (**30.0%**), **1982** from Video-R1 (**18.0%**), and **1991** from Vript (**18.1%**), for a total pool of about **11k videos** [2509.23652].
**ReWatch-Caption-10k** provides temporally dense captions intended to preserve long-range structure. The reported statistics are **4370.0 avg / 68279 max** caption tokens and **504.3 avg / 16370 max** summary tokens. The captioning pipeline is hierarchical. A long video \(V\) is first partitioned into segments,
$$
S = \{s_1, \ldots, s_k\} = M_{\text{seg}}(V),
$$
each segment is densely described with relative timestamps,
$$
D_{\text{rel}} = \{(c_{ij}, T_{ij})\}_{j=1}^{m_i} = M_{\text{cap}}(s_i),
$$
those timestamps are realigned to absolute time,
$$
t_{ij} = P(T_{ij}, t_{\text{start}}) = t_{\text{start}} + T_{ij},
$$
and the final detailed caption is formed by union:
$$
C_{\text{detail}}(V) = \bigcup_{i=1}^{k} \{(c_{ij}, t_{ij})\}_{j=1}^{m_i}.
$$
**ReWatch-QA-170k** is generated contrastively. The pipeline first compresses the detailed caption into a summary \(C_{\text{sum}}\), then asks a QA generator to create questions that are **answerable from the detailed caption but not from the summary alone**:
$$
(Q, A)_{\text{raw}} = M_{\text{ga}}(C_{\text{detail}}, C_{\text{sum}}).
$$
The paper then applies three filters: **Answer Verification**, **Text Bias Elimination**, and **Summary Bias Elimination**. The thresholds are reported as \(\theta_{\text{text}} = 1\) and \(\theta_{\text{sum}} = 1\). After filtering, **85k** QA pairs survive and are rewritten into multiple-choice form, yielding **170k total QA pairs**. The dataset contains **85,792** multiple-choice questions (**50.2%**) and **85,070** open-ended questions (**49.8%**), with **70.5 avg / 256 max** question tokens and **6.2 avg / 256 max** answer tokens. The paper further partitions them into 10 reasoning types, including **Event Localization**, **Temporal Localization**, **Counting**, **Cause and Effect**, **Reading**, **Spatial Perception**, **Object Recognition**, **State Changes**, **Numerical Reasoning**, and **Counterfactual Reasoning** [2509.23652].
**ReWatch-CoT-135k** contains reasoning traces with **2.3 avg / 11 max** reasoning steps and **332.5 avg / 2045 max** reasoning tokens. These traces are not plain rationales: they are structured with explicit action and observation tags. This distinction is central. The paper is not merely synthesizing verbose explanations; it is synthesizing traces that represent retrieval and verification operations.
## 3. Multi-Agent ReAct and the simulated “re-watching” process
The paper’s core synthesis innovation is a **Multi-Agent ReAct framework** that separates planning from evidence retrieval. It uses two agents: a **Reasoner Agent** \(A_R\), which produces thoughts \(T_t\) and actions \(Act_t\), and an **Observer Agent** \(A_O\), which executes those actions against the detailed caption and returns observations \(Obs_t\) [2509.23652].
For a question \(Q\), the interaction history is
$$
H_{t-1} = (Q, T_1, Act_1, Obs_1, \ldots, T_{t-1}, Act_{t-1}, Obs_{t-1}),
$$
and the next step is generated as
$$
(T_t, Act_t) = A_R(H_{t-1}),
$$
followed by observation retrieval,
$$
Obs_t = A_O(Act_t, C_{\text{detail}}).
$$
The action space is intentionally small. The paper lists two core actions: **segment_retrieval(query)**, which finds the timestamp of an event from a natural-language query, and **segment_query(timestamp)**, which retrieves the detailed description of an event at a timestamp. This makes the “re-watching” metaphor concrete. The model is not supposed to reason from a monolithic summary; it is supposed to retrieve, inspect, and connect temporally localized evidence.
The resulting execution trajectory is then converted into a natural-language CoT string with explicit tags such as ` ... ` and ` ... `. Appendix A.2 specifies the models used in synthesis: **Gemini2.5-Flash (Non-Thinking)** for \(M_{\text{seg}}\) and \(M_{\text{cap}}\), **Gemini2.5-Flash-Lite (Non-Thinking)** for \(M_{\text{sum}}\) and \(M_{\text{convert}}\), **Gemini2.5-Flash (Thinking)** for \(M_{\text{ga}}\) and the Reasoner \(A_R\), **GPT-4.1** for \(M_{\text{verify}}\) and the Observer \(A_O\), and **Qwen3-235B-A22B-Instruct** together with **Qwen2.5-VL-72B-Instruct** as probe models.
A key clarification concerns how this mechanism appears during RL. The paper states that rollouts produce a **purely textual CoT**. During RL, the policy generates thought, action, observation, and answer as text segments. The “re-watch” loop is therefore **not implemented as live external tool calls during policy rollout**; it is a textual simulation of a Thought–Action–Observation process that is subsequently scored by reward models. This is one of the most important architectural distinctions of ReWatch-R1. A plausible implication is that the method gains training-time process supervision without incurring live tool-usage complexity in the RL inner loop.
## 4. Post-training: supervised warm start, RLVR, and the O&R reward
ReWatch-R1 is trained in two stages: **Supervised Fine-Tuning (SFT)** to obtain **ReWatch-R1-SFT**, followed by **RLVR** to obtain **ReWatch-R1**. The paper states explicitly that it uses the **SFT+RL paradigm to train Qwen2.5-VL** [2509.23652].
The SFT stage jointly optimizes three losses:
$$
L_{\text{SFT}}(\theta)=L_{\text{Cap}}+L_{\text{QA}}+L_{\text{COT}},
$$
with
$$
L_{\text{Cap}}=-\mathbb{E}_{(V,C_{\text{detail}})\in D_{\text{Cap}}}\left[\log \pi_\theta(C_{\text{detail}}\mid V)\right],
$$
$$
L_{\text{QA}}=-\mathbb{E}_{(V,Q,A)\in D_{\text{QA}}}\left[\log \pi_\theta(A\mid V, I_{\text{direct}}, Q)\right],
$$
$$
L_{\text{COT}}=-\mathbb{E}_{(V,Q,R)\in D_{\text{CoT}}}\left[\log \pi_\theta(R\mid V, I_{\text{think}}, Q)\right].
$$
The “thinking mode” prompt instructs the model to answer step by step, **“re-watch” relevant clips when needed**, mark actions with ``, observations with ``, and the final answer with ``. SFT implementation details are reported as **context length 16k**, **default fps 2.0**, **maximum frames 192**, **max resolution per frame 128 × 28 × 28**, **train batch size per device 1**, **gradient accumulation 4**, **learning rate 1e-6**, **max grad norm 1.0**, **optimizer AdamW**, **epochs 10**, on **16 H800 GPUs**.
The RL stage uses **GRPO** and **40k QA pairs** sampled from **ReWatch-QA (20k)**, **Video-R1-QA (10k)**, and **Long VideoReason-QA (10k)**. Reported RL settings are **number of rollouts 8**, **sampling temperature 0.8**, **top-p 0.9**, **train batch size 14**, **PPO mini-batch size 14**, **PPO micro-batch size per GPU 1**, **learning rate 1e-5**, **max grad norm 5.0**, **optimizer AdamW**, **epochs 1**, also on **16 H800 GPUs**.
The principal RL innovation is the **Observation & Reasoning (O&R) reward**. A baseline answer-only reward is defined as
$$
r_{\text{acc}} = M_{\text{judge}}(A, A_{\text{gt}}).
$$
The model output \(R\) is parsed into action-observation pairs,
$$
\{Act_i, Obs_i\}_{i=1}^{n} = \text{Parse}(R),
$$
and an observation-grounding reward is computed as
$$
r_{\text{obs}} = \text{mean}\left(\left\{M_{\text{judge}}(C_{\text{detail}}, \{Act_i, Obs_i\})\right\}_{i=1}^{n}\right).
$$
The reasoning sufficiency reward asks whether those actions and observations are enough to answer the question:
$$
A_{\text{ao}} = M_{\text{infer}}(Q, \{Act_i, Obs_i\}_{i=1}^{n}),
$$
$$
r_{\text{rea}} = M_{\text{judge}}(A_{\text{ao}}, A_{\text{gt}}).
$$
The combined reward is
$$
r_{\text{O\&R}} = r_{\text{acc}} \times (1 + r_{\text{obs}} + r_{\text{rea}}) + r_{\text{fmt}},
$$
with format reward
$$
r_{\text{fmt}} =
\begin{cases}
0.1, & \text{correct format} \\
0, & \text{otherwise}.
\end{cases}
$$
The reward models \(M_{\text{infer}}\) and \(M_{\text{judge}}\) are both **Qwen3-30B-A3B-Instruct**. The paper’s interpretation is explicit: unsupported intermediate observations reduce \(r_{\text{obs}}\), and insufficient or irrelevant observation sequences reduce \(r_{\text{rea}}\), thereby penalizing hallucinated reasoning. A common misconception is that ReWatch-R1 only rewards final-answer correctness; the paper’s novelty is precisely that it also evaluates whether the generated observations are grounded and sufficient.
At inference time, the paper evaluates both **thinking mode** and **non-thinking mode**, but positions the final ReWatch-R1 primarily for **thinking mode**.
## 5. Empirical results, ablations, and long-video behavior
The main reported claim is that ReWatch-R1 achieves **state-of-the-art average performance on five challenging video reasoning benchmarks** among comparably sized models [2509.23652]. At **192 frames**, average reasoning accuracy is **35.51** for **ReWatch-R1 + O&R**, **34.86** for **ReWatch-R1**, and **33.25** for **ReWatch-R1-SFT**. Strong baselines at the same setting include **Long VideoReason-RL (31.58)**, **GLM4.1V-9B (31.19)**, **Video-R1 (31.00)**, **VideoRFT (30.83)**, and **Video-R1-RL (30.28)**. The paper reports a **+3.93** margin for **ReWatch-R1 + O&R** over the next-best comparable model at 192 frames. At **384 frames**, average reasoning accuracy is **35.78** for **ReWatch-R1 + O&R**, **35.59** for **ReWatch-R1**, and **32.80** for **ReWatch-R1-SFT**, with a reported **+1.30** margin over **GLM4.1V-9B (34.48)**.
The per-benchmark reasoning scores for **ReWatch-R1 + O&R** are also given. At **192 frames**: **40.43** on **VCR Bench**, **36.05** on **MINERVA**, **43.88** on **Video Holmes**, **31.67** on **Video MathQA**, and **25.51** on **CG-AV Counting**. At **384 frames**: **38.78**, **36.54**, **44.26**, **32.62**, and **26.68** respectively. On understanding benchmarks, the model remains competitive. Average understanding accuracy at **192 frames** is **54.15** for **ReWatch-R1 + O&R**, compared with **53.84** for **GLM4.1V-9B** and **53.12** for **VideoRFT**. At **384 frames**, **ReWatch-R1 + O&R** reaches **54.60**, while **GLM4.1V-9B** reports **56.43**.
The ablation results identify the dataset components and reward design as the main drivers. **SFT is indispensable**: removing it causes catastrophic degradation. Replacing **ReWatch-CoT** with **Video-R1-CoT** degrades performance significantly, which the authors interpret as evidence that the multi-agent synthesis pipeline produces higher-quality reasoning supervision. RL also depends strongly on QA quality. The paper reports that baseline-QA-only RL reaches **42.0% all**, **34.3% reasoning**, and **51.7% understanding**, whereas **ReWatch-QA** yields better results.
The paper also argues that ReWatch-QA is harder to shortcut than prior datasets. In a comparison using **5,000 correctly answered samples**, **ReWatch-QA** has **3.31** average reasoning steps versus **1.82** for **Video-R1-QA**, **398.75** average response length versus **205.74**, and **29.4%** text-only accuracy versus **68.9%**, against a **25%** random-guess baseline. This suggests that the filtering pipeline effectively removes much of the text-only shortcut signal.
Training dynamics further support the paper’s interpretation of SFT and RL roles. During SFT, **non-thinking mode improves faster** and **thinking mode converges more slowly**; after RL, **thinking mode experiences a large jump and eventually surpasses non-thinking mode**. The authors summarize this as SFT teaching the **format** of reasoning and RL teaching the **logic/function** of reasoning.
Long-video robustness remains a central concern. Figure 9 reports a general **“Long Video Tax.”** ReWatch-R1 still degrades as video duration increases, but less severely than its comparators. On **long reasoning videos** (>20 minutes), **ReWatch-R1** scores **27.46**, versus **24.03** for **Long VideoReason-RL**, a **+3.43** margin. The paper treats this as evidence of stronger robustness to long-range temporal dependencies, while also stating plainly that long-video reasoning remains unsolved.
## 6. Position within the video-R1 literature, misconceptions, and limitations
ReWatch-R1 belongs to a rapidly expanding family of video-R1 methods, but its mechanism is distinctive. **VideoChat-R1** emphasizes **spatio-temporal perception** through **GRPO-based reinforcement fine-tuning** with rule-based rewards, reporting large gains in **temporal grounding (+31.8)** and **object tracking (+31.2)** over **Qwen2.5-VL-7B** while preserving general capability [2504.06958]. **VersaVid-R1** frames progress as a matter of **video-native reasoning tasks**, notably **DarkEventInfer** and **MixVidQA**, and trains **Qwen2.5-VL-7B-Instruct** with direct **GRPO** and explicit `/` outputs across MCQA, open-ended QA, and captioning [2506.09079]. **ViSS-R1** is explicitly relevant as a counterpoint: it also targets video-R1 failure modes, but it does so by forcing models to solve **self-supervised visual transformation tasks** during RL; it is therefore a close conceptual neighbor but **not** a re-watching or iterative observation method [2511.13054]. **Reflect-R1**, a later system for long-video self-correction, is conceptually closer to the re-inspection idea because it uses an **intuition–verification–arbitration** pipeline with iterative temporal retrieval and **SD-GRPO** [2606.27922]. This suggests that the video-R1 field has already diversified into at least three strategies: process-grounded data synthesis and reward shaping, visual-centric auxiliary tasks, and evidence-driven self-correction.
Several misconceptions are therefore best dispelled explicitly. First, ReWatch-R1 is not merely “more CoT.” Its distinctive claim is that RLVR for video requires process supervision that is **video-grounded** and **retrieval-structured**, not just longer answers. Second, the “re-watching” in the paper is not implemented as live external tool use during RL rollout; it is synthesized into textual Thought–Action–Observation trajectories and judged post hoc. Third, the state-of-the-art claim is qualified: the paper states that ReWatch-R1 achieves state of the art **among comparably sized models**. Fourth, the O&R reward is not a generic stylistic preference model; it is a structured reward that checks answer correctness, observation grounding, reasoning sufficiency, and output format.
The limitations are equally concrete. The pipeline is computationally heavy: data synthesis uses multiple frontier models such as **Gemini2.5-Flash**, **GPT-4.1**, and large Qwen variants; both SFT and RL use **16 H800 GPUs**; and captions can be extremely long. The entire dataset is synthetic, which makes the system dependent on segmentation quality, caption faithfulness, QA generation quality, verifier reliability, and conversion quality. The O&R reward depends on \(M_{\text{judge}}\) and \(M_{\text{infer}}\), so reward noise remains possible, and the paper does not present a deep analysis of reward hacking or judge failure modes. Finally, although ReWatch-R1 improves long-video reasoning, the paper explicitly notes that all models, including its own, degrade as duration increases. A plausible implication is that future work will need to combine stronger evidence retrieval and self-correction mechanisms with the data-and-reward innovations introduced here.
In the broader history of video-R1 research, ReWatch-R1 is best understood as a system that reframes progress as a joint problem of **dataset construction**, **reasoning-trace synthesis**, and **process-sensitive RL reward design**. Its defining contribution is not a new visual encoder or a larger backbone, but a training regime in which video reasoning is made more learnable and more verifiable by simulating a human-like “re-watching” process inside the supervision and reward pipeline [2509.23652].
Source: https://www.emergentmind.com/topics/rewatch-r1