---
title: 'Video-R4-RL-30k: RL Video Reasoning Dataset'
url: https://www.emergentmind.com/topics/video-r4-rl-30k
type: topic
---

# Video-R4-RL-30k: RL Video Reasoning Dataset

Video-R4-RL-30k is a curated large-scale reinforcement learning (RL) dataset designed to enable and benchmark iterative, tool-based, pixel-grounded video reasoning in language multimodal models (LMMs) through the paradigm of "visual rumination." It emphasizes high-density textual cues, frame- and region-level operations, and multi-step reasoning trajectories, providing a foundation for the training and evaluation of agents that must reread, revisit, and re-inspect video content to answer complex questions. The dataset is central to the Video-R4 project, an approach that unifies executable visual "tool" sequences and chain-of-thought (CoT) reasoning under RL objectives, driving state-of-the-art results on text-rich VideoQA tasks [2511.17490].

## 1. Dataset Origin and Construction

Video-R4-RL-30k is primarily constructed from the M4-ViteVQA training split (≈50,000 QA pairs focused on text-rich videos) and further supplemented with additional text-centered video QA sources such as WatchNews and EgoTextVQA, post-filtering for relevance and quality. Sample construction follows a hybrid semi-automated workflow:
- Answer evidence is automatically matched using fuzzy string alignment between gold answers and extracted OCR tokens (or object detections). 
- For each matched QA pair, "rumination trajectories" are synthesized using chain-of-thought templates that interleave textual reasoning with parametrized visual tool calls: "clip" (selecting subclips) and "crop" (zooming into spatiotemporal regions).
- Initial trajectory generation is model-driven (e.g., Qwen2.5-VL, GPT-4o), refined and validated through a human-in-the-loop interface to eliminate hallucinations and ensure coverage of supporting evidence.
- The final release totals ≈30,000 unique video-question-answer-tool sequences, with further subdivisions (2×15,000) tailored for staged RL finetuning.

Each trajectory in the dataset is a JSON object specifying the video path, question, chain-of-thought/observations, stepwise tool operations, bounding box/frame indices, and final answer—the tool calls are directly executable by models trained on the dataset.

## 2. Dataset Content and Statistical Profile

Video-R4-RL-30k emphasizes dense textual context and complex evidence localization:
- All video clips are inherited from the original source with typical durations of 5–20 seconds and resolutions downsampled to 224×224 or 256×256 for vision backbones.
- Videos are predominantly sampled at 1–5 FPS for tool call granularity.
- Textual density is ensured via per-frame OCR (10–50 tokens/frame) and 5–20 object detection boxes/frame.
- The dataset uniformly spans question types from simple single-frame text retrieval (≈40%) through multi-frame textual comparison (≈45%) to compositional object+text queries (≈15%).
- Trajectories feature median question lengths of 5–15 tokens and require a median of 3 tool calls (mixing “clip”, “crop”, or both). The majority (60%) involve mixed clip-and-crop operations, 25% are clip-only, and 15% crop-only.

## 3. Trajectory Format, Action Space, and State Representation

Each trajectory is structured as an executable sequence alternating reasoning text ("think"/"observation") and visual tool invocations.

**Format schema:**
- "trajectory_id": unique identifier.
- "video_path": pointer to the video file.
- "question": the natural-language video question.
- "initial_observation": a scene-level description.
- "steps": ordered actions.
  - Each "action" is either:
    - "clip": select a sequence of frame indices.
    - "crop": select a frame and normalized bounding box (x₁,y₁,x₂,y₂).
    - "answer": terminate and emit the predicted answer string.
  - Each action produces an "observation" (caption/description at the step).

The environment state $s_t$ at timestep $t$ consists of the current language model token context (prompt, question, all prior observations and tool calls) and metadata about which frames/regions have been accessed.

The action space $\mathcal{A}$ includes:
- Discrete clip: select $k$ consecutive frames.
- Crop: specify frame and spatial region.
- Answer: produce textual response, ending the trajectory.

## 4. Reinforcement Learning Formulation and Objective

The dataset underpins a Markov Decision Process suited for RL from expert-like trajectories:
- The state $s_t$ is the incremental sequence of reasoning text and tool actions.
- At each step, the policy $\pi_\theta$ produces an autoregressive action (next tool call or answer).
- Transition: each tool call augments the prompt with new visual evidence (pixels/caption), deterministically updating the environment.
- Reward: Provided only at trajectory completion; computed as:

  $$
  R' = R_{\mathrm{ans}} + \lambda_{\mathrm{div}} R_{\mathrm{div}} + \lambda_{\mathrm{rep}} R_{\mathrm{rep}} + \lambda_{\mathrm{cur}} R_{\mathrm{cur}}
  $$
  - $R_{\mathrm{ans}}$ is binary (exact/ANLS match).
  - $R_{\mathrm{div}}$: promotes diversity in accessed regions.
  - $R_{\mathrm{rep}}$: promotes representativeness of selected evidence.
  - $R_{\mathrm{cur}}$: encourages curiosity and penalizes redundancy/overlap in selected cues.

Models are trained using Group Relative Policy Optimization (GRPO), where rewards across multiple sampled trajectories for a single input are normalized and used for policy gradient estimation:

  $$
  \mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}_{q,\{o_i\}} \left[ \frac{1}{G} \sum_{i=1}^G \min(\rho_i(\theta) A_i, \mathrm{clip}(\rho_i(\theta), 1-\epsilon, 1+\epsilon) A_i) - \gamma D_{KL}(\pi_\theta||\pi_{\mathrm{ref}}) \right]
  $$

Here, $A_i$ is the normalized advantage, $\rho_i(\theta)$ is the importance weight, and the KL penalty enforces proximity to a reference policy. Both single- and multi-stage RL schedules are explored.

## 5. Evaluation Protocols and Empirical Findings

Evaluation leverages both the dataset holdout and multiple external benchmarks:
- Metrics include average cumulative reward $R'$, success rate (fraction with $R_{\mathrm{ans}} = 1$), and ANLS/exact-match on answer tokens.
- Baseline comparisons encompass single-stage RL, SFT-only, and various ablations (e.g., omitting reward terms).
- Video-R4 RL training on this dataset achieves +4–6% accuracy on M4-ViteVQA (over SFT alone), and +20–25% over off-the-shelf video LMMs that lack rumination capabilities.
- Policies display reduced hallucination and robust learned strategies for evidence verification (“read–ground–verify” emerges as a generalizable reasoning routine).

## 6. Best Practices and Research Extensions

For reproducibility and extension:
- A CLI data-loader and batching/serialization recipes are provided (see code snippet in original source).
- Recommended training pipeline is DRP-SFT (single-tool), RL$_d$ (first 15k), CRP-SFT (mixed-tool), RL$_c$ (second 15k).
- The dataset supports extensibility: adding new visual tools (such as “track”), integrating audio/multimodal cues, and adapting reward functions to task-specific objectives (e.g., optimizing F1).
- When scaling to larger language backbone models, it is advised to proportionally increase the group size $G$ and tune reward coefficients.
- The dataset structure lends itself to cross-domain transfer, with empirical evidence for strong zero-shot generalization to multi-page document QA, slide QA, and generic video reasoning tasks.

## 7. Positioning relative to Related RL and Video Reasoning Corpora

Video-R4-RL-30k is distinguished by its explicit actionability (executable chain-of-thought interaction history), focus on text-rich multimodal evidence, and support for RL with reward shaping and trajectory-level feedback. Unlike VideoSSR-30K—centered on self-supervised pretext tasks with synthetic reward signals [2511.06281]—or MARC, which targets token compression for scalability [2510.07915], Video-R4-RL-30k is uniquely tailored to visually grounded, “rumination-based” QA, supports discrete vision-language tool policies, and underpins competitive results on challenging evidence-intensive video benchmarks [2511.17490]. 

A plausible implication is that the dataset, in conjunction with the Video-R4 rumination learning framework, establishes a new standard for fine-grained video reasoning under RL by bridging frame-level vision with multimodal chain-of-thought, addressing challenges of hallucination, localization, and compositionality prevalent in text-rich video domains.

Source: https://www.emergentmind.com/topics/video-r4-rl-30k