Video-R4-RL-30k: RL Video Reasoning Dataset

Updated 3 July 2026

Video-R4-RL-30k is a curated dataset providing high-density textual cues and synthesized, executable trajectories for iterative video reasoning.
It employs chain-of-thought templates interleaved with visual tool operations like clip and crop to support multi-step, evidence-based decision making.
Evaluation demonstrates state-of-the-art improvements on text-rich VideoQA tasks using reward shaping and group-based policy optimization.

Video-R4-RL-30k is a curated large-scale reinforcement learning (RL) dataset designed to enable and benchmark iterative, tool-based, pixel-grounded video reasoning in language multimodal models (LMMs) through the paradigm of "visual rumination." It emphasizes high-density textual cues, frame- and region-level operations, and multi-step reasoning trajectories, providing a foundation for the training and evaluation of agents that must reread, revisit, and re-inspect video content to answer complex questions. The dataset is central to the Video-R4 project, an approach that unifies executable visual "tool" sequences and chain-of-thought (CoT) reasoning under RL objectives, driving state-of-the-art results on text-rich VideoQA tasks (Tang et al., 21 Nov 2025).

1. Dataset Origin and Construction

Video-R4-RL-30k is primarily constructed from the M4-ViteVQA training split (≈50,000 QA pairs focused on text-rich videos) and further supplemented with additional text-centered video QA sources such as WatchNews and EgoTextVQA, post-filtering for relevance and quality. Sample construction follows a hybrid semi-automated workflow:

Answer evidence is automatically matched using fuzzy string alignment between gold answers and extracted OCR tokens (or object detections).
For each matched QA pair, "rumination trajectories" are synthesized using chain-of-thought templates that interleave textual reasoning with parametrized visual tool calls: "clip" (selecting subclips) and "crop" (zooming into spatiotemporal regions).
Initial trajectory generation is model-driven (e.g., Qwen2.5-VL, GPT-4o), refined and validated through a human-in-the-loop interface to eliminate hallucinations and ensure coverage of supporting evidence.
The final release totals ≈30,000 unique video-question-answer-tool sequences, with further subdivisions (2×15,000) tailored for staged RL finetuning.

Each trajectory in the dataset is a JSON object specifying the video path, question, chain-of-thought/observations, stepwise tool operations, bounding box/frame indices, and final answer—the tool calls are directly executable by models trained on the dataset.

2. Dataset Content and Statistical Profile

Video-R4-RL-30k emphasizes dense textual context and complex evidence localization:

All video clips are inherited from the original source with typical durations of 5–20 seconds and resolutions downsampled to 224×224 or 256×256 for vision backbones.
Videos are predominantly sampled at 1–5 FPS for tool call granularity.
Textual density is ensured via per-frame OCR (10–50 tokens/frame) and 5–20 object detection boxes/frame.
The dataset uniformly spans question types from simple single-frame text retrieval (≈40%) through multi-frame textual comparison (≈45%) to compositional object+text queries (≈15%).
Trajectories feature median question lengths of 5–15 tokens and require a median of 3 tool calls (mixing “clip”, “crop”, or both). The majority (60%) involve mixed clip-and-crop operations, 25% are clip-only, and 15% crop-only.

3. Trajectory Format, Action Space, and State Representation

Each trajectory is structured as an executable sequence alternating reasoning text ("think"/"observation") and visual tool invocations.

Format schema:

"trajectory_id": unique identifier.
"video_path": pointer to the video file.
"question": the natural-language video question.
"initial_observation": a scene-level description.
"steps": ordered actions.
- Each "action" is either:
- "clip": select a sequence of frame indices.
- "crop": select a frame and normalized bounding box (x₁,y₁,x₂,y₂).
- "answer": terminate and emit the predicted answer string.
- Each action produces an "observation" (caption/description at the step).

The environment state $s_t$ at timestep $t$ consists of the current LLM token context (prompt, question, all prior observations and tool calls) and metadata about which frames/regions have been accessed.

The action space $\mathcal{A}$ includes:

Discrete clip: select $k$ consecutive frames.
Crop: specify frame and spatial region.
Answer: produce textual response, ending the trajectory.

4. Reinforcement Learning Formulation and Objective

The dataset underpins a Markov Decision Process suited for RL from expert-like trajectories:

The state $s_t$ is the incremental sequence of reasoning text and tool actions.
At each step, the policy $\pi_\theta$ produces an autoregressive action (next tool call or answer).
Transition: each tool call augments the prompt with new visual evidence (pixels/caption), deterministically updating the environment.
Reward: Provided only at trajectory completion; computed as:

$R' = R_{\mathrm{ans}} + \lambda_{\mathrm{div}} R_{\mathrm{div}} + \lambda_{\mathrm{rep}} R_{\mathrm{rep}} + \lambda_{\mathrm{cur}} R_{\mathrm{cur}}$

$R_{\mathrm{ans}}$ is binary (exact/ANLS match).
$R_{\mathrm{div}}$ : promotes diversity in accessed regions.
$R_{\mathrm{rep}}$ : promotes representativeness of selected evidence.
$t$ 0: encourages curiosity and penalizes redundancy/overlap in selected cues.

Models are trained using Group Relative Policy Optimization (GRPO), where rewards across multiple sampled trajectories for a single input are normalized and used for policy gradient estimation:

$t$ 1

Here, $t$ 2 is the normalized advantage, $t$ 3 is the importance weight, and the KL penalty enforces proximity to a reference policy. Both single- and multi-stage RL schedules are explored.

5. Evaluation Protocols and Empirical Findings

Evaluation leverages both the dataset holdout and multiple external benchmarks:

Metrics include average cumulative reward $t$ 4, success rate (fraction with $t$ 5), and ANLS/exact-match on answer tokens.
Baseline comparisons encompass single-stage RL, SFT-only, and various ablations (e.g., omitting reward terms).
Video-R4 RL training on this dataset achieves +4–6% accuracy on M4-ViteVQA (over SFT alone), and +20–25% over off-the-shelf video LMMs that lack rumination capabilities.
Policies display reduced hallucination and robust learned strategies for evidence verification (“read–ground–verify” emerges as a generalizable reasoning routine).

6. Best Practices and Research Extensions

For reproducibility and extension:

A CLI data-loader and batching/serialization recipes are provided (see code snippet in original source).
Recommended training pipeline is DRP-SFT (single-tool), RL $t$ 6 (first 15k), CRP-SFT (mixed-tool), RL $t$ 7 (second 15k).
The dataset supports extensibility: adding new visual tools (such as “track”), integrating audio/multimodal cues, and adapting reward functions to task-specific objectives (e.g., optimizing F1).
When scaling to larger language backbone models, it is advised to proportionally increase the group size $t$ 8 and tune reward coefficients.
The dataset structure lends itself to cross-domain transfer, with empirical evidence for strong zero-shot generalization to multi-page document QA, slide QA, and generic video reasoning tasks.

Video-R4-RL-30k is distinguished by its explicit actionability (executable chain-of-thought interaction history), focus on text-rich multimodal evidence, and support for RL with reward shaping and trajectory-level feedback. Unlike VideoSSR-30K—centered on self-supervised pretext tasks with synthetic reward signals (He et al., 9 Nov 2025)—or MARC, which targets token compression for scalability (Wu et al., 9 Oct 2025), Video-R4-RL-30k is uniquely tailored to visually grounded, “rumination-based” QA, supports discrete vision-language tool policies, and underpins competitive results on challenging evidence-intensive video benchmarks (Tang et al., 21 Nov 2025).

A plausible implication is that the dataset, in conjunction with the Video-R4 rumination learning framework, establishes a new standard for fine-grained video reasoning under RL by bridging frame-level vision with multimodal chain-of-thought, addressing challenges of hallucination, localization, and compositionality prevalent in text-rich video domains.