Video-R4-CoT-17k Benchmark for Video Reasoning

• Video-R4-CoT-17k is a benchmark composed of 17,000 annotated video reasoning trajectories that integrate textual reasoning with visual operations like frame selection and zooming.
• It employs a multi-turn, iterative framework which pairs each logical step with a specific visual action to mitigate hallucinations and maintain evidence traceability.
• Empirical results show that models trained on this dataset achieve over 10-point accuracy improvements over text-only baselines on challenging video QA tasks.

Video-R4-CoT-17k is a benchmark-scale dataset comprising 17,000 executable, visually grounded Chain-of-Thought (CoT) trajectories curated for training and evaluating large multimodal LLMs (LMMs) on text-rich video reasoning. Its primary design objective is to teach models to perform iterative, interleaved reasoning and visual operations (such as frame selection and region zooming), thereby grounding abstract logical arguments in concrete, pixel-level evidence. This approach is motivated by the need to overcome hallucinations and logical gaps pervasive in text-only or single-pass video QA on content where key information may appear transiently or occupy restricted spatial regions, such as lecture slides, UI walkthroughs, or road-sign-rich scenes (Tang et al., 21 Nov 2025).

1. Motivation and Design Principles

Video-R4-CoT-17k addresses two bottlenecks in video reasoning:

1. Sparsity and briefness of critical text cues—many videos contain essential information (e.g., UI labels, document snippets, slide text overlays) that is only visible in a handful of frames or within small image regions.
2. Insufficiency of text-only CoT supervision—textual reasoning chains fail to enforce direct pixel grounding, leading to hallucinated or unsubstantiated claims.

To resolve this, each trajectory in Video-R4-CoT-17k is structured as an executable, multi-turn sequence in which every logical step is coupled with a specific visual operation (frame selection, region zoom) and coupled state update, simulating a human-like process of "visual rumination"—iteratively pausing, inspecting, and accumulating evidence before answering.

2. Dataset Composition and Statistics

Video-R4-CoT-17k contains 17,000 annotated (video, question, answer, CoT) quadruples. The composition is summarized as:

Category	# Examples	Operation Type	Description
Atomic cropping	5,000	image-only	One-frame, single-region zoom
Atomic clipping	2,000	video-only	Selects informative frame clip(s)
Compositional CoT	10,000	multi-step, mixed	Interleaves frame selection and cropping

Statistical properties:

• Video duration: 3–20 s (mean ≈ 8 s), downsampled at 2 fps, yielding ∼16 frames/example.
• Text density: ∼7 OCR tokens per frame (∼0.4 tokens/100 px²).
• CoT complexity: On average, each trajectory has 4.2 visual operations and 6.1 turns (combining "thinking" and "action").
• Distribution: About 75% of examples are true sequential video (multi-frame), with the remainder focused on single-frame visual inspection.

3. Annotation Pipeline and Data Format

Annotation proceeds in three stages:

A. Evidence Matching:

Starting from M4-ViteVQA QA pairs (with aligned OCR/object labels), heuristic algorithms identify frames and bounding boxes containing answer-relevant content.

B. CoT Template Synthesis:

A fixed template encodes each reasoning step as either natural language or formal tool call:

• <tool_call>SelectFrame(frames=[...])</tool_call>
• <tool_call>ZoomRegion(frame=..., box=[x₁,y₁,x₂,y₂])</tool_call>

C. LMM-Guided Generation and Human Verification:

Qwen2.5-VL generates stepwise captions; GPT-4o refines trajectories; all samples undergo human audit for hallucinations, coherence, and pixel-to-answer traceability.

JSON structure per example:

$h_t$2 Each tool invocation is functionally executable, allowing models to interact with actual video data during training or evaluation (Tang et al., 21 Nov 2025).

4. Formalism and Learning Objective

At each step $t$, the model operates on a memory state $h_t$, comprising the history of previous tokens and visual features. Operations are formally defined:

• Frame selection: $o_t = \mathrm{SelectFrame}(V, q, h_t)$, with $o_t$ = frame indices to read.
• Region zoom: $r_t = \mathrm{ZoomRegion}(F_t, B_t)$, where $B_t$ is the predicted bounding box on selected frame(s).
• State update: $h_{t+1} = \mathrm{ReEncode}(h_t, r_t)$, where extracted pixel features (via vision encoder $\phi$) are appended to the model context.

Training minimizes the standard autoregressive cross-entropy:

$$\mathcal{L}_{\mathrm{CoT}} = -\sum_{t=1}^T \log p(\hat a_t \mid h_t)$$

Both natural language and tool-call arguments are part of the supervision sequence.

Evaluation on downstream QA tasks uses Accuracy (Exact Match) and Average Normalized Levenshtein Similarity (ANLS), as defined:

$$\mathrm{ANLS} = \frac{1}{N} \sum_{i=1}^N \max_j s(a_{ij}, o_{q_i})$$

where $h_t$0 is a reference answer and $h_t$1 is the model output.

5. Empirical Results and Model Insights

Supervised fine-tuning of a 7B LMM (Video-R4-7B) on Video-R4-CoT-17k ("DRP" for atomic, "CRP" for compositional) yields substantial improvements over baseline and text-only CoT models on text-rich video QA benchmarks (e.g., M4-ViteVQA). Visual-tool grounding:

• Reduces answer hallucination by enforcing explicit step-by-step evidence acquisition
• Raises accuracy by >10 points on challenging, multi-cue queries compared to text-only or single-pass baselines
• Benefits from curriculum learning: beginning with atomic examples followed by compositional chains accelerates convergence and improves final performance

Trajectories capped at 8–10 steps balance the tradeoff between deep evidence search and inference latency. Tool-call tokens must be present at both training and inference for maximal effect.

6. Usage Considerations and Best Practices

For practitioners employing Video-R4-CoT-17k:

1. Curriculum schedule: Train on atomic (DRP) before compositional (CRP) trajectories.
2. Strict tool-call syntax: Mark explicit tool calls to distinguish reasoning from action phases.
3. Trajectory length: Cap at 8–10 steps to avoid overly lengthy or inefficient reasoning.
4. Metrics monitoring: Use both task-level QA performance and token-level CoT loss to ensure both answer quality and reasoning fidelity.
5. Tool execution at inference: Where possible, the model’s proposed visual operations should be run on real pixels to verify that evidence acquisition is physically realizable and not merely hypothetical.

7. Impact, Limitations, and Broader Context

Video-R4-CoT-17k constitutes a substantial advance in pixel-grounded, executable CoT supervision for video reasoning. Its design directly addresses state-of-the-art limitations regarding hallucination, poor localization, and inability of LMMs to "verify in pixels" prior to answer, especially in text-intensive contexts (Tang et al., 21 Nov 2025). However, residual limitations include:

• Scope: Focused primarily on text-rich domains; performance on purely visual (non-text) reasoning may remain sub-optimal.
• Annotation cost: Generation involves significant human refinement to guarantee grounding and reduce hallucination.
• Scalability: Increased trajectory length can result in computational overhead at both training and inference.

A plausible implication is that the "executable visual CoT" paradigm may become foundational for broader multimodal reasoning systems, especially where transparent, auditable reasoning steps are needed. The dataset exemplifies an emerging trend aligning machine inference more closely to the visual re-inspection behavior observed in expert human video analysis.