---
title: 'MTVR-CoT-72k: Video Reasoning Fine-Tuning Dataset'
url: https://www.emergentmind.com/topics/mtvr-cot-72k
type: topic
---

# MTVR-CoT-72k: Video Reasoning Fine-Tuning Dataset

MTVR-CoT-72k is the supervised fine-tuning portion of the Multi-Task Video Reasoning data introduced for VITAL, the framework in "Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning" [2508.04416]. It is a 72k-sample multi-task video reasoning dataset with chain-of-thought annotations, constructed for supervised fine-tuning (cold start) of VITAL. A recurrent misconception is to treat it as a single monolithic dataset; in the paper’s terminology, it is the aggregate of two concrete subsets—MTVR-CoT and MTVR-CoT-Tool—used to teach video understanding, temporal grounding, reasoning VQA, grounded VQA, and both text-based and tool-augmented multimodal chain-of-thought reasoning, with particular emphasis on long videos.

## 1. Dataset identity and internal composition

MTVR-CoT-72k consists of two subsets that together form the supervised fine-tuning corpus for VITAL [2508.04416].

| Subset | Samples | Role |
|---|---:|---|
| MTVR-CoT | 54k | Stage-1 SFT; text-based CoT; no tools |
| MTVR-CoT-Tool | 18k | Stage-3 SFT; multimodal CoT with tools |
| MTVR-CoT-72k | 72k | Total SFT portion |

The dataset supports three core tasks. **Temporal Grounding (TG)** takes a video and a natural language query and requires localization of the relevant temporal segment $([t_s, t_e])$. **Reasoning VQA (RQA)** takes a video and a question and returns an answer; the supported answer forms include multiple-choice, open-ended, number, and OCR. **Grounded VQA (GQA)** jointly predicts the answer and the time range in the video where the evidence lies. The paper treats these three tasks as jointly beneficial and uses them together in the same training corpus.

Its distinctive role is not merely to provide labels, but to provide explicit chain-of-thought supervision. All samples include text-based CoT. The tool subset additionally includes multimodal CoT in which the reasoning process interleaves textual thinking with tool calls and subsequent visual evidence. The intended behavior is “thinking with videos” rather than reasoning over a fixed, sparsely sampled visual context alone.

## 2. Source datasets and construction pipeline

MTVR-CoT-72k is built from established temporal grounding and video QA sources [2508.04416]. The temporal grounding sources are Charades-STA, ActivityNet-MR, VidChapters-7M, and ReXTime. The video QA and reasoning sources are Video-R1, LongVideo-Reason, and NExT-GQA. These source datasets provide raw videos, temporal annotations for TG and GQA, and question-answer annotations for QA and GQA.

The overall construction follows a rollout-filter pipeline. For each raw sample $(\mathcal{V}, \mathcal{Q}, \mathcal{A})$, Qwen2.5-VL-7B generates $k=8$ rollouts at temperature $1.0$ to encourage diversity. For each rollout, a task-specific reward $\mathcal{R}_{i,j}^k$ is computed, and the reward range is defined as
$$
\Delta \mathcal{R}_{i,j} = \max_k \mathcal{R}_{i,j}^k - \min_k \mathcal{R}_{i,j}^k.
$$
Samples with very small reward range, $\Delta \mathcal{R}_{i,j} \le 0.05$, are discarded. The retained samples are therefore neither uniformly trivial nor uniformly impossible for the base model.

On the filtered set, a stronger reasoning MLLM—described in the paper as Gemini 2.5 Pro—is used to generate annotations. It produces text-based CoT for all samples in MTVR-CoT and multimodal CoT with tool calls for long video samples in MTVR-CoT-Tool. For long video temporal grounding, tool parameter suggestions are derived from the ground-truth segment $[s,e]$ by injecting $20\%$ noise:
$$
s^\prime = \text{clamp}(s - \lambda |s| \cdot \text{rand}(), 0, L)
$$
$$
e^\prime = \text{clamp}(e + \lambda |L-e| \cdot \text{rand}(), 0, L)
$$
with $\lambda = 0.2$ and $L$ the video duration. For long video QA, by contrast, Gemini decides tool parameters autonomously.

The multimodal CoT subsets are generated with a three-round multi-turn prompt. Round 1 produces initial `<think>` reasoning over the video and question. Round 2 produces a tool call, such as a JSON invocation of a `video_clip` function with start and end timestamps. Round 3 conditions on the tool result and produces reflection thinking and the final `<answer>`. After annotation, post-processing discards samples whose CoT is incomplete, whose final answer does not match the known ground truth, or whose reasoning refers to meta-concepts such as “ground truth” or “suggestion text.”

## 3. Annotation schema, task formulations, and tool traces

Each MTVR-CoT-72k sample includes a question-answer pair, temporal grounding labels when applicable, and one or more reasoning traces [2508.04416]. The answer may be a multiple-choice label, open-ended text, numeric value, OCR text, or another answer form inherited from the underlying task. For TG and GQA, the dataset preserves ground-truth start and end timestamps $[t_s, t_e]$ from the source datasets.

The chain-of-thought annotations use an explicit tagged format. Textual reasoning steps are stored in `<think> ... </think>`, tool invocations in `<tool_call> ... </tool_call>`, and final outputs in `<answer> ... </answer>`. The tool subset therefore records not only a final answer, but a structured interaction sequence in which the model first reasons, then requests additional video evidence, then reasons again conditioned on the returned clip.

The paper specifies tool calls in JSON, following a Hermes-like protocol:

```json
{
  "name": "video_clip",
  "arguments": {"start": ..., "end": ...}
}
```

The tool returns a “video clip visual tokens” object, which is then incorporated into subsequent reasoning. In the tool subset, the full interaction trajectory is embedded in the CoT and formalized as
$$
\tau = \{\mathcal{T}_1, \mathcal{C}_1, \mathcal{V}_1,\ \mathcal{T}_2, \mathcal{C}_2, \mathcal{V}_2,\dots, \mathcal{T}_n, \mathcal{A}_n\},
$$
where $\mathcal{T}_k$ is a thinking step, $\mathcal{C}_k$ a tool call, $\mathcal{V}_k$ a tool result, and $\mathcal{A}_n$ the final answer.

The task formulations align with the paper’s training and reward definitions. For TG, the input is $(\mathcal{V}, \mathcal{Q})$ and the output is a predicted segment $[\hat{t}_s, \hat{t}_e]$, evaluated by intersection-over-union:
$$
\text{IoU} = \frac{\text{len}([\hat{t}_s, \hat{t}_e] \cap [t_s, t_e])}{\text{len}([\hat{t}_s, \hat{t}_e] \cup [t_s, t_e])}.
$$
For RQA, the input is again $(\mathcal{V}, \mathcal{Q})$ and the output is an answer $\hat{\mathcal{A}}$; the paper lists exact match for MCQ, ROUGE for open-ended answers, WER for OCR, and a clamped $L_1$-style score for regression-type outputs. For GQA, the output couples an answer with a grounded temporal segment, and the paper evaluates it with combined answer-and-IoU criteria such as $(\text{IoU} + \text{EM})/2$ or $(\text{IoU} + \text{Rouge})/2$.

## 4. Function within the VITAL training pipeline

The training pipeline in the paper has four stages, and MTVR-CoT-72k occupies two of them directly [2508.04416]. Stage-1 performs supervised fine-tuning on MTVR-CoT, the 54k text-based subset without tools. Its purpose is to teach basic video understanding, temporal grounding, question answering, grounded VQA, and the basic text-based CoT format. In this stage, `using_tools = False`, the learning rate is $1\text{e-}5$, and the batch size is $256`.

Stage-3 performs supervised fine-tuning with tools on MTVR-CoT-Tool, the 18k multimodal subset. This stage teaches the model how to call tools and how to integrate tool results into subsequent reasoning. Here, `using_tools = True`, `max_num_turns = 2`, the learning rate is again $1\text{e-}5$, and the batch size is $256`.

The other two stages use reinforcement learning data rather than MTVR-CoT-72k itself. Stage-2 applies DGRPO to MTVR-RL, a 94k no-tool dataset, and Stage-4 applies DGRPO to MTVR-RL-Tool, a 16k tool-augmented dataset. This makes the status of MTVR-CoT-72k precise: it is the supervised “cold start” substrate that establishes output formats, reasoning conventions, and tool-usage patterns before RL refinement. The paper further reports that RL with tools is inefficient and unstable without this cold-start supervision, as indicated by the cold-start ablations referenced in Table S9.

## 5. Distinguishing properties, quality controls, and limitations

Several properties distinguish MTVR-CoT-72k from earlier video datasets [2508.04416]. First, it is explicitly multi-task: TG, RQA, and GQA are trained jointly rather than as isolated objectives. The paper reports that these tasks are “mutually beneficial,” and the training-data-composition analysis referenced as Table 7 indicates that combining TG, RQA, and GQA yields the best performance. Second, it provides explicit CoT annotations rather than only final labels. Third, the tool subset provides tool-centric multimodal supervision, encoding step-wise tool usage patterns—especially `video_clip` invocation—within the reasoning trace. Fourth, the dataset is designed for long video reasoning, using sources such as VidChapters-7M and LongVideo-Reason and coupling long context with focused sub-clips. Fifth, its construction incorporates difficulty balancing through rollout filtering.

The quality controls are correspondingly explicit. Rollout-based filtering with Qwen2.5-VL-7B removes samples that are too easy or too hard in the sense of reward-range homogeneity, retaining those with informative variance across rollouts. Post-processing then removes incomplete traces, traces with incorrect final answers, and traces contaminated by off-topic or meta-level language. The resulting dataset is described as high-quality and balanced-difficulty.

The paper also states several limitations. The tasks and tools mainly cover temporal grounding and QA rather than spatial grounding or segmentation. Audio is not used, so the corpus is vision-text only; audio-dependent questions are therefore underrepresented or ignored. Video duration and token length distributions are broad, and the paper’s supplementary figures indicate large variation in reasoning complexity and domain vocabulary, but the main text does not provide an exact per-task count split for the SFT subsets.

## 6. Relation to MTVR-RL-110k, release status, and theoretical significance

MTVR-CoT-72k is separate from MTVR-RL-110k, the reinforcement-learning dataset used later in the same system [2508.04416]. The RL corpus is composed of MTVR-RL with 94k samples and MTVR-RL-Tool with 16k samples, for a total of approximately 110k samples. The relationship between the two corpora is functional rather than interchangeable: MTVR-CoT-72k provides clean supervised demonstrations of CoT formatting and tool usage, whereas MTVR-RL-110k supports policy optimization with rule-based rewards, including accuracy, format, and tool rewards plus IoU-based scaling and difficulty-aware adjustments.

The release plan is stated directly: all code, data, and model weight will be made publicly available. The paper further indicates that MTVR-CoT-72k will be released with video references, prompts, answers, temporal annotations, CoT sequences with tags, and tool schemas and traces for the tool subsets. Licensing is not specified in the main text.

A broader interpretive frame comes from the theory paper "CoT Information: Improved Sample Complexity under Chain-of-Thought Supervision" [2505.15927]. That work formalizes CoT learning with inputs $x$, outputs $y$, and reasoning traces $z$, and introduces the CoT information measure $\mathcal{I}_{\mathcal{D}, h_\star}^{\mathrm{CoT}}(\epsilon; \mathcal{H})$ as a way to quantify the discriminative value of observing reasoning traces in addition to final answers. Its central result is that, under CoT supervision, sample complexity can scale like $d / \mathcal{I}_{\mathcal{D}, h_\star}^{\mathrm{CoT}}(\epsilon; \mathcal{H})$ rather than $d / \epsilon$. This suggests that MTVR-CoT-72k’s emphasis on faithful multi-step traces, explicit `<think>` structure, and tool-grounded intermediate evidence is not only an engineering choice for VITAL but also a plausible mechanism for improving sample efficiency when the CoT structure is aligned with the target hypothesis class. Conversely, the same theory implies that noisy, inconsistent, or weakly informative rationales would reduce that benefit, which clarifies why the paper’s rollout filtering and post-processing are central rather than incidental.

Source: https://www.emergentmind.com/topics/mtvr-cot-72k