MTVR-RL-110k Video Reasoning Dataset

• MTVR-RL-110k is a reinforcement learning video reasoning dataset designed for long videos, supporting temporal grounding, VQA, and grounded VQA with multimodal chain-of-thought.
• It is constructed by curating diverse samples from established video datasets using rollout sampling and moderate difficulty filtering to ensure robust training signals.
• The dataset underpins RL post-training with DGRPO in the VITAL framework, contributing to state-of-the-art performance on multiple challenging video benchmarks.

Searching arXiv for the cited papers to ground the article in current paper metadata. MTVR-RL-110k is a reinforcement-learning-oriented video reasoning dataset introduced in the VITAL framework, "Video Intelligence via Tool-Augmented Learning," for long video reasoning with multimodal LLMs (MLLMs) (Zhang et al., 6 Aug 2025). It is constructed for RL post-training rather than as a standalone legacy-style benchmark, and it comprises approximately 110,000 samples spanning temporal grounding, video question answering (VQA), and grounded VQA. Each sample may include video, a question, a reasoning chain, tool calls and responses, and an answer; this suggests a primary role as supervision for multimodal chain-of-thought (CoT) and explicit tool use in long-video settings (Zhang et al., 6 Aug 2025).

1. Definition and intended function

MTVR-RL-110k was designed to support reinforcement learning post-training for MLLMs in a tool-augmented setting, specifically to enable long video reasoning (Zhang et al., 6 Aug 2025). The motivating problem is that recent text-based CoT approaches for multimodal LLMs often exhibit limited cross-modal interaction and increased hallucination, especially with longer videos or longer reasoning chains. Within this framing, the dataset is meant to teach models to leverage multimodal CoT reasoning involving both textual and visual cues, to encourage explicit step-wise tool usage, especially temporal localization via video clipping, and to cover a broad set of video-understanding tasks: temporal grounding, VQA, and grounded VQA (Zhang et al., 6 Aug 2025).

The dataset is part of a larger training recipe rather than an isolated corpus. The paper introduces two multi-task video reasoning datasets: MTVR-CoT-72k for supervised fine-tuning and MTVR-RL-110k for reinforcement learning (Zhang et al., 6 Aug 2025). This division reflects a methodological distinction between supervised CoT induction and RL-based policy improvement, with MTVR-RL-110k serving the latter stage.

2. Construction pipeline and corpus composition

MTVR-RL-110k is constructed by curating and converting samples from several established video datasets, including Charades-STA, ActivityNet-MR, VidChapters-7M, Video-R1, LongVideo-Reason, ReXTime, and NExT-GQA (Zhang et al., 6 Aug 2025). The generation pipeline begins with diverse rollout sampling: for each source sample, a strong multimodal LLM, such as Qwen2.5-VL or Gemini 2.5, generates $k=8$ rollouts using high temperature to encourage output diversity. Those rollouts are then filtered for moderate difficulty via two criteria. PassAll@k denotes that all rollouts are correct, in which case the sample is considered too easy and discarded; PassNone@k denotes that no rollout is correct, in which case the sample is considered too hard and discarded. Only moderately difficult samples are retained for RL (Zhang et al., 6 Aug 2025).

After filtering, reasoning labels are added. Text-based CoT is generated for all data, and multimodal CoT is provided especially for long videos, where each reasoning chain includes tool calls and visual evidence. For long video temporal grounding, tool parameters, specifically begin and end timestamps, are derived from ground truth and perturbed with 20% random noise to enhance robustness. For long video QA, tool usage is left open to model discretion. The final corpus is post-processed to remove incomplete or irrelevant samples and then organized into two RL-oriented splits (Zhang et al., 6 Aug 2025).

Component	Approx. size	Role
MTVR-RL	≈94k	Basic video reasoning samples for RL
MTVR-RL-Tool	≈16k	Multi-round tool-augmented long video reasoning samples for RL
MTVR-RL-110k	≈110k	Union of the RL-oriented splits

The paper further characterizes the resulting corpus as having rich video durations, complex temporal ranges, varied linguistic question/answer complexity, and coverage of both short and long video settings (Zhang et al., 6 Aug 2025). Supplementary material is reported to include token and duration distributions and word clouds illustrating coverage.

3. Task structure and annotation schema

MTVR-RL-110k covers three task categories. Temporal grounding requires a model, given a query, to localize the response to a time span defined by start and end times in the video. Video question answering includes open-ended, multiple-choice, numerical, OCR, and regression questions about video content. Grounded VQA requires the model to provide both an answer and a supporting temporal grounding (Zhang et al., 6 Aug 2025).

The dataset is multimodal at the sample level. Each sample can contain a video of varying length, a question, a reasoning chain, tool calls and responses, and an answer (Zhang et al., 6 Aug 2025). In the RL-Tool portion, reasoning chains can include tool call markup, and each chain can invoke a video clipping tool to extract or frame relevant video regions for subsequent reasoning. The annotation design therefore ties answer generation to intermediate evidence selection rather than treating temporal localization as a purely downstream evaluation artifact.

A common misunderstanding is to treat the tool component as uniformly mandatory across the corpus. The paper does not describe the dataset that way. Instead, it separates basic RL samples from tool-augmented RL samples, and it states explicitly that for long video QA, tool usage is left open to model discretion (Zhang et al., 6 Aug 2025). The tool layer is therefore integral to the dataset family, but not identically instantiated in every sample.

4. Role in RL post-training and DGRPO

Within VITAL, MTVR-RL-110k is the principal RL post-training resource for multi-task, multimodal, tool-augmented learning (Zhang et al., 6 Aug 2025). The associated optimization method is Difficulty-aware Group Relative Policy Optimization (DGRPO), which is introduced to mitigate task-wise and sample-wise difficulty imbalance in multi-task reinforcement learning. The dataset’s moderate-difficulty filtering is directly aligned with this objective: easy samples are removed because they provide weak learning signal, and extremely hard samples are removed because they can destabilize RL (Zhang et al., 6 Aug 2025).

DGRPO combines multiple reward components: an accuracy reward $\mathcal{R}_{acc}$, for example based on IoU, EM, or Rouge depending on task; a format reward $\mathcal{R}_{format}$ for output parsing correctness; and a tool reward $\mathcal{R}_{tool}$ for successful tool usage (Zhang et al., 6 Aug 2025). The algorithm also applies task-wise scaling, including normalization for grounding-oriented IoU rewards with hyperparameters $\alpha_i$ and $\beta_i$, and sample-wise balancing via clamped linear functions that upweight harder samples in the RL objective (Zhang et al., 6 Aug 2025). This makes MTVR-RL-110k not just a source of examples, but a corpus whose construction is explicitly coupled to the reward geometry of the training algorithm.

The training regimen described in the paper uses the data across several phases: supervised fine-tuning on text-based CoT data; RL post-training on MTVR-RL-110k, both without tools and with tools for multi-round reasoning; and tool-augmented SFT and RL on the MTVR-RL-Tool split for multimodal CoT (Zhang et al., 6 Aug 2025). This sequencing places the dataset at the interface between instruction tuning, CoT supervision, and policy optimization.

5. Evaluation relevance and reported empirical effects

The paper situates MTVR-RL-110k in a broad evaluation setting spanning 11 challenging video understanding benchmarks (Zhang et al., 6 Aug 2025). Relevant benchmarks include LongVideo-Reason and Video-MME for long video QA; VidChapters-7M, VUE-TR, Charades-STA, and ActivityNet-MR for temporal grounding; VSI-Bench, Video-MMMU, and MMVU for complex reasoning; and NExT-GQA and ReXTime for grounded VQA. The reported metrics include answer accuracy, mean Intersection over Union, Recall @ $x$, and area under the curve for precision, recall, or IoU, especially on long video tasks (Zhang et al., 6 Aug 2025).

The reported findings attribute a substantial portion of VITAL’s gains to the training enabled by MTVR-RL-110k. The paper states that VITAL achieves state-of-the-art results across long video QA and grounding, with VITAL-7B outperforming previous open-source models; specific examples given are LongVideo-Reason accuracy of 79.3% versus 67.9% and VidChapters-7M [email protected] of 34.7% versus 27.4% (Zhang et al., 6 Aug 2025). It also reports that tool-augmented RL, enabled by the MTVR-RL-Tool split, is decisive for lengthy and complex video reasoning and reduces hallucination via visual evidence integration (Zhang et al., 6 Aug 2025).

A further result concerns multi-task composition. The paper states that combining temporal grounding, reasoning VQA, and grounded VQA data from MTVR-RL-110k yields the best results, indicating complementary supervision signals across tasks (Zhang et al., 6 Aug 2025). It also argues that the combination of moderate-difficulty data construction and DGRPO’s reward balancing helps RL avoid stalling on easy samples or diverging on hard ones, leading to more robust generalization (Zhang et al., 6 Aug 2025).

6. Relation to the earlier MTVR dataset and terminological ambiguity

The label "MTVR" predates MTVR-RL-110k. In the 2021 paper "MTVR: Multilingual Moment Retrieval in Videos," MTVR denotes a large-scale multilingual video moment retrieval dataset containing 218K English and Chinese queries from 21.8K TV show video clips, collected by extending the TVR dataset with paired Chinese queries and subtitles (Lei et al., 2021). That earlier dataset is focused on multilingual moment retrieval, precise timestamp grounding, multilingual subtitles, and paired English-Chinese annotations; it is accompanied by the mXML model, which shares encoder parameters across languages and applies a language neighborhood constraint (Lei et al., 2021).

MTVR-RL-110k differs in task scope, annotation structure, and training role. It is a 2025 RL-oriented, multi-task video reasoning dataset for temporal grounding, VQA, grounded VQA, multimodal CoT, and tool use within VITAL (Zhang et al., 6 Aug 2025). By contrast, the 2021 MTVR dataset is a multilingual retrieval benchmark centered on English and Chinese query-subtitle-video alignment (Lei et al., 2021). A plausible implication is that the shared acronym can obscure the distinction between a multilingual retrieval benchmark and a later tool-augmented RL training corpus. For technical clarity, these resources are best treated as separate datasets associated with different problem formulations, despite the overlapping label.