---
title: MTVR-RL-110k Video Reasoning Dataset
url: https://www.emergentmind.com/topics/mtvr-rl-110k
type: topic
---

# MTVR-RL-110k Video Reasoning Dataset

Searching arXiv for the cited papers to ground the article in current paper metadata.
MTVR-RL-110k is a reinforcement-learning-oriented video reasoning dataset introduced in the VITAL framework, "Video Intelligence via Tool-Augmented Learning," for long video reasoning with multimodal large language models (MLLMs) [2508.04416]. It is constructed for RL post-training rather than as a standalone legacy-style benchmark, and it comprises approximately 110,000 samples spanning temporal grounding, video question answering (VQA), and grounded VQA. Each sample may include video, a question, a reasoning chain, tool calls and responses, and an answer; this suggests a primary role as supervision for multimodal chain-of-thought (CoT) and explicit tool use in long-video settings [2508.04416].

## 1. Definition and intended function

MTVR-RL-110k was designed to support reinforcement learning post-training for MLLMs in a tool-augmented setting, specifically to enable long video reasoning [2508.04416]. The motivating problem is that recent text-based CoT approaches for multimodal large language models often exhibit limited cross-modal interaction and increased hallucination, especially with longer videos or longer reasoning chains. Within this framing, the dataset is meant to teach models to leverage multimodal CoT reasoning involving both textual and visual cues, to encourage explicit step-wise tool usage, especially temporal localization via video clipping, and to cover a broad set of video-understanding tasks: temporal grounding, VQA, and grounded VQA [2508.04416].

The dataset is part of a larger training recipe rather than an isolated corpus. The paper introduces two multi-task video reasoning datasets: MTVR-CoT-72k for supervised fine-tuning and MTVR-RL-110k for reinforcement learning [2508.04416]. This division reflects a methodological distinction between supervised CoT induction and RL-based policy improvement, with MTVR-RL-110k serving the latter stage.

## 2. Construction pipeline and corpus composition

MTVR-RL-110k is constructed by curating and converting samples from several established video datasets, including Charades-STA, ActivityNet-MR, VidChapters-7M, Video-R1, LongVideo-Reason, ReXTime, and NExT-GQA [2508.04416]. The generation pipeline begins with diverse rollout sampling: for each source sample, a strong multimodal LLM, such as Qwen2.5-VL or Gemini 2.5, generates $k=8$ rollouts using high temperature to encourage output diversity. Those rollouts are then filtered for moderate difficulty via two criteria. PassAll@k denotes that all rollouts are correct, in which case the sample is considered too easy and discarded; PassNone@k denotes that no rollout is correct, in which case the sample is considered too hard and discarded. Only moderately difficult samples are retained for RL [2508.04416].

After filtering, reasoning labels are added. Text-based CoT is generated for all data, and multimodal CoT is provided especially for long videos, where each reasoning chain includes tool calls and visual evidence. For long video temporal grounding, tool parameters, specifically begin and end timestamps, are derived from ground truth and perturbed with 20% random noise to enhance robustness. For long video QA, tool usage is left open to model discretion. The final corpus is post-processed to remove incomplete or irrelevant samples and then organized into two RL-oriented splits [2508.04416].

| Component | Approx. size | Role |
|---|---:|---|
| MTVR-RL | ≈94k | Basic video reasoning samples for RL |
| MTVR-RL-Tool | ≈16k | Multi-round tool-augmented long video reasoning samples for RL |
| MTVR-RL-110k | ≈110k | Union of the RL-oriented splits |

The paper further characterizes the resulting corpus as having rich video durations, complex temporal ranges, varied linguistic question/answer complexity, and coverage of both short and long video settings [2508.04416]. Supplementary material is reported to include token and duration distributions and word clouds illustrating coverage.

## 3. Task structure and annotation schema

MTVR-RL-110k covers three task categories. Temporal grounding requires a model, given a query, to localize the response to a time span defined by start and end times in the video. Video question answering includes open-ended, multiple-choice, numerical, OCR, and regression questions about video content. Grounded VQA requires the model to provide both an answer and a supporting temporal grounding [2508.04416].

The dataset is multimodal at the sample level. Each sample can contain a video of varying length, a question, a reasoning chain, tool calls and responses, and an answer [2508.04416]. In the RL-Tool portion, reasoning chains can include tool call markup, and each chain can invoke a video clipping tool to extract or frame relevant video regions for subsequent reasoning. The annotation design therefore ties answer generation to intermediate evidence selection rather than treating temporal localization as a purely downstream evaluation artifact.

A common misunderstanding is to treat the tool component as uniformly mandatory across the corpus. The paper does not describe the dataset that way. Instead, it separates basic RL samples from tool-augmented RL samples, and it states explicitly that for long video QA, tool usage is left open to model discretion [2508.04416]. The tool layer is therefore integral to the dataset family, but not identically instantiated in every sample.

## 4. Role in RL post-training and DGRPO

Within VITAL, MTVR-RL-110k is the principal RL post-training resource for multi-task, multimodal, tool-augmented learning [2508.04416]. The associated optimization method is Difficulty-aware Group Relative Policy Optimization (DGRPO), which is introduced to mitigate task-wise and sample-wise difficulty imbalance in multi-task reinforcement learning. The dataset’s moderate-difficulty filtering is directly aligned with this objective: easy samples are removed because they provide weak learning signal, and extremely hard samples are removed because they can destabilize RL [2508.04416].

DGRPO combines multiple reward components: an accuracy reward $\mathcal{R}_{acc}$, for example based on IoU, EM, or Rouge depending on task; a format reward $\mathcal{R}_{format}$ for output parsing correctness; and a tool reward $\mathcal{R}_{tool}$ for successful tool usage [2508.04416]. The algorithm also applies task-wise scaling, including normalization for grounding-oriented IoU rewards with hyperparameters $\alpha_i$ and $\beta_i$, and sample-wise balancing via clamped linear functions that upweight harder samples in the RL objective [2508.04416]. This makes MTVR-RL-110k not just a source of examples, but a corpus whose construction is explicitly coupled to the reward geometry of the training algorithm.

The training regimen described in the paper uses the data across several phases: supervised fine-tuning on text-based CoT data; RL post-training on MTVR-RL-110k, both without tools and with tools for multi-round reasoning; and tool-augmented SFT and RL on the MTVR-RL-Tool split for multimodal CoT [2508.04416]. This sequencing places the dataset at the interface between instruction tuning, CoT supervision, and policy optimization.

## 5. Evaluation relevance and reported empirical effects

The paper situates MTVR-RL-110k in a broad evaluation setting spanning 11 challenging video understanding benchmarks [2508.04416]. Relevant benchmarks include LongVideo-Reason and Video-MME for long video QA; VidChapters-7M, VUE-TR, Charades-STA, and ActivityNet-MR for temporal grounding; VSI-Bench, Video-MMMU, and MMVU for complex reasoning; and NExT-GQA and ReXTime for grounded VQA. The reported metrics include answer accuracy, mean Intersection over Union, Recall @ $x$, and area under the curve for precision, recall, or IoU, especially on long video tasks [2508.04416].

The reported findings attribute a substantial portion of VITAL’s gains to the training enabled by MTVR-RL-110k. The paper states that VITAL achieves state-of-the-art results across long video QA and grounding, with VITAL-7B outperforming previous open-source models; specific examples given are LongVideo-Reason accuracy of 79.3% versus 67.9% and VidChapters-7M Recall@0.5 of 34.7% versus 27.4% [2508.04416]. It also reports that tool-augmented RL, enabled by the MTVR-RL-Tool split, is decisive for lengthy and complex video reasoning and reduces hallucination via visual evidence integration [2508.04416].

A further result concerns multi-task composition. The paper states that combining temporal grounding, reasoning VQA, and grounded VQA data from MTVR-RL-110k yields the best results, indicating complementary supervision signals across tasks [2508.04416]. It also argues that the combination of moderate-difficulty data construction and DGRPO’s reward balancing helps RL avoid stalling on easy samples or diverging on hard ones, leading to more robust generalization [2508.04416].

## 6. Relation to the earlier MTVR dataset and terminological ambiguity

The label "MTVR" predates MTVR-RL-110k. In the 2021 paper "MTVR: Multilingual Moment Retrieval in Videos," MTVR denotes a large-scale multilingual video moment retrieval dataset containing 218K English and Chinese queries from 21.8K TV show video clips, collected by extending the TVR dataset with paired Chinese queries and subtitles [2108.00061]. That earlier dataset is focused on multilingual moment retrieval, precise timestamp grounding, multilingual subtitles, and paired English-Chinese annotations; it is accompanied by the mXML model, which shares encoder parameters across languages and applies a language neighborhood constraint [2108.00061].

MTVR-RL-110k differs in task scope, annotation structure, and training role. It is a 2025 RL-oriented, multi-task video reasoning dataset for temporal grounding, VQA, grounded VQA, multimodal CoT, and tool use within VITAL [2508.04416]. By contrast, the 2021 MTVR dataset is a multilingual retrieval benchmark centered on English and Chinese query-subtitle-video alignment [2108.00061]. A plausible implication is that the shared acronym can obscure the distinction between a multilingual retrieval benchmark and a later tool-augmented RL training corpus. For technical clarity, these resources are best treated as separate datasets associated with different problem formulations, despite the overlapping label.

Source: https://www.emergentmind.com/topics/mtvr-rl-110k