---
title: 'MLLM4D-2M: 4D Video Supervision Dataset'
url: https://www.emergentmind.com/topics/mllm4d-2m
type: topic
---

# MLLM4D-2M: 4D Video Supervision Dataset

MLLM4D-2M is the core supervised fine-tuning dataset in the MLLM-4D framework, introduced to provide large-scale, automatically generated 4D spatiotemporal question-answer supervision from stereo videos for multimodal large language models [2603.00515]. It is designed to address a stated bottleneck in visual-based spatial-temporal intelligence: existing 4D instructional datasets are too small, too manually annotated, and too unstructured to support robust post-training at scale. Within MLLM-4D, MLLM4D-2M serves as the broad supervision source that teaches a standard video MLLM foundational 4D understanding from purely 2D RGB inputs, without architectural changes, by converting stereo video corpora into physics-grounded instruction data [2603.00515].

## 1. Motivation and problem setting

The paper situates MLLM4D-2M against the claim that current MLLMs remain bottlenecked on vision-based 4D spatial-temporal intelligence, defined as the ability to perceive and reason about the evolution of 3D space over time from visual inputs alone [2603.00515]. MLLM4D-2M is proposed specifically because prior 4D instructional resources are described as too limited for scalable learning. The paper states that prior benchmarks such as VLM4D are only on the order of a few thousand samples and are therefore not suitable for robust post-training [2603.00515].

In that framing, MLLM4D-2M is not merely an auxiliary dataset but the main data substrate enabling the MLLM-4D training strategy. Its function is to provide high-quality, physics-grounded supervision for short-horizon spatiotemporal reasoning tasks, so that a video MLLM can internalize correct spatial-temporal anchoring before later reinforcement fine-tuning [2603.00515]. A plausible implication is that the dataset is intended less as a benchmark artifact than as an infrastructure layer for post-training.

## 2. Construction pipeline from stereo videos

The data curation pipeline begins from stereoscopic video frames and repurposes them into explicit 4D supervision by extracting per-frame camera motion, object-level 3D geometry, and object semantics [2603.00515]. For each frame sequence $\{I_i\}_{i=1}^K$, the method obtains camera poses $\{C_i\}_{i=1}^K$, where each pose is written as $C_i=[R_i|t_i]$, together with per-frame metric 3D points $\{P_i\}_{i=1}^K$ [2603.00515]. The paper emphasizes that this stage is stereo-based rather than monocular because monocular reconstruction suffers from depth ambiguity and lower spatial accuracy [2603.00515].

A filtering stage removes low-quality estimates. Moving entities are then identified using Gemini-2.5-flash with an entity-extraction prompt that requests “active” entities and their noun categories [2603.00515]. Those entities are tracked with GroundedSAM2 to produce temporally consistent 2D masks, and scene-level 3D points are projected into the masks to obtain per-object 3D point sets $\{P_{i,m}\}$ [2603.00515]. PixelRefer is subsequently applied on frames plus masks to generate fine-grained semantic descriptions $\{T_m\}$ for each object [2603.00515].

The resulting structured representation consists of camera pose, object geometry, and object description. This representation is then used to generate all training QA pairs through physical computation and natural-language templating [2603.00515]. The paper describes the overall procedure algorithmically as stereo video $\rightarrow$ camera pose and depth $\rightarrow$ object tracking and description $\rightarrow$ geometric solving $\rightarrow$ template rendering $\rightarrow$ filtering and balancing [2603.00515].

## 3. Task taxonomy and physics-grounded supervision

MLLM4D-2M is explicitly physics-based. The paper defines several categories of spatiotemporal reasoning and derives ground-truth answers using geometric formulas [2603.00515]. The task space is organized into three broad groups: independent object motion, camera ego-motion, and object-camera dynamics. Independent object motion covers object absolute distance change. Camera ego-motion covers camera absolute distance and camera relative direction. Object-camera dynamics includes object-camera absolute distance, relative distance, and relative direction [2603.00515].

For camera relative direction, the displacement between frames $i$ and $j$ is first computed in world coordinates as $\Delta t=t_j-t_i$, and then projected into the local camera frame:
$$
d_{\text{cam\_rel\_dir}} = R_i^T \cdot \Delta t
$$
The resulting $[d_x,d_y,d_z]$ is interpreted with standard camera axes [2603.00515]. Other task labels are similarly derived from exact geometry: camera absolute distance is the Euclidean distance between camera centers; object absolute distance is the Euclidean distance between object centroids; object-camera absolute distance is the minimum distance between the camera center and the object point cloud; object-camera relative distance is determined by comparing distances at two frames against a threshold $T$; and object-camera relative direction classifies lateral and longitudinal changes in camera coordinates using a threshold [2603.00515].

The paper gives the decision rule for object-camera relative distance as:
$$
S = \begin{cases}
\text{farther} & \Delta d > T \\
\text{closer} & \Delta d < -T \\
\text{not moving} & \text{otherwise}
\end{cases}
$$
These computed quantities are converted into natural-language multiple-choice questions using templates [2603.00515]. The resulting supervision is therefore not based on free-form human annotation but on solved geometric quantities rendered into instruction format.

This design makes the dataset short-horizon and tightly grounded in video evidence. The paper lists representative questions such as how far the camera moved between two frames, whether a moving object became closer to or farther from the camera, whether the camera moved left, right, or forward relative to its start orientation, how far an object moved across the scene, and the absolute distance between a camera and an object at a chosen frame pair [2603.00515].

## 4. Scale, balance, and instruction formatting

After filtering, MLLM4D-2M contains 2 million high-quality QA pairs drawn from approximately 100,000 videos [2603.00515]. The paper states that the task distribution is balanced across the six benchmark-style components shown in Fig. 7, with the three broad groups appearing in roughly the proportions of the benchmark design [2603.00515]. In the dataset construction appendix, the authors further state that they “limit the number of QA pairs per video to maintain scene diversity,” shuffle multiple-choice options to remove positional bias, and generate distractors for numeric answers randomly within 25%–175% of the true value to avoid unrealistic alternatives [2603.00515].

These design choices distinguish MLLM4D-2M from a raw auto-labeled corpus. The dataset is filtered, balanced, and formatted into instruction-style examples suitable for supervised learning [2603.00515]. The training format is generally a video plus a natural-language question with multiple-choice answers, where the target is often the option letter [2603.00515]. This suggests that the curation process is intended not only to increase scale but also to regularize supervision so that it aligns with the response format expected from instruction-tuned MLLMs.

## 5. Function within the MLLM-4D training stack

Within the training pipeline, MLLM4D-2M is the large supervised fine-tuning corpus used to establish “foundational 4D spatiotemporal comprehension” before reinforcement fine-tuning [2603.00515]. In the SFT stage, the model is trained to answer the generated questions directly. The paper reports using LoRA on top of Qwen3-VL-8B-Instruct and Qwen2.5-VL-7B-Instruct with standard cross-entropy loss:
$$
L_{ce} = - \sum_j \log P(o^{(j)} \mid o^{(1:j-1)}, q, \{I_i\}_{i=1}^{N_k})
$$
where the input includes the sequence of video frames, the question, and the target answer trajectory [2603.00515].

The paper distinguishes MLLM4D-2M from the later reasoning-focused dataset MLLM4D-R1-30k. MLLM4D-R1-30k is derived from MLLM4D-2M by sampling across the seven original scenarios and using Gemini-2.5-Pro to generate multiple reasoning paths per example, followed by a scenario-specific reward threshold that keeps roughly the top-performing generations per scenario [2603.00515]. The cold-start set contains about 7,000 samples, and the final MLLM4D-R1-30k contains 30,000 QA pairs with significant motion and ground-truth solutions for GRPO training [2603.00515]. By contrast, MLLM4D-2M is the broad supervision source that teaches the underlying 4D concepts and aligns the model to answer the tasks correctly [2603.00515].

A concise comparison among the three principal data components in the framework is as follows:

| Component | Role | Scale |
|---|---|---|
| MLLM4D-2M | SFT training set | 2 million QA pairs from approximately 100,000 videos |
| MLLM4D-R1-30k | Reasoning-heavy RFT dataset | 30,000 QA pairs |
| MLLM4D-Bench | Evaluation suite | 6,000 questions across six subtasks |

The benchmark is described as more diagnostic and balanced, with fine-grained frame-wise temporal tagging, whereas MLLM4D-2M is the data engine for learning [2603.00515].

## 6. Empirical role and measured impact

The paper’s ablations directly attribute a central role to MLLM4D-2M in performance gains [2603.00515]. On VLM4D, scaling SFT data from 10K to 2M QA pairs yields steady improvement: the model improves from a 52.1% baseline to 57.7% at 200K samples and reaches 59.7% at 2M samples [2603.00515]. In the framework ablation table, adding SFT on top of the baseline improves MLLM4D-Bench performance from 35.3% to 70.1% [2603.00515]. The paper interprets this as confirmation that MLLM4D-2M establishes strong foundational 4D understanding.

The paper also compares the stereo-based curation pipeline to an alternative monocular-video pipeline and reports that the stereo-based pipeline gives better results [2603.00515]. This is consistent with the dataset’s premise that metric geometry quality matters in addition to corpus scale. Final performance with the full pipeline and ST-reward reaches 72.7% on MLLM4D-Bench and 63.1% on VLM4D, but these results are explicitly described as building on the SFT foundation provided by MLLM4D-2M [2603.00515].

A plausible interpretation is that the dataset’s main contribution is not isolated benchmark specialization but improved spatial-temporal grounding that subsequent reasoning procedures can exploit.

## 7. Assumptions, limitations, and prospective extensions

Several assumptions are explicit in the paper’s description of MLLM4D-2M. First, the pipeline assumes that stereo videos provide sufficiently accurate metric geometry for the physical solver; this is one reason the pipeline is stereo-based and the monocular alternative is treated as weaker [2603.00515]. Second, it assumes that moving entities can be reliably extracted and tracked by the combination of Gemini, GroundedSAM2, and PixelRefer [2603.00515]. Third, the tasks are short-horizon and template-based, reflecting the dataset’s emphasis on foundational competence rather than long-context reasoning [2603.00515].

The main limitation noted for the broader framework is difficulty with long-duration video because current architectures have input-length limits and therefore rely on frame sampling [2603.00515]. The paper states that future work should explore long-context 4D spatiotemporal reasoning [2603.00515]. It also frames MLLM4D-2M as a foundation for future extensions toward interactive AI systems such as robotics, VR/AR, and autonomous driving [2603.00515].

These limitations do not alter the dataset’s stated role. MLLM4D-2M is presented as the large-scale, stereo-video-derived instructional corpus that makes the MLLM-4D framework operational: it turns dynamic scenes into explicit, physics-grounded 4D supervision; powers supervised fine-tuning; is filtered, balanced, and templated; and is empirically validated through scaling and ablation studies as the main driver of the framework’s 4D understanding gains [2603.00515].

Source: https://www.emergentmind.com/topics/mllm4d-2m