MLLM4D-2M: 4D Video Supervision Dataset

• The paper introduces MLLM4D-2M as the core supervised fine-tuning dataset that converts stereo videos into physics-grounded 4D instructional QA pairs.
• It details a stereo-based curation pipeline that extracts camera poses, object geometry, and object dynamics to generate balanced, high-quality spatiotemporal supervision.
• Empirical evaluations show that scaling the dataset from 10K to 2M QA pairs drives significant gains in spatial-temporal performance on MLLM-4D benchmarks.

MLLM4D-2M is the core supervised fine-tuning dataset in the MLLM-4D framework, introduced to provide large-scale, automatically generated 4D spatiotemporal question-answer supervision from stereo videos for multimodal LLMs (Yin et al., 28 Feb 2026). It is designed to address a stated bottleneck in visual-based spatial-temporal intelligence: existing 4D instructional datasets are too small, too manually annotated, and too unstructured to support robust post-training at scale. Within MLLM-4D, MLLM4D-2M serves as the broad supervision source that teaches a standard video MLLM foundational 4D understanding from purely 2D RGB inputs, without architectural changes, by converting stereo video corpora into physics-grounded instruction data (Yin et al., 28 Feb 2026).

1. Motivation and problem setting

The paper situates MLLM4D-2M against the claim that current MLLMs remain bottlenecked on vision-based 4D spatial-temporal intelligence, defined as the ability to perceive and reason about the evolution of 3D space over time from visual inputs alone (Yin et al., 28 Feb 2026). MLLM4D-2M is proposed specifically because prior 4D instructional resources are described as too limited for scalable learning. The paper states that prior benchmarks such as VLM4D are only on the order of a few thousand samples and are therefore not suitable for robust post-training (Yin et al., 28 Feb 2026).

In that framing, MLLM4D-2M is not merely an auxiliary dataset but the main data substrate enabling the MLLM-4D training strategy. Its function is to provide high-quality, physics-grounded supervision for short-horizon spatiotemporal reasoning tasks, so that a video MLLM can internalize correct spatial-temporal anchoring before later reinforcement fine-tuning (Yin et al., 28 Feb 2026). A plausible implication is that the dataset is intended less as a benchmark artifact than as an infrastructure layer for post-training.

2. Construction pipeline from stereo videos

The data curation pipeline begins from stereoscopic video frames and repurposes them into explicit 4D supervision by extracting per-frame camera motion, object-level 3D geometry, and object semantics (Yin et al., 28 Feb 2026). For each frame sequence $\{I_i\}_{i=1}^K$, the method obtains camera poses $\{C_i\}_{i=1}^K$, where each pose is written as $C_i=[R_i|t_i]$, together with per-frame metric 3D points $\{P_i\}_{i=1}^K$ (Yin et al., 28 Feb 2026). The paper emphasizes that this stage is stereo-based rather than monocular because monocular reconstruction suffers from depth ambiguity and lower spatial accuracy (Yin et al., 28 Feb 2026).

A filtering stage removes low-quality estimates. Moving entities are then identified using Gemini-2.5-flash with an entity-extraction prompt that requests “active” entities and their noun categories (Yin et al., 28 Feb 2026). Those entities are tracked with GroundedSAM2 to produce temporally consistent 2D masks, and scene-level 3D points are projected into the masks to obtain per-object 3D point sets $\{P_{i,m}\}$ (Yin et al., 28 Feb 2026). PixelRefer is subsequently applied on frames plus masks to generate fine-grained semantic descriptions $\{T_m\}$ for each object (Yin et al., 28 Feb 2026).

The resulting structured representation consists of camera pose, object geometry, and object description. This representation is then used to generate all training QA pairs through physical computation and natural-language templating (Yin et al., 28 Feb 2026). The paper describes the overall procedure algorithmically as stereo video $\rightarrow$ camera pose and depth $\rightarrow$ object tracking and description $\rightarrow$ geometric solving $\rightarrow$ template rendering $\{C_i\}_{i=1}^K$0 filtering and balancing (Yin et al., 28 Feb 2026).

3. Task taxonomy and physics-grounded supervision

MLLM4D-2M is explicitly physics-based. The paper defines several categories of spatiotemporal reasoning and derives ground-truth answers using geometric formulas (Yin et al., 28 Feb 2026). The task space is organized into three broad groups: independent object motion, camera ego-motion, and object-camera dynamics. Independent object motion covers object absolute distance change. Camera ego-motion covers camera absolute distance and camera relative direction. Object-camera dynamics includes object-camera absolute distance, relative distance, and relative direction (Yin et al., 28 Feb 2026).

For camera relative direction, the displacement between frames $\{C_i\}_{i=1}^K$1 and $\{C_i\}_{i=1}^K$2 is first computed in world coordinates as $\{C_i\}_{i=1}^K$3, and then projected into the local camera frame:

$\{C_i\}_{i=1}^K$4

The resulting $\{C_i\}_{i=1}^K$5 is interpreted with standard camera axes (Yin et al., 28 Feb 2026). Other task labels are similarly derived from exact geometry: camera absolute distance is the Euclidean distance between camera centers; object absolute distance is the Euclidean distance between object centroids; object-camera absolute distance is the minimum distance between the camera center and the object point cloud; object-camera relative distance is determined by comparing distances at two frames against a threshold $\{C_i\}_{i=1}^K$6; and object-camera relative direction classifies lateral and longitudinal changes in camera coordinates using a threshold (Yin et al., 28 Feb 2026).

The paper gives the decision rule for object-camera relative distance as:

$\{C_i\}_{i=1}^K$7

These computed quantities are converted into natural-language multiple-choice questions using templates (Yin et al., 28 Feb 2026). The resulting supervision is therefore not based on free-form human annotation but on solved geometric quantities rendered into instruction format.

This design makes the dataset short-horizon and tightly grounded in video evidence. The paper lists representative questions such as how far the camera moved between two frames, whether a moving object became closer to or farther from the camera, whether the camera moved left, right, or forward relative to its start orientation, how far an object moved across the scene, and the absolute distance between a camera and an object at a chosen frame pair (Yin et al., 28 Feb 2026).

4. Scale, balance, and instruction formatting

After filtering, MLLM4D-2M contains 2 million high-quality QA pairs drawn from approximately 100,000 videos (Yin et al., 28 Feb 2026). The paper states that the task distribution is balanced across the six benchmark-style components shown in Fig. 7, with the three broad groups appearing in roughly the proportions of the benchmark design (Yin et al., 28 Feb 2026). In the dataset construction appendix, the authors further state that they “limit the number of QA pairs per video to maintain scene diversity,” shuffle multiple-choice options to remove positional bias, and generate distractors for numeric answers randomly within 25%–175% of the true value to avoid unrealistic alternatives (Yin et al., 28 Feb 2026).

These design choices distinguish MLLM4D-2M from a raw auto-labeled corpus. The dataset is filtered, balanced, and formatted into instruction-style examples suitable for supervised learning (Yin et al., 28 Feb 2026). The training format is generally a video plus a natural-language question with multiple-choice answers, where the target is often the option letter (Yin et al., 28 Feb 2026). This suggests that the curation process is intended not only to increase scale but also to regularize supervision so that it aligns with the response format expected from instruction-tuned MLLMs.

5. Function within the MLLM-4D training stack

Within the training pipeline, MLLM4D-2M is the large supervised fine-tuning corpus used to establish “foundational 4D spatiotemporal comprehension” before reinforcement fine-tuning (Yin et al., 28 Feb 2026). In the SFT stage, the model is trained to answer the generated questions directly. The paper reports using LoRA on top of Qwen3-VL-8B-Instruct and Qwen2.5-VL-7B-Instruct with standard cross-entropy loss:

$\{C_i\}_{i=1}^K$8

where the input includes the sequence of video frames, the question, and the target answer trajectory (Yin et al., 28 Feb 2026).

The paper distinguishes MLLM4D-2M from the later reasoning-focused dataset MLLM4D-R1-30k. MLLM4D-R1-30k is derived from MLLM4D-2M by sampling across the seven original scenarios and using Gemini-2.5-Pro to generate multiple reasoning paths per example, followed by a scenario-specific reward threshold that keeps roughly the top-performing generations per scenario (Yin et al., 28 Feb 2026). The cold-start set contains about 7,000 samples, and the final MLLM4D-R1-30k contains 30,000 QA pairs with significant motion and ground-truth solutions for GRPO training (Yin et al., 28 Feb 2026). By contrast, MLLM4D-2M is the broad supervision source that teaches the underlying 4D concepts and aligns the model to answer the tasks correctly (Yin et al., 28 Feb 2026).

A concise comparison among the three principal data components in the framework is as follows:

Component	Role	Scale
MLLM4D-2M	SFT training set	2 million QA pairs from approximately 100,000 videos
MLLM4D-R1-30k	Reasoning-heavy RFT dataset	30,000 QA pairs
MLLM4D-Bench	Evaluation suite	6,000 questions across six subtasks

The benchmark is described as more diagnostic and balanced, with fine-grained frame-wise temporal tagging, whereas MLLM4D-2M is the data engine for learning (Yin et al., 28 Feb 2026).

6. Empirical role and measured impact

The paper’s ablations directly attribute a central role to MLLM4D-2M in performance gains (Yin et al., 28 Feb 2026). On VLM4D, scaling SFT data from 10K to 2M QA pairs yields steady improvement: the model improves from a 52.1% baseline to 57.7% at 200K samples and reaches 59.7% at 2M samples (Yin et al., 28 Feb 2026). In the framework ablation table, adding SFT on top of the baseline improves MLLM4D-Bench performance from 35.3% to 70.1% (Yin et al., 28 Feb 2026). The paper interprets this as confirmation that MLLM4D-2M establishes strong foundational 4D understanding.

The paper also compares the stereo-based curation pipeline to an alternative monocular-video pipeline and reports that the stereo-based pipeline gives better results (Yin et al., 28 Feb 2026). This is consistent with the dataset’s premise that metric geometry quality matters in addition to corpus scale. Final performance with the full pipeline and ST-reward reaches 72.7% on MLLM4D-Bench and 63.1% on VLM4D, but these results are explicitly described as building on the SFT foundation provided by MLLM4D-2M (Yin et al., 28 Feb 2026).

A plausible interpretation is that the dataset’s main contribution is not isolated benchmark specialization but improved spatial-temporal grounding that subsequent reasoning procedures can exploit.

7. Assumptions, limitations, and prospective extensions

Several assumptions are explicit in the paper’s description of MLLM4D-2M. First, the pipeline assumes that stereo videos provide sufficiently accurate metric geometry for the physical solver; this is one reason the pipeline is stereo-based and the monocular alternative is treated as weaker (Yin et al., 28 Feb 2026). Second, it assumes that moving entities can be reliably extracted and tracked by the combination of Gemini, GroundedSAM2, and PixelRefer (Yin et al., 28 Feb 2026). Third, the tasks are short-horizon and template-based, reflecting the dataset’s emphasis on foundational competence rather than long-context reasoning (Yin et al., 28 Feb 2026).

The main limitation noted for the broader framework is difficulty with long-duration video because current architectures have input-length limits and therefore rely on frame sampling (Yin et al., 28 Feb 2026). The paper states that future work should explore long-context 4D spatiotemporal reasoning (Yin et al., 28 Feb 2026). It also frames MLLM4D-2M as a foundation for future extensions toward interactive AI systems such as robotics, VR/AR, and autonomous driving (Yin et al., 28 Feb 2026).

These limitations do not alter the dataset’s stated role. MLLM4D-2M is presented as the large-scale, stereo-video-derived instructional corpus that makes the MLLM-4D framework operational: it turns dynamic scenes into explicit, physics-grounded 4D supervision; powers supervised fine-tuning; is filtered, balanced, and templated; and is empirically validated through scaling and ablation studies as the main driver of the framework’s 4D understanding gains (Yin et al., 28 Feb 2026).