MLLM4D-R1-30k: 4D Spatiotemporal RFT Data

• MLLM4D-R1-30k is a reinforcement fine-tuning dataset with 30k QA pairs emphasizing significant 4D motion and ground-truth physical supervision.
• The dataset is constructed from stereoscopic videos using a multi-stage pipeline that extracts 4D metadata and generates chain-of-thought reasoning annotated QA pairs.
• Incorporating GRPO and ST-CoT, its reward-grounded approach enhances model accuracy in dynamic scene understanding and spatiotemporal reasoning tasks.

MLLM4D-R1-30k is the 30,000-sample reinforcement fine-tuning dataset introduced within the MLLM-4D framework for 4D spatiotemporal reasoning from purely 2D RGB video inputs. It consists of question-answer pairs with significant 4D motion and ground-truth solutions, curated from stereoscopic videos and used after supervised fine-tuning on MLLM4D-2M and a short cold-start alignment stage to optimize spatiotemporal reasoning with Group Relative Policy Optimization (GRPO), Spatiotemporal Chain of Thought (ST-CoT), and spatiotemporal reward functions (Yin et al., 28 Feb 2026).

1. Position within the MLLM-4D data suite

Within MLLM-4D, MLLM4D-R1-30k occupies the second post-training stage. The framework separates data resources by function: MLLM4D-2M provides large-scale supervised fine-tuning data for foundational 4D understanding, MLLM4D-R1-30k provides reinforcement fine-tuning data for advanced 4D reasoning, and MLLM4D-Bench provides evaluation-only questions for model assessment (Yin et al., 28 Feb 2026).

Resource	Scale	Role
MLLM4D-2M	~2 million QA pairs	SFT for foundational 4D understanding
MLLM4D-R1-30k	30,000 QA pairs	RFT for advanced 4D reasoning
MLLM4D-Bench	6,000 questions	Evaluation benchmark

The distinction is functional rather than merely quantitative. MLLM4D-2M is designed for broad foundational spatiotemporal alignment, whereas MLLM4D-R1-30k emphasizes samples with significant 4D motion and ground-truth solutions suited to reward-based optimization. MLLM4D-Bench is not training data; it is organized into six subtasks for measuring dynamic 4D spatial-temporal understanding and reasoning. The paper also notes that MLLM4D-R1-30k appears to contain a subset or reweighted distribution of motion-centric scenarios relative to MLLM4D-2M, suggesting specialization for reasoning-intensive post-training rather than broad coverage alone (Yin et al., 28 Feb 2026).

2. Construction from stereoscopic video and physical metadata

MLLM4D-R1-30k is built by repurposing existing stereoscopic video datasets, specifically Stereo4D, through a three-stage pipeline: extraction of 4D spatial-temporal metadata, generation of 4D question-answer pairs, and generation of 4D reasoning data leading to MLLM4D-R1-30k (Yin et al., 28 Feb 2026).

The first stage extracts left-rectified RGB video frames $\{I_i\}_{i=1}^{K}$, camera poses $\{C_i\}_{i=1}^{K}$ with $C_i = [R_i \mid t_i]$, and per-frame metric 3D points $\{P_i\}_{i=1}^{K}$ from stereo depth. A robust filtering stage removes low-quality estimations. This metadata is then refined into object-centric trajectories. Gemini-2.5-Flash identifies moving entities and their noun categories; GroundedSAM2 performs instance segmentation and temporal tracking to produce masks $\mathrm{mask}_{i,m}$; scene-level 3D points are projected onto those masks to isolate per-frame object-level 3D points $\{P_{i,m}\}$; and PixelRefer produces fine-grained semantic descriptions $\{T_m\}_{m=1}^{M}$ such as object-level textual references.

The resulting representation combines per-frame camera poses, object-level metric 3D points, and semantic descriptions. These are then consumed by a physics-based solver that computes exact spatiotemporal relations rather than relying on language-model inference alone. The solver supports quantities such as camera absolute distance, camera relative direction, object absolute distance, object-camera absolute distance, object-camera relative distance, and object-camera relative direction. Template-based conversion then turns those physical quantities into natural-language multiple-choice questions.

For the broader MLLM-4D curation pipeline, the paper specifies additional filtering and balancing procedures: a cap on question-answer pairs per video to preserve scene diversity, shuffling of options to reduce position bias, and numerical distractors sampled from 25% to 175% of the true value. This process yields 2M high-quality QA pairs across approximately 100k videos for MLLM4D-2M, while MLLM4D-R1-30k is defined as the 30k-sample subset with significant 4D motion and ground-truth solutions for reinforcement fine-tuning. The exact filtering criterion for “significant 4D motion” is not specified in the paper (Yin et al., 28 Feb 2026).

A further reasoning-data generation stage uses Gemini-2.5-Pro with video frames, QA pairs, 4D physical values, and ST-CoT guidelines to synthesize detailed reasoning data. The paper explicitly states that this produces 7k cold-start samples and MLLM4D-R1-30k. This suggests tight coupling between the RFT dataset and ST-CoT-oriented supervision, although the exact extent of stored external chain-of-thought traces across all 30k samples is not fully specified (Yin et al., 28 Feb 2026).

3. Data contents, modalities, and task structure

A typical MLLM4D-R1-30k example contains sampled video frames, a question about a spatiotemporal relation, multiple-choice options, a ground-truth answer, and access to ground-truth physical quantities or solutions used for reward computation. The paper also implies the use of start and end frame indices, target object descriptions for object-centric questions, and ground-truth camera or object centers for key frames, at least during reward computation; however, whether these coordinates are explicit released fields or internally derived supervision is not specified (Yin et al., 28 Feb 2026).

The modality split is a defining feature. During model input, the system consumes only 2D RGB video frames and text. During curation and reward construction, however, the framework exploits stereo-derived 3D geometry, camera pose, and object-level metric 3D points. This design separates inference-time modality from supervision-time geometry: the model reasons over RGB video, while the dataset and reward machinery inject physically grounded 4D supervision.

For reinforcement fine-tuning, the expected output format is strict. The model generates:

$C_i = [R_i \mid t_i]$5

Inside <thinking>, the paper requires explicit spatial-state fields for initial and final frames, including camera center and object center in bracket-array form or null when inapplicable. This format is central to both parsing and reward assignment.

The task distribution reported for MLLM4D-R1-30k is narrower than that of the larger SFT corpus and more concentrated on motion-heavy scenarios (Yin et al., 28 Feb 2026).

Scenario group	Share	Subtask breakdown
Independent Object Motion	20.0%	Object absolute distance: 20.0%
Camera Ego-Motion	35.9%	Camera absolute distance: 19.2%; Camera relative direction: 16.7%
Object-Camera Dynamics	44.1%	Object-camera absolute distance: 19.2%; Object-camera relative distance: 16.7%; Object-camera relative direction: 8.3%

At the task level, the paper describes six core question types: camera absolute distance, camera relative direction, object absolute distance, object-camera absolute distance, object-camera relative distance, and object-camera relative direction. Elsewhere, the broader pipeline refers to seven scenarios, which likely indicates a finer partition of one of those families. The exact scenario taxonomy for MLLM4D-R1-30k beyond the percentages above is not completely specified (Yin et al., 28 Feb 2026).

4. GRPO, ST-CoT, and reward-grounded post-training

MLLM4D-R1-30k is the principal dataset for the reinforcement fine-tuning stage. After supervised fine-tuning and cold-start alignment, the model is optimized with GRPO using 12 rollouts per question, sampling temperature 1, 15k training steps, learning rate $5 \times 10^{-5}$, and KL coefficient $\beta = 0.1$. The reference model is the cold-start checkpoint. The reported hardware budget is 8 H100 80G GPUs, and RFT training time is 50 hours (Yin et al., 28 Feb 2026).

The ST-CoT template defines the reasoning structure encouraged during RFT. The paper describes a five-step organization: objective alignment and temporal anchoring; start-frame 3D state parsing; temporal progression and visual cue collection; end-frame 3D state verification; and evidence-based synthesis followed by the final answer. This format is not treated as presentation alone. It is directly coupled to reward functions that favor correct answers, proper structure, and physically grounded coordinate predictions.

The GRPO objective is given as:

$$L_{\mathrm{grpo}} = \mathbb{E}\left[ \frac{1}{G}\sum_{g=1}^{G} \min\left( P_g A_g,\; \mathrm{clip}(P_g, 1-\epsilon, 1+\epsilon)A_g \right) - \beta D_{\mathrm{KL}}(\pi_\theta \,\|\, \pi_{\mathrm{ref}}) \right],$$

with

$\{C_i\}_{i=1}^{K}$0

This formulation makes MLLM4D-R1-30k a reward-grounded dataset rather than a pure imitation corpus. Each sampled reasoning trajectory is scored, normalized relative to peers from the same question, and used to update the policy without a separate value model.

The reward decomposition is equally central. Final answer correctness is captured by:

$\{C_i\}_{i=1}^{K}$1

Format compliance is captured by:

$\{C_i\}_{i=1}^{K}$2

with appendix hyperparameters $\{C_i\}_{i=1}^{K}$3 and $\{C_i\}_{i=1}^{K}$4. The spatiotemporal grounding term maps coordinate error to a bounded reward:

$\{C_i\}_{i=1}^{K}$5

and combines camera and object terms as

$\{C_i\}_{i=1}^{K}$6

with $\{C_i\}_{i=1}^{K}$7 and $\{C_i\}_{i=1}^{K}$8 (Yin et al., 28 Feb 2026).

The total reward is then:

$\{C_i\}_{i=1}^{K}$9

with appendix weights $C_i = [R_i \mid t_i]$0, $C_i = [R_i \mid t_i]$1, and $C_i = [R_i \mid t_i]$2. The intended effect is to discourage spatiotemporal hallucinations by rewarding not only the correct option but also physically grounded intermediate state predictions and a parsable ST-CoT structure (Yin et al., 28 Feb 2026).

5. Empirical effect of the 30k RFT stage

The clearest direct evidence for MLLM4D-R1-30k comes from the paper’s RFT scaling study. On the real-world split of VLM4D, accuracy rises from a pre-RFT baseline of 52.1% to 58.5% with 1k RFT samples, 59.8% with 3k, 60.7% with 7k, 61.8% with 15k, and 63.1% with the full 30k set. The paper notes a minor initial fluctuation at 1k due to adaptation to the reasoning format, followed by a steady scaling trend beyond 3k (Yin et al., 28 Feb 2026).

A stage-wise ablation further isolates the contribution of RFT on MLLM4D-R1-30k. On MLLM4D-Bench and VLM4D respectively, the paper reports: baseline 35.3 and 52.1; SFT without the Section 3 data pipeline 59.9 and 56.2; SFT with the Section 3 data pipeline 70.1 and 59.7; GRPO without ST-reward 70.5 and 61.4; and GRPO with ST-reward 72.7 and 63.1. This progression indicates that RFT contributes beyond SFT alone, and that ST-reward provides additional gains beyond GRPO without explicit spatiotemporal grounding.

The full models trained with the complete MLLM-4D recipe, including the RFT stage on MLLM4D-R1-30k, achieve 72.7 on MLLM4D-Bench and $C_i = [R_i \mid t_i]$3 on VLM4D for the Qwen3-VL-8B backbone, and 70.2 on MLLM4D-Bench and $C_i = [R_i \mid t_i]$4 on VLM4D for the Qwen2.5-VL-7B backbone. The paper compares these against baseline Qwen3-VL-8B scores of 35.3 on MLLM4D-Bench and 52.2 on VLM4D, as well as against other baselines such as VG-LLM and Gemini-2.5-Pro. These end-to-end results reflect the entire framework rather than MLLM4D-R1-30k alone, but the stage ablations and scaling study identify the R1-30k stage as a material contributor to the final performance profile (Yin et al., 28 Feb 2026).

6. Scope, limitations, and disambiguation

MLLM4D-R1-30k is specifically a 4D spatiotemporal reinforcement fine-tuning dataset. It is not a general-purpose video corpus, not an evaluation benchmark, and not a static 3D reasoning dataset. Its scope is bounded by the MLLM-4D formulation: reasoning about the evolution of 3D space over time from 2D RGB inputs using supervision distilled from stereo geometry and physically computed labels (Yin et al., 28 Feb 2026).

The paper states several limitations and leaves several implementation details unspecified. The broader framework has difficulty with long-duration video sequences and still requires frame sampling because of input-length limits. Dataset quality depends on stereo-derived estimates, and the pipeline includes robust filtering to remove poor reconstructions. For MLLM4D-R1-30k specifically, the paper does not specify the exact thresholding rule for “significant 4D motion,” the exact schema of released records, whether all 30k samples contain fully stored external CoT annotations, or the final licensing and access conditions.

The name is also potentially confusable with other contemporary resources that share “R1” or “30K” terminology. “3D-R1” is a different system for static 3D scene understanding; its associated “Scene-30K” is a synthetic chain-of-thought dataset for 3D VLMs rather than a 4D spatiotemporal RFT dataset (Huang et al., 31 Jul 2025). Likewise, “R1dacted” is a text-only study of local censorship in DeepSeek R1 and introduces an unnamed approximately 10,030-prompt English dataset, not a multimodal 30k-scale resource (Naseh et al., 19 May 2025). These distinctions matter because MLLM4D-R1-30k is defined by its use of stereo-derived physical supervision, motion-centric QA construction, and reward-grounded spatiotemporal reasoning.

In that sense, the dataset’s distinguishing property is methodological rather than nominal. It is designed not merely to teach an MLLM which answer to select, but to support reward-driven refinement of reasoning trajectories in dynamic scenes through ST-CoT formatting and physically grounded camera and object state prediction.