MLLM-4D: 4D Spatiotemporal Intelligence

Updated 5 July 2026

MLLM-4D is a framework that advances MLLMs from static 2D video understanding to dynamic 4D spatial-temporal intelligence by reasoning over 3D space evolving over time.
It utilizes a stereo-video data curation pipeline along with a Spatiotemporal Chain of Thought to extract geometric and motion cues from RGB inputs.
The method achieves state-of-the-art performance through geometry-derived supervision and reinforcement tuning with GRPO, improving accuracy in spatiotemporal tasks.

Searching arXiv for papers on MLLM-4D and closely related 4D MLLM work. MLLM-4D denotes, in its most specific usage, the framework introduced in "MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence" for upgrading multimodal LLMs from ordinary 2D video understanding to visual-based 4D spatial-temporal intelligence, defined as the ability to reason about 3D space evolving over time from only RGB video input (Yin et al., 28 Feb 2026). In the surrounding literature, the term also functions as a broader organizing idea for MLLM systems that couple language reasoning with spatiotemporal representations, including region-level 4D video understanding, 4D LiDAR reasoning, explicit surgical world models, and physics-aware reconstruction (Yang et al., 18 Dec 2025, Choi et al., 7 Aug 2025, Fehrentz et al., 1 Apr 2026, Liu et al., 15 May 2026). This suggests that "MLLM-4D" is both a named framework and an emerging research direction centered on grounded reasoning over space, motion, and time.

1. Conceptual definition and problem setting

The core problem addressed by MLLM-4D is a gap between strong 2D semantic competence and weak dynamic geometric reasoning in current MLLMs. The framework defines 4D intelligence as reasoning in 3D space + time, including how camera pose, object position, object-camera distance, and relative direction evolve over a video (Yin et al., 28 Feb 2026). This is explicitly distinguished from static 3D reasoning: the target is not a single scene layout, but changing trajectories, motion direction, distance changes, and ego-motion.

A central claim of the framework is that this bottleneck does not require architecture modification. The paper states that a standard MLLM backbone with a pure 2D visual encoder is sufficient if trained with the right 4D data and reasoning objectives. Concretely, the reported backbones are Qwen3-VL-8B-Instruct and Qwen2.5-VL-7B-Instruct, adapted through post-training rather than a new multimodal architecture (Yin et al., 28 Feb 2026).

The motivating application domains are interactive systems such as robotics, VR/AR, autonomous driving, and embodied agents. The underlying rationale is that such systems must interpret both motion and geometry over time from visual input alone. A plausible implication is that MLLM-4D treats spatiotemporal reasoning less as a modality-expansion problem than as a supervision and post-training problem.

2. Data curation and benchmark construction

The original MLLM-4D framework is built around a stereo-video-based data engine that repurposes existing stereoscopic video datasets into instructional 4D data (Yin et al., 28 Feb 2026). The starting point is a sequence of stereo frames $\{I_i\}_{i=1}^{K}$ from which the pipeline extracts per-frame camera poses $\{C_i\}_{i=1}^{K}$ , per-frame metric 3D points $\{P_i\}_{i=1}^{K}$ , object-level 3D point clouds $\{P_{i,m}\}$ , and fine-grained semantic descriptions $\{T_m\}$ .

The metadata extraction procedure uses robust filtering, Gemini-2.5-flash to identify active moving entities and extract noun categories, GroundedSAM2 for instance segmentation and tracking, projection of scene-level 3D points onto object masks to isolate object-level 3D points, and PixelRefer to generate fine-grained object descriptions. The paper emphasizes stereo supervision because monocular video alone has depth ambiguity, making accurate large-scale 4D supervision difficult (Yin et al., 28 Feb 2026).

The resulting data are organized into three artifacts:

Artifact	Role	Scale
MLLM4D-2M	Supervised Fine-Tuning	2 million QA pairs across about 100k videos
MLLM4D-R1-30k	Reinforcement Fine-Tuning	30k QA pairs; includes a 7k cold-start set
MLLM4D-Bench	Evaluation	6k questions organized into six subtasks

The QA generation is physically grounded. The benchmark and training data cover three logical groups: Independent Object Motion, Camera Ego-Motion, and Object-Camera Dynamics. The six subtasks are object absolute distance, camera absolute distance, camera relative direction, object-camera absolute distance, object-camera relative distance, and object-camera relative direction (Yin et al., 28 Feb 2026).

Several answer types are computed directly from geometry. For camera relative direction, the world-space displacement is first defined as $\Delta t = t_j - t_i$ and then projected into camera coordinates as

$d_{\text{cam\_rel\_dir}} = R_i^{\top} \cdot \Delta t.$

For object-camera absolute distance, the paper gives

$d_{\text{obj-cam\_abs}} = \min \left\| P_{j,m} - \text{Center}_i \right\|_2.$

For relative distance, the change $\Delta d = d_j - d_i$ is thresholded into farther, closer, or not moving (Yin et al., 28 Feb 2026). The significance of this design is that the supervision is generated from metric 3D evidence rather than free-form captioning alone.

3. Post-training recipe: SFT, GRPO, ST-CoT, and ST-reward

The training strategy has two stages: SFT for foundational 4D understanding and RFT with GRPO for reasoning improvement (Yin et al., 28 Feb 2026). During SFT, the model is fine-tuned on MLLM4D-2M with LoRA using standard cross-entropy:

$L_{ce} = - \sum_j \log P(o^{(j)} \mid o^{(1:j-1)}, q, \{I_i\}_{i=1}^{N_k}).$

The paper characterizes this stage as teaching the model to identify relevant frames, localize camera and object centers, understand the intended 4D category of the question, and ground answers in visual motion rather than generic language priors.

The second stage applies Group Relative Policy Optimization (GRPO) on MLLM4D-R1-30k. For each question, the current policy samples a group of outputs, computes normalized within-group advantages, and optimizes a clipped policy objective regularized by a KL term to a frozen reference model (Yin et al., 28 Feb 2026). The cold-start set is used to align the model's reasoning format before full GRPO.

A major contribution is Spatiotemporal Chain of Thought (ST-CoT), a reasoning format intended to force the model to behave like a visual physics engine rather than a purely linguistic reasoner. The paper specifies five steps:

objective alignment and temporal anchoring
start-frame 3D state parsing
temporal progression and evidence collection
end-frame 3D state verification
evidence-based synthesis and probabilistic inference

The reward design extends beyond answer correctness. The total reward is

$\{C_i\}_{i=1}^{K}$ 0

Here, $\{C_i\}_{i=1}^{K}$ 1 checks both required output structure and the explicit provision of Object Center and Camera Center in bracketed array format, while $\{C_i\}_{i=1}^{K}$ 2 penalizes Euclidean error between predicted and ground-truth centers through exponentially decaying coordinate rewards (Yin et al., 28 Feb 2026). The intended effect is to suppress hallucinated motion and enforce geometric consistency during reinforcement fine-tuning.

4. Empirical performance and ablation results

On MLLM4D-Bench, the best reported model is MLLM-4D (Qwen3-VL-8B) with 72.7% average accuracy, compared with 46.6% for Gemini2.5-Pro, 44.9% for GPT-4o, 35.3% for the Qwen3-VL-8B baseline, and 56.3% for VG-LLM (Yin et al., 28 Feb 2026). The MLLM-4D (Qwen2.5-VL-7B) variant reaches 70.2%.

On VLM4D, the reported generalization results are 61.08% for MLLM-4D (Qwen3-VL-8B) and 57.0% for MLLM-4D (Qwen2.5-VL-7B), again above the corresponding base models and specialized 3D reasoning methods (Yin et al., 28 Feb 2026). The qualitative characterization is that the model explains motion in a physically coherent way, distinguishes whether objects are closer or farther, and separates camera motion from object motion.

The ablations are central to the framework's interpretation. Data scaling improves performance: on SFT, accuracy on VLM4D rises from 52.1% baseline to 57.7% at 200K samples and 59.7% at 2M samples; on RFT, accuracy scales to 63.1% at 30k samples (Yin et al., 28 Feb 2026). Table-level component analysis shows that SFT without the curation pipeline is much weaker than SFT with the pipeline, GRPO without ST-reward improves over SFT but is weaker than the full method, and GRPO with ST-reward performs best. The paper also reports that the stereo-based pipeline is superior to an alternative monocular-video pipeline because monocular supervision suffers from depth ambiguity and less accurate geometry.

A common misconception is that MLLM-4D necessarily depends on explicit 3D inference modules at test time. The original framework argues the opposite: its central claim is that strong 4D reasoning can be induced by scalable geometry-derived supervision and post-training while keeping the base architecture unchanged (Yin et al., 28 Feb 2026).

Although the original MLLM-4D paper is architecture-agnostic and RGB-only, related work shows that the broader 4D MLLM space has diversified rapidly. One line emphasizes stronger spatial priors from 2D video alone. Spatial-MLLM uses a dual-encoder design in which a pretrained 2D visual encoder extracts semantic features and a spatial encoder initialized from the backbone of VGGT extracts 3D structure features, followed by simple additive fusion. It also introduces space-aware frame sampling by casting frame selection as a maximum coverage problem over voxelized scene coverage. On VSI-Bench, Spatial-MLLM-4B achieves 48.4 average accuracy, exceeding Gemini-1.5 Pro's 45.4, with especially strong gains on numerical spatial tasks such as object counting (65.3) and absolute distance (34.8); on ScanQA it reaches 44.4 BLEU-1, 14.8 BLEU-4, 18.4 METEOR, 45.0 ROUGE-L, and 91.8 CIDEr; on SQA3D it obtains 55.9 EM-1 and 58.7 EM-R1 (Wu et al., 29 May 2025).

A second line emphasizes region-grounded dynamic perception. 4D-RGPT introduces Perceptual 4D Distillation (P4D), which transfers latent 4D embeddings and explicit depth, flow, motion, and camray signals from a frozen expert model into an MLLM without inference-time overhead. It also adds Timestamp Positional Encoding (TPE) and evaluates on R4D-Bench, a region-level benchmark with 1,517 multiple-choice VQAs, including 418 static and 1,098 dynamic questions. The model improves over NVILA-Lite-8B on STI-Bench (33.8 → 37.6), VLM4D-real (46.5 → 52.7), and VSTI-Bench (45.2 → 59.1), while on R4D-Bench it improves average performance from 37.9 to 42.2 and the dynamic split from 41.3 to 45.7 (Yang et al., 18 Dec 2025).

A third line moves beyond RGB video into other 4D substrates. B4DL defines a benchmark for raw 4D LiDAR with sequences

$\{C_i\}_{i=1}^{K}$ 3

introduces a CLIP-aligned LiDAR encoder, a linear LiDAR aligner, and a Metatoken for motion metadata, and reports Accuracy 0.762, mIoU 0.311, BLEU-4 0.095, ROUGE-L 0.322, METEOR 0.275, BERTScore 0.897, and GPT Score 59.513 (Choi et al., 7 Aug 2025). This broadens the meaning of MLLM-4D from video-based geometric reasoning to LiDAR-native spatiotemporal language modeling.

Medical and manipulation settings further expand the term. In surgery, one framework builds an explicit tracked semantic 4D representation from monocular laparoscopic video using Depth Anything 3, CoTracker 3, and SASVi, then exposes tools such as minimum distance, 3D overlap, relative motion, and 4D trajectories to a backbone-agnostic MLLM for training-free agentic reasoning. On a benchmark of 134 clinically relevant questions, it reports spatial improvement from 90.76 L2 for the MLLM baseline to 46.54 L2 for the agent, and directional error reduction from 0.83 for MLLM + SoM to 0.18 for the agent (Fehrentz et al., 1 Apr 2026). In endoscopy, EndoGSim combines 4D Gaussian Splatting, Surgical-SAM-2, a pre-trained depth estimator, MLLM-based material initialization, and differentiable MLS-MPM to produce physics-aware dynamic scene simulation, achieving average RE/EPE of 0.139/0.184 and surgeon-rated physical realism of 4.02 (Liu et al., 15 May 2026). In hand-articulated-object reconstruction, ArtHOI uses MLLM-derived frame-wise contact states and contacting fingers as optimization constraints, improving the reported $\{C_i\}_{i=1}^{K}$ 4 score from 0.046 / 0.035 / 0.059 without the MLLM to 0.029 / 0.022 / 0.039 with the MLLM on ArtHOI-RGBD / RSRD / ArtHOI-Wild (Wang et al., 26 Mar 2026).

Taken together, these works indicate that 4D MLLM research has split into at least three methodological families: implicit 4D reasoning from RGB video, explicit 4D world-model construction, and non-RGB 4D sensing such as LiDAR. This suggests that "4D" is converging semantically around persistent space-time structure, even when the underlying representation differs.

6. Evaluation ecology, security, and unresolved issues

As the area broadens, evaluation has become a distinct research problem. 4DWorldBench proposes a unified framework for 3D/4D world generation models across four dimensions: Perceptual Quality, Condition-4D Alignment, Physical Realism, and 4D Consistency. It supports Image-to-3D/4D, Video-to-4D, and Text-to-3D/4D, maps text, image, and video conditions into a unified textual space during evaluation, and combines LLM-as-judge, MLLM-as-judge, and traditional network-based metrics. Human studies report improved agreement with subjective judgment, including attribute-control PLCC/SRCC increasing from 0.167 / 0.236 to 0.483 / 0.443 under the proposed adaptive method (Lu et al., 25 Nov 2025). The benchmark's decomposition is consequential because current models may be strong on style consistency or scene-level semantics while remaining weak on motion reasoning, physics, and stable 4D semantics over time.

On the data-generation side, Follow-Your-Instruction treats MLLMs as a control plane for synthesizing 2D, 3D, and 4D world data. Its pipeline comprises MLLM-Collector, MLLM-Generator, MLLM-Optimizer, and MLLM-Planner, with VLM-guided frame prediction used to improve temporal smoothness and consistency. In a downstream 4D comparison using ReCamMaster, adding the synthetic data improves FID 62.48 → 60.32, FVD 160.72 → 155.71, CLIP-T 34.97 → 35.88, RotErr 1.45 → 1.35, and TransErr 5.22 → 4.69; its frame-prediction ablation reports Temporal Consis. 0.6524 → 0.9128 when VLM guidance is added (Feng et al., 7 Aug 2025). This suggests that the 4D ecosystem is now shaped not only by models and benchmarks, but also by synthetic data engines.

The security picture is correspondingly more complex. "The Wolf Within" studies a multi-agent "society of MLLMs" in which one compromised multimodal agent generates a malicious prompt that causes downstream agents to emit harmful content. The work evaluates 14 prohibited scenarios and reports high Attack Success Rate values, including 88.24% on illegal activity and several 100% scores for LLaVA, with transferability across sheep models also demonstrated (Tan et al., 2024). For MLLM-4D-style systems, this is relevant because spatiotemporal pipelines increasingly involve model-to-model communication, tool use, and chained intermediate representations. A plausible implication is that 4D grounding does not remove prompt-propagation risk; it may enlarge the attack surface when multiple agents exchange summaries, plans, or tool outputs.

The literature also contains a cautionary bibliographic anomaly. The arXiv entry titled "Survey on AI-Generated Media Detection: From Non-MLLM to MLLM" is, according to the extracted details, actually an IEEEtran LaTeX template/how-to document and contains no domain content about detection methods or MLLM-4D (Zou et al., 7 Feb 2025). For a field with rapidly proliferating terminology, this is a reminder that title-level retrieval alone is insufficient.

The major open issues named across the corpus are consistent: long-video processing under frame-budget constraints, depth ambiguity when stereo or stronger geometry is unavailable, dependence on proxy annotations or proxy modalities during dataset construction, temporal inference without explicit cues, hallucination in safety-critical deployments, and the persistent challenge of evaluating physical realism rather than only semantic correctness (Yin et al., 28 Feb 2026, Yang et al., 18 Dec 2025, Choi et al., 7 Aug 2025, Ping et al., 15 Jun 2025). The cumulative direction of the literature is clear: MLLM-4D is moving from a narrowly defined RGB video post-training recipe toward a general program for language-mediated reasoning over explicit or implicit world models in 3D space and time.