MLLM4D-Bench: 4D Visual Reasoning Benchmark

• MLLM4D-Bench is a physics-grounded evaluation benchmark that measures 4D spatial-temporal intelligence from monocular RGB video.
• It assesses combined capabilities in 3D spatial understanding, temporal evolution, and object–camera dynamics using stereo-derived metadata.
• The benchmark features 6,000 multiple-choice questions over six subtasks, providing a diagnostic tool for precise metric and geometric reasoning.

Searching arXiv for the benchmark and closely related 4D MLLM evaluation papers. MLLM4D-Bench is a physics-grounded evaluation benchmark for visual 4D spatial-temporal intelligence in multimodal LLMs, introduced as the held-out evaluation component of the MLLM-4D framework (Yin et al., 28 Feb 2026). It is designed to measure whether a model can infer metric 3D structure, temporal evolution, and joint object–camera dynamics from purely 2D RGB video, rather than from explicit depth, pose, or 3D supervision at inference time. Within the broader MLLM-4D ecosystem, it complements MLLM4D-2M, a 2M-question corpus for supervised fine-tuning, and MLLM4D-R1-30k, a 30k-question corpus for reinforcement fine-tuning, by providing a structured 6,000-question benchmark for standardized evaluation (Yin et al., 28 Feb 2026).

1. Definition and scope

MLLM4D-Bench targets what the MLLM-4D authors describe as vision-based 4D spatial-temporal intelligence: perception and reasoning over the evolution of 3D space through time from RGB video alone (Yin et al., 28 Feb 2026). In operational terms, the benchmark evaluates three capabilities simultaneously: 3D spatial understanding, temporal evolution, and 4D object–camera dynamics. This focus distinguishes it from static 3D benchmarks such as ScanQA, SQA3D, and OmniSpatial, which emphasize spatial reasoning in fixed scenes, and from conventional video QA datasets such as NExT-QA and MSRVTT-QA, which predominantly test semantic activity understanding rather than metric geometry or precise object–camera motion in 3D (Yin et al., 28 Feb 2026).

A central design feature is that benchmark labels are not derived from human impressionistic judgments about motion. Instead, they are computed from stereo-derived 4D reconstructions and camera poses, making the benchmark explicitly physics-grounded (Yin et al., 28 Feb 2026). This means that questions about distance, direction, and relative motion are tied to computed geometric quantities rather than free-form annotation. A common misconception is therefore to treat MLLM4D-Bench as a generic video QA suite; more precisely, it is a metric, geometry-centered 4D benchmark in which language questions are generated from reconstructed scene dynamics.

Another misconception is that the benchmark evaluates native 3D or stereo inference. In fact, models receive only monocular RGB inputs at test time: the left rectified RGB frames from short stereo-based video sequences (Yin et al., 28 Feb 2026). The stereo pipeline is used for benchmark construction, not for model input. This separation is central to the benchmark’s purpose: it tests whether an MLLM can recover 4D structure from visual evidence alone.

2. Task structure and input-output design

MLLM4D-Bench contains 6,000 multiple-choice video QA questions organized into six subtasks under three high-level categories (Yin et al., 28 Feb 2026). The task taxonomy decomposes 4D understanding into independent object motion, camera ego-motion, and object–camera dynamics.

Category	Subtask	Evaluated ability
Independent Object Motion	Object Absolute Distance	How far an object itself moved in physical 3D between two frames
Camera Ego-Motion	Camera Absolute Distance	How far the camera moved in 3D between two frames
Camera Ego-Motion	Camera Relative Direction	Direction of camera translation in the start-frame camera coordinate system
Object–Camera Dynamics	Object–Camera Absolute Distance	Metric distance between the camera at frame $i$ and the object at frame $j$
Object–Camera Dynamics	Object–Camera Relative Distance	Whether object and camera get closer or farther
Object–Camera Dynamics	Object–Camera Relative Direction	Directional change of the object relative to the camera

The benchmark distribution is balanced by subtask share: Object Absolute Distance contributes 16.7% of questions; Camera Absolute Distance and Camera Relative Direction together account for 33.4%; and the three Object–Camera Dynamics subtasks together account for 50.0% (Yin et al., 28 Feb 2026). This balance makes the benchmark diagnostic rather than dominated by a single reasoning mode.

Each instance consists of a short video sequence, sampled to at most 32 frames during evaluation, a natural-language question, and multiple-choice options, typically four choices labeled A–D (Yin et al., 28 Feb 2026). Questions reference one or more objects through textual descriptions and specify one or two frame indices, such as “between frame 2 and frame 28 of 31” (Yin et al., 28 Feb 2026). Options are either numeric distances in meters or discrete relational labels such as “closer,” “farther,” or directional categories (Yin et al., 28 Feb 2026).

The models are not given camera intrinsics or poses (Yin et al., 28 Feb 2026). A question can therefore require joint inference over appearance change, perspective shift, occlusion, and temporal correspondence. The benchmark also supports arbitrary frame-pair queries rather than only start/end comparisons, which gives it finer temporal resolution than summary-style video benchmarks (Yin et al., 28 Feb 2026).

Although the MLLM-4D system itself produces a structured Spatiotemporal Chain of Thought with <thinking> and <answer> tags, benchmark scoring uses only the final answer letter (Yin et al., 28 Feb 2026). This preserves compatibility with external MLLMs that do not implement the training-time reasoning format.

3. Automatic construction from stereo-derived 4D metadata

MLLM4D-Bench is built from Stereo4D, a large-scale Internet stereoscopic video dataset (Yin et al., 28 Feb 2026). The benchmark construction pipeline derives 4D spatial-temporal metadata for each clip, including left-rectified RGB frames $\{I_i\}_{i=1}^K$, per-frame camera poses $C_i = [R_i \mid t_i]$ obtained using COLMAP, metric 3D point clouds from stereo depth, instance-level segmentation and tracking with Grounded-SAM2, and fine-grained object descriptions $T_m$ generated by PixelRefer (Yin et al., 28 Feb 2026). The details summary further attributes motion-aware 3D estimation to SeaRAFT and notes a monocular alternative involving SpatialTrackerV2 and MoGe-2, although the main benchmark uses the stereo-based pipeline (Yin et al., 28 Feb 2026).

From these intermediate representations, the authors define a physics-based spatiotemporal relationship solver that computes ground-truth labels. The camera center at frame $i$ is defined as

$\text{Center}_i = -R_i^\top t_i,$

and the camera absolute distance between frames $i$ and $j$ is

$$d_{\text{cam\_abs\_dis}} = \lVert \text{Center}_j - \text{Center}_i \rVert_2.$$

Object motion is computed from object-specific 3D points $j$0, with object centroid

$j$1

and object absolute distance

$j$2

For object–camera queries, the benchmark uses quantities such as

$j$3

and classifies relative distance change by comparing $j$4 and $j$5 against a threshold $j$6 (Yin et al., 28 Feb 2026).

Question generation is then template-based. Numerical answers are converted into realistic multiple-choice options by perturbing the true value within 25–175%, and option order is shuffled to avoid positional bias (Yin et al., 28 Feb 2026). Entity extraction uses Gemini-2.5-Flash to identify active moving objects, Grounded-SAM2 provides temporally consistent masks, PixelRefer supplies descriptive referring expressions, and robust filtering removes low-quality geometry or inconsistent tracks (Yin et al., 28 Feb 2026). The result is a set of 6,000 high-quality, physics-consistent QA pairs (Yin et al., 28 Feb 2026).

This construction method has two significant implications. First, it enables scale without relying on exhaustive manual annotation. Second, it makes the benchmark sensitive to exact geometric reasoning rather than only coarse semantic recognition. A plausible implication is that benchmark errors can often be attributed to deficiencies in motion-conditioned spatial inference rather than ambiguity in annotation.

4. Evaluation protocol and relation to the MLLM-4D framework

MLLM4D-Bench is an evaluation-only benchmark; it is not used as a training source for MLLM-4D (Yin et al., 28 Feb 2026). Training instead uses MLLM4D-2M for supervised fine-tuning and MLLM4D-R1-30k for reinforcement fine-tuning (Yin et al., 28 Feb 2026). This separation matters because it allows the benchmark to function as a held-out diagnostic of post-training effectiveness.

The benchmark metric is answer accuracy,

$j$7

reported per subtask and as a macro average across the six subtasks (Yin et al., 28 Feb 2026). There is no direct benchmark-time metric for geometric consistency of the generated reasoning trace; evaluation is strictly based on the selected option.

Within the broader MLLM-4D framework, however, benchmark performance is interpreted alongside a post-training method based on Group Relative Policy Optimization. The training objective combines an answer-accuracy reward, a format reward enforcing structured <thinking> and <answer> outputs, and a Spatiotemporal reward that measures the closeness between predicted and ground-truth Object Center and Camera Center coordinates in the generated reasoning trace (Yin et al., 28 Feb 2026). The benchmark itself does not score these intermediate coordinates, but it is used to test whether that training signal improves final-answer correctness.

The associated Spatiotemporal Chain of Thought formalizes reasoning in five stages: temporal anchoring, initial 3D state parsing, temporal progression, final 3D state verification, and answer synthesis (Yin et al., 28 Feb 2026). In the model’s output format, the reasoning trace can include structured entries such as:

$j$8

For external models evaluated on MLLM4D-Bench, this structured reasoning is not mandatory for scoring (Yin et al., 28 Feb 2026). The benchmark is therefore simultaneously narrow in metric definition and broad in model applicability.

5. Empirical findings and benchmark sensitivity

The benchmark reports substantial gaps between general-purpose MLLMs and 4D-specialized post-trained models (Yin et al., 28 Feb 2026). Among proprietary baselines, GPT-4o reaches an average accuracy of 34.8, Gemini-2.5-Pro 37.4, and Gemini-2.5-Flash 35.2. Among open-source models without specialized 4D post-training, Qwen3-VL-32B reaches 41.3 average accuracy, while the strongest 3D-specialized baseline cited in the summary, VG-LLM based on Qwen2.5-VL-7B, reaches 56.3 (Yin et al., 28 Feb 2026).

MLLM-4D substantially improves on these baselines. The Qwen2.5-VL-7B version reaches 70.2 average accuracy, and the Qwen3-VL-8B version reaches 72.7, which the summary characterizes as state of the art on the benchmark (Yin et al., 28 Feb 2026). Reported subtask values in the summary include, for the Qwen3-VL-8B MLLM-4D model, 71.9 on Object Absolute Distance, 76.3 on Camera Absolute Distance, 74.3 on Camera Relative Direction, 69.2 on Object–Camera Absolute Distance, and 70.9 on Object–Camera Relative Distance (Yin et al., 28 Feb 2026). Relative to VG-LLM’s 56.3 average, this represents an improvement of approximately 16.4 points (Yin et al., 28 Feb 2026).

Ablation results show that the benchmark is sensitive to both data curation quality and reinforcement fine-tuning design. On Qwen3-VL-8B, the baseline model scores 35.3; supervised fine-tuning with alternative monocular data reaches 59.9; supervised fine-tuning with the stereo-based data curation pipeline reaches 70.1; GRPO without ST-reward reaches 70.5; and GRPO with ST-reward reaches 72.7 (Yin et al., 28 Feb 2026). These results indicate that MLLM4D-Bench is not merely separating stronger from weaker backbones; it is responsive to specifically 4D-oriented supervision.

This empirical profile also clarifies what the benchmark rewards. It favors models that can infer ego-motion, disentangle object motion from camera motion, and maintain frame-indexed geometric consistency. The authors’ qualitative analysis, as summarized in the details, reports that baseline models often ignore frame indices, rely on heuristic distance guesses, or misinterpret motion direction, whereas MLLM-4D’s structured reasoning more often reconstructs plausible 4D trajectories (Yin et al., 28 Feb 2026).

6. Position within the 4D benchmark landscape and limitations

MLLM4D-Bench occupies a specific niche within the rapidly emerging literature on 4D MLLM evaluation. Compared with "4D-Bench" (Zhu et al., 22 Mar 2025), which focuses on synthetic single-object 4D assets and evaluates 4D object QA and captioning, MLLM4D-Bench is more explicitly geometry-grounded and organized around metric object–camera relations. Compared with "Spatial4D-Bench" (Wang et al., 31 Dec 2025), which covers approximately 40,000 QA pairs across 18 tasks and six cognitive categories, MLLM4D-Bench is narrower in scope but more targeted toward frame-indexed distance and direction reasoning from RGB. Compared with B4DL (Choi et al., 7 Aug 2025), which addresses 4D LiDAR sequences in outdoor driving and includes 178,416 QA samples across six tasks, MLLM4D-Bench evaluates purely RGB-based recovery of 4D structure rather than direct reasoning over LiDAR geometry. This suggests that MLLM4D-Bench is best understood not as a general 4D benchmark, but as a specialized benchmark for metric visual 4D reasoning from monocular video.

The benchmark also differs from long-context and multi-image evaluations such as MileBench (Song et al., 2024) and MIBench (Liu et al., 2024). Those benchmarks emphasize many-image context handling, retrieval, OCR, and multi-image reasoning, whereas MLLM4D-Bench emphasizes physically meaningful geometric relations computed from stereo-derived 3D metadata. A plausible implication is that strong performance on generic multi-image reasoning does not by itself imply competence on MLLM4D-Bench; the latter requires a more rigid form of spatiotemporal inference.

The limitations noted in the MLLM-4D summary are correspondingly specific (Yin et al., 28 Feb 2026). Video length is capped at 32 frames during evaluation because of architecture constraints. Domain coverage depends on Stereo4D, which may bias scenes toward Internet-video distributions. Task coverage is concentrated on metric and relational geometry—distance, direction, and relative motion—and does not extend to higher-level physical dynamics, collisions, causality, or planning. The authors identify longer temporal contexts, broader domains such as indoor robotics and embodied navigation, and task extensions such as collision prediction or multi-object interactions as future directions (Yin et al., 28 Feb 2026).

In the broader history of 4D MLLM benchmarks, MLLM4D-Bench therefore represents a transition from qualitative video understanding toward quantitatively grounded 4D evaluation. Its distinctive contribution is to cast visual 4D reasoning as answerable, benchmarkable multiple-choice QA whose labels are derived from reconstructed geometry rather than manual narrative judgment. For research on spatiotemporal MLLMs, that design makes it a particularly useful instrument for measuring whether progress reflects better world modeling or merely stronger semantic video priors (Yin et al., 28 Feb 2026).