4DThinker: Thinking with 4D Imagery for Dynamic Spatial Understanding

Abstract: Dynamic spatial reasoning from monocular video is essential for bridging visual intelligence and the physical world, yet remains challenging for vision-LLMs (VLMs). Prior approaches either verbalize spatial-temporal reasoning entirely as text, which is inherently verbose and imprecise for complex dynamics, or rely on external geometric modules that increase inference complexity without fostering intrinsic model capability. In this paper, we present 4DThinker, the first framework that enables VLMs to "think with 4D" through dynamic latent mental imagery, i.e., internally simulating how scenes evolve within the continuous hidden space. Specifically, we first introduce a scalable, annotation-free data generation pipeline that synthesizes 4D reasoning data from raw videos. We then propose Dynamic-Imagery Fine-Tuning (DIFT), which jointly supervises textual tokens and 4D latents to ground the model in dynamic visual semantics. Building on this, 4D Reinforcement Learning (4DRL) further tackles complex reasoning tasks via outcome-based rewards, restricting policy gradients to text tokens to ensure stable optimization. Extensive experiments across multiple dynamic spatial reasoning benchmarks demonstrate that 4DThinker consistently outperforms strong baselines and offers a new perspective toward 4D reasoning in VLMs. Our code is available at https://github.com/zhangquanchen/4DThinker.

• The paper introduces intrinsic 4D latent simulation by interleaving visual tokens with text, enabling accurate dynamic spatial reasoning.
• It employs a two-stage training pipeline combining Dynamic-Imagery Fine-Tuning and 4D Reinforcement Learning to ensure robust latent alignment.
• Empirical results on DSR-Bench and Dyn-Bench demonstrate significant performance gains over conventional models, achieving state-of-the-art results without external priors.

4DThinker: Intrinsic 4D Latent Simulation for Dynamic Spatial Reasoning

Motivation and Background

Dynamic spatial reasoning from monocular video presents a fundamental challenge for vision-LLMs (VLMs), especially in settings where accurate understanding of object trajectories, camera ego-motion, and their disentanglement is required for tasks in autonomous driving, robotics, and embodied AI. Mainstream VLMs are limited either by reliance on purely textual chain-of-thought (CoT) reasoning—which is inherently verbose and imprecise for geometrically complex or dynamic scenes—or through heavy use of external geometric modules, which make inference brittle and non-intrinsic. Prior latent reasoning approaches have been largely limited to static scenarios, lacking scalable extensions to the dynamic, annotation-scarce video domain.

Architecture and Latent Mental Imagery

4DThinker introduces a framework enabling VLMs to reason with dynamic latent visual tokens, interleaving them as "mental imagery" in a continuous manifold alongside textual reasoning traces. This design achieves direct simulation of scene evolution in the hidden space, rather than explicit reliance on verbalization or external structured modules.

Figure 1: Overview of 4DThinker’s inference architecture and two-stage training, showing the model’s interleaving of text and latent visual tokens during dynamic reasoning.

At inference, for each task instance, the model synthesizes latent representations of key frames, using them as explicit reasoning pivots that allow correct dynamic prediction—unlike standard CoT approaches (e.g., Gemini-3.1-Pro) that typically fail to grasp the correct geometry or temporal evolution.

Scalable Annotation-Free Data Generation

A central component is a three-stage, annotation-free data generation pipeline. Starting from large-scale videos, salient static and dynamic objects are algorithmically identified; landmark masks are produced using segmentation models and verified for temporal consistency by prompting high-level VLMs.

Figure 2: Three-stage 4D data pipeline—video preprocessing for landmarks, motion-centric QA construction, and imagery-based CoT synthesis for end-to-end dynamic scene understanding.

Motion-centric QA pairs are constructed for both camera and object dynamics, covering canonical axes such as movement direction, speed, distance change, orientation, and bounding-box grounded descriptions. Each sample is augmented with interleaved "mental imagery" placeholders, mapped to low-rank latent blocks over the model’s hidden space.

Training Paradigm: DIFT and 4DRL

4DThinker deploys a unified two-stage training pipeline:

1. Dynamic-Imagery Fine-Tuning (DIFT): Joint supervision of text and image-latent blocks; the loss combines standard cross-entropy on text tokens and a cosine-similarity term at latent positions, enforcing that the model’s forward-propagated hidden state aligns with the ground-truth visual embeddings at imagery steps.
2. 4D Reinforcement Learning (4DRL): Complex compound-motion tasks are tackled using outcome-based policy gradients, but only text tokens are included in the gradient computation, explicitly omitting latent blocks to prevent instability from the mismatch between discrete and continuous log-probabilities. The reward aggregates accuracy and structural compliance with the "think with 4D" format.

Dynamic Mental Simulation Mechanism

DIFT enables an autoregressive loop in which the hidden state at each latent position is recursively used as the next-step representation, creating an internal dynamic simulation of object and camera motion. This self-conditioning loop supports mental tracking of multi-phase trajectories and disentangling appearance dynamics (e.g., size changes due to posture vs. depth).

Qualitative instances demonstrate that 4DThinker can segment complex motion patterns, track holistic player movement trajectories, distinguish apparent object size variation from camera effects, and predict environmental transitions in dynamic driving videos.

Figure 3: On DSR-Bench, 4DThinker simulates a two-phase guinea pig trajectory, outperforming Gemini-3 and Qwen2.5-VL-3B, which lack explicit latent simulation.

Figure 4: 4DThinker tracks a diagonal full-court movement, capturing holistic spatial patterns beyond local heuristics.

Figure 5: The model accurately classifies stable panda size under posture change, decoupling camera and object effects.

Figure 6: Example of scene transition reasoning by tracking environmental shifts via 4D latents in driving videos.

Empirical Results and Analysis

On DSR-Bench (fine-grained, procedural 4D geometric measurement), 4DThinker achieves large improvements over baselines. For instance, Qwen2.5-VL-3B’s mean accuracy rises from 24.6 to 34.2 with full pipeline training, and Qwen3-VL-32B’s improves from 28.0 to 62.0. Notably, the best 4DThinker variant surpasses the previous SOTA DSR Suite-Model without leveraging external 3D priors. The improvements are even more pronounced on tasks probing absolute spatial attributes where conventional models are near-chance.

Holistic evaluation on Dyn-Bench, targeting semantic and causal dynamic understanding, reveals consistent gains across object-object, object-scene, and camera-object axes. Even with moderate-sized base models, 4DThinker outperforms high-profile proprietary VLMs, as well as dedicated spatial reasoning systems.

Ablation studies conclusively show that the cosine-similarity loss on latent blocks is essential—without it, the model's 4D latents degenerate into ungrounded representations. Increasing latent block size improves performance up to a limit (optimal at K=4), beyond which excessive latent tokens dilute context, decreasing accuracy.

Implications and Future Directions

4DThinker demonstrates that embedding geometric and temporal simulation intrinsically in VLMs via interleaved latent mental imagery enables robust, scalable dynamic spatial reasoning. This contradicts the prevailing view that external priors or heavyweight 3D/geometry modules are essential, showing instead that direct latent grounding is often more effective for real-world 4D reasoning.

Potential future extensions include:

• Applying the method to open-ended embodied planning, Egocentric RL, and other settings not limited to multiple-choice
• Integrating more robust geometric pre-processing modules to minimize noise in large-scale, annotation-free data pipelines
• Exploring further memory and recurrence mechanisms for even longer-horizon spatial decomposition

Theoretical implications include the importance of continuous latent supervision for aligning model internal dynamics with physical world evolution and the limits of language-only reasoning in spatial domains. Practically, the approach offers a scalable framework for equipping VLMs, regardless of base architecture, with advanced 4D simulation capacity without inference-time modularity increases.

Conclusion

4DThinker introduces a principled solution for dynamic spatial understanding in VLMs, leveraging interleaved latent simulation blocks—co-trained with textual reasoners in a scalable annotation-free paradigm—and demonstrates consistent, substantial performance gains over major baselines on both quantitative and holistic 4D benchmarks. The results establish intrinsic dynamic latent reasoning as a preferred path to bridging the gap between high-level language modeling and low-level spatial reasoning in complex visual environments (2605.05997).