Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models

Abstract: Vision-LLMs (VLM) excel at general understanding yet remain weak at dynamic spatial reasoning (DSR), i.e., reasoning about the evolvement of object geometry and relationship in 3D space over time, largely due to the scarcity of scalable 4D-aware training resources. To bridge this gap across aspects of dataset, benchmark and model, we introduce DSR Suite. First, we propose an automated pipeline that generates multiple-choice question-answer pairs from in-the-wild videos for DSR. By leveraging modern vision foundation models, the pipeline extracts rich geometric and motion information, including camera poses, local point clouds, object masks, orientations, and 3D trajectories. These geometric cues enable the construction of DSR-Train for learning and further human-refined DSR-Bench for evaluation. Compared with previous works, our data emphasize (i) in-the-wild video sources, (ii) object- and scene-level 3D requirements, (iii) viewpoint transformations, (iv) multi-object interactions, and (v) fine-grained, procedural answers. Beyond data, we propose a lightweight Geometry Selection Module (GSM) to seamlessly integrate geometric priors into VLMs, which condenses question semantics and extracts question-relevant knowledge from pretrained 4D reconstruction priors into a compact set of geometry tokens. This targeted extraction avoids overwhelming the model with irrelevant knowledge. Experiments show that integrating DSR-Train and GSM into Qwen2.5-VL-7B significantly enhances its dynamic spatial reasoning capability, while maintaining accuracy on general video understanding benchmarks.

• The paper introduces DSR Suite, featuring an automated QA generation pipeline that transforms in‐the-wild videos into large-scale 4D dynamic spatial reasoning data.
• The paper proposes a Geometry Selection Module with dual Q-Formers that efficiently integrates geometric priors, enhancing model performance on dynamic spatial tasks.
• The approach achieves 58.9% average accuracy on the DSR-Bench while maintaining general video understanding and scalability for real-world applications.

Dynamic Spatial Reasoning in Vision-LLMs: DSR Suite and Geometry-Selective Integration

Motivation and Problem Statement

Despite rapid advances in VLMs, existing models show significant deficiencies in Dynamic Spatial Reasoning (DSR)—the ability to infer object geometry and relational changes in unconstrained, realistic 4D environments (i.e., 3D space evolving over time). Current data regimes and benchmarks primarily emphasize static spatial relations, short time horizons, or domain-restricted scenarios, limiting progress towards robust 4D multimodal intelligence. The scarcity of large-scale 4D-aware datasets and benchmarks, coupled with inefficient geometric prior integration mechanisms, impedes DSR research and the deployment of embodied AI systems capable of operating in dynamic, real-world environments.

DSR Suite: Automated Dataset and Benchmark Generation

The paper proposes DSR Suite, a comprehensive framework for dataset creation, evaluation, and model enhancement targeting DSR:

• Automated QA Generation Pipeline: In-the-wild video data undergoes multi-stage processing leveraging LLM/VLM-driven curation, vision foundation models for geometric cue extraction, and QA synthesis mechanisms.

  Figure 1: DSR Suite's pipeline—video curation, geometric cue extraction, and QA data generation—producing multi-object, viewpoint-variant, and procedural questions from unconstrained videos.

• The pipeline ensures meaningful motion content by filtering large public corpora (Koala-36M) with LLM/VLM-based selection and duration constraints (20–120s). Object and scene-level annotations are extracted using models such as Grounded SAM2, Orient Anything, and $\pi^3$, producing camera poses, point clouds, object masks, agent orientations, and temporally consistent trajectories.
• Template-Based and Free-Form QA Synthesis: Question generation includes both rigid templates (e.g., assessing distance, speed, direction, orientation, and viewpoint transformation over time) and LLM-driven free-form questions, each grounded in 4D evidence. Answers emphasize fine-grained procedural descriptions of attribute evolution, reflecting object interaction and temporal context.
• DSR-Train and DSR-Bench: The pipeline yields a 10K video DSR-Train corpus for model learning and a human-verified DSR-Bench benchmark (575 videos, 1,484 questions) for systematic evaluation. DSR-Bench's distribution spans diverse activity classes and maintains balance across QA types.

  Figure 2: DSR-Bench statistical distributions by scene category, question type proportions, and content, evidencing thorough coverage and diversity.

• Benchmark comparisons demonstrate DSR-Bench's superiority in diversity (in-the-wild scenes), 3D requirement (strong, not superficial), support for viewpoint transformations, multi-object interaction evaluation, and answer granularity versus prior datasets.

Model Architecture: Geometry Selection Module (GSM)

Traditional methods fuse extensive 3D priors into VLMs via cross-attention or direct addition, engendering overfitting and degraded performance on generic tasks. To address this, the authors introduce a Geometry Selection Module (GSM):

• Dual Q-Former Mechanism: GSM employs stacked Q-Formers:
  • The first condenses variable-length question tokens into fixed-size, semantics-rich queries.
  • The second selectively attends to the geometric priors, extracting a compact set of geometry tokens maximally relevant to the question.
  • 

  • Figure 3: GSM architecture composed of two Q-Formers, filtering and condensing geometric priors for efficient fusion with vision and question tokens.

• This selection prevents overwhelming the core model with irrelevant or noisy geometric features, preserving general video understanding while enhancing DSR capability.
• The concatenated stream of vision tokens, geometry tokens, and question tokens is presented to the VLM, supporting efficient and robust multimodal reasoning.

Experimental Evaluation and Numerical Results

Extensive experiments utilize Qwen2.5-VL-7B (and additionally Qwen3-VL-8B-Instruct), incorporating GSM, trained on 50K instances. Results compare proprietary, video, and generalist VLMs, as well as spatial reasoning-specialized models.

• On DSR-Bench, the proposed approach achieves 58.9% average accuracy, vastly outperforming all baselines (random-guess-level performance among top LLMs and generalist VLMs). Even recent spatial VLMs, such as VG-LLM, underperform due to their static-scene bias.
• GSM integration maintains or improves accuracy on generic benchmarks (e.g., Video-MME), unlike naive feature fusion, which typically induces performance collapse outside the training task.
• Ablation studies show monotonic gains in DSR with increased GSM capacity (number of queries), albeit with diminishing returns and potential tradeoff in generic understanding.
• The benefit of explicit 3D priors (GSM vs. SFT) is substantial, particularly when incorporating diverse, high-coverage QA data; the highest accuracy is achieved with balanced mixtures of template and free-form questions.

  Figure 4: DSR-Bench accuracy as a function of training data size, confirming scaling properties and data efficiency with the presented approach.

Implications, Limitations, and Future Directions

This work establishes a formal pipeline and scalable suite for dynamic spatial reasoning, addressing a critical capability gap in current VLMs. By systematizing the rendering of 4D trajectory data into carefully structured QA pairs and proposing a lightweight selective fusion module for geometric priors, this framework moves toward robust 4D spatial intelligence. The approach is domain-agnostic and extensible to various VLM backbones and underlying geometric encoders.

Practically, these advances support the deployment of VLM-based agents in applications requiring continuous spatial awareness: robotics, AR/VR, navigation, autonomous driving, and embodied world modeling. The procedural, multi-view QA and fine-grained answer design highlight a pathway for evaluating and training generalizable spatial intelligence in diverse environments.

Future work can extend the pipeline to real-world robotic systems, investigate scaling with richer geometric priors, and explore continual learning paradigms for embodied lifelong DSR. Research may also focus on robust handling of metric uncertainties, cross-modal grounding for action policies, and integrating predictive dynamics for anticipatory reasoning.

Conclusion

The introduction of DSR Suite and the Geometry Selection Module delivers a significant methodological advance for dynamic spatial reasoning in VLMs. This combination of large-scale, 4D-grounded QA data, a rigorous human-refined benchmark, and selective prior integration yields state-of-the-art results with balanced generalization. These contributions drive the field toward unified 4D multimodal intelligence, supporting effective embodied interaction and comprehensive world modeling in realistic, dynamic scenarios.