SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models (2412.07755v2)

Abstract: Reasoning about motion and space is a fundamental cognitive capability that is required by multiple real-world applications. While many studies highlight that large multimodal LLMs (MLMs) struggle to reason about space, they only focus on static spatial relationships, and not dynamic awareness of motion and space, i.e., reasoning about the effect of egocentric and object motions on spatial relationships. Manually annotating such object and camera movements is expensive. Hence, we introduce SAT, a simulated spatial aptitude training dataset comprising both static and dynamic spatial reasoning across 175K question-answer (QA) pairs and 20K scenes. Complementing this, we also construct a small (150 image-QAs) yet challenging dynamic spatial test set using real-world images. Leveraging our SAT datasets and 6 existing static spatial benchmarks, we systematically investigate what improves both static and dynamic spatial awareness. Our results reveal that simulations are surprisingly effective at imparting spatial aptitude to MLMs that translate to real images. We show that perfect annotations in simulation are more effective than existing approaches of pseudo-annotating real images. For instance, SAT training improves a LLaVA-13B model by an average 11% and a LLaVA-Video-7B model by an average 8% on multiple spatial benchmarks, including our real-image dynamic test set and spatial reasoning on long videos -- even outperforming some large proprietary models. While reasoning over static relationships improves with synthetic training data, there is still considerable room for improvement for dynamic reasoning questions.

• The paper introduces Spatial Aptitude Training (SAT), a novel instruction-tuning method using a large synthetic dataset to significantly improve Multimodal Language Models' understanding of both static and dynamic spatial reasoning.
• The SAT dataset consists of 218,000 question-answer pairs derived from 22,000 physics-engine scenes, covering static spatial relations, object counting, and complex dynamic tasks like perspective-taking and predicting action consequences.
• Instruction tuning with SAT data substantially enhanced the baseline LLaVA model's performance on spatial reasoning benchmarks like CVBench, BLINK, and VSR, allowing it to match or outperform models like GPT-4V and Gemini-3.1.

The paper introduces Spatial Aptitude Training (SAT), a novel approach aimed at enhancing the spatial reasoning abilities of Multimodal LLMs (MLMs). This training focuses on both the static and dynamic aspects of spatial reasoning, recognizing that previous studies have primarily focused on static aspects. Static reasoning involves understanding fixed object positions, while dynamic reasoning requires engaging with tasks like perspective-taking and recognizing egocentric actions—capacities crucial for real-world deployment, such as in smart glasses and embodied AI.

Key Methodologies:

1. Dataset Creation: The SAT dataset consists of 218,000 question-answer pairs derived from 22,000 synthetically generated scenes using a photo-realistic physics engine. This setup allows for scalability and adaptability to new actions, scenes, and 3D assets.
2. Types of Spatial Questions:
   • Static Spatial Questions: These include relative spatial relations, depth queries, and counting of object instances.
   • Dynamic Spatial Questions: These are more complex and include:
     • Egocentric Movement: Judging movement direction from frames.
     • Object Movement: Identifying changes in an object's location.
     • Allocentric Perspective: Viewing the scene from another's perspective.
     • Goal Aiming: Determining directions to objects.
     • Action Consequence: Predicting changes from specific actions.
3. Training and Evaluation: The paper uses the 13B parameter LLaVA model as the baseline. The training process showed that instruction tuning with SAT data not only improved performance on dynamic tasks but also enhanced static spatial reasoning on existing benchmarks like CVBench, BLINK, and VSR.

Results:

• Instruction tuning with SAT data led to significant improvements in dynamic spatial reasoning capabilities. For instance, performance improvements were noted at 23% on CVBench, 8% on the BLINK benchmark, and 18% on VSR.
• The SAT-trained LLaVA model matched or outperformed larger proprietary MLMs such as GPT4-V and Gemini-3-1.0 in spatial reasoning tasks.
• The paper also revealed that even state-of-the-art MLMs struggle significantly with dynamic spatial questions, indicating the gap present in current models.

Conclusion and Future Directions:

The paper emphasizes the central role of spatial reasoning in cognitive processes. It shows that synthetic data, enhanced by a physics engine, could substantially close the MLM performance gap in spatial reasoning tasks. The research opens doors for more robust real-world applications of MLMs, particularly in areas where understanding dynamic environments is necessary.

Future work may focus on expanding the dataset, refining the types of dynamic spatial reasoning tasks, and improving the models' generalization abilities to perform even better on real-world tasks without necessitating human annotation. The paper also suggests potential enhancements in embodied AI, where improved spatial reasoning could significantly impact navigation and interaction tasks.