Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (2412.14171v1)

Abstract: Humans possess the visual-spatial intelligence to remember spaces from sequential visual observations. However, can Multimodal LLMs (MLLMs) trained on million-scale video datasets also ``think in space'' from videos? We present a novel video-based visual-spatial intelligence benchmark (VSI-Bench) of over 5,000 question-answer pairs, and find that MLLMs exhibit competitive - though subhuman - visual-spatial intelligence. We probe models to express how they think in space both linguistically and visually and find that while spatial reasoning capabilities remain the primary bottleneck for MLLMs to reach higher benchmark performance, local world models and spatial awareness do emerge within these models. Notably, prevailing linguistic reasoning techniques (e.g., chain-of-thought, self-consistency, tree-of-thoughts) fail to improve performance, whereas explicitly generating cognitive maps during question-answering enhances MLLMs' spatial distance ability.

Analyzing Visual-Spatial Intelligence in Multimodal LLMs

The paper "Thinking in Space: How Multimodal LLMs See, Remember, and Recall Spaces" presents a comprehensive paper on the spatial reasoning capabilities of Multimodal LLMs (MLLMs) using an innovative benchmark named VSI-Bench. This work is pivotal in bridging the gap between visual and linguistic understanding, especially focusing on how models comprehend and navigate through three-dimensional spaces depicted in videos.

Overview of VSI-Bench

VSI-Bench serves as a meticulously curated benchmark containing over 5,000 question-answer pairs drawn from 288 real-world indoor-scene videos. The benchmark evaluates various facets of visual-spatial intelligence through eight task categories, ranging from object counting and distance estimation to more complex spatial tasks like relative direction and route planning. Crucially, VSI-Bench differentiates itself by not just testing static image understanding but by requiring models to process dynamic visual inputs to solve spatial problems, further echoing real-world applications such as robotics and autonomous navigation.

Key Findings

The broad evaluation of 15 MLLMs across open-source and proprietary categories, including state-of-the-art models like Gemini and GPT-4o, yields several insights:

1. Human-Level Performance: Human evaluators outperformed MLLMs significantly, with an average accuracy of 79% compared to the model's best of 48.8%. This performance gap underlines the challenges MLLMs face in tasks requiring nuanced spatial reasoning.
2. Model Capabilities and Shortcomings:
   • MLLMs showed competitive performance on tasks involving quantitative estimation, such as object size and room size estimation.
   • A marked difficulty was observed in tasks demanding relational reasoning, such as relative direction, indicating a bottleneck in the egocentric-allocentric transformation and relational reasoning.
3. Prompting Techniques: Contrary to expectations from language tasks, prevailing linguistic prompting methods, including Chain-of-Thought and Tree-of-Thought, resulted in deteriorated spatial task performance, suggesting that these techniques might not extend well to visual-spatial reasoning.
4. Cognitive Mapping: When prompted to construct cognitive maps, MLLMs demonstrated reliable local spatial awareness, although global spatial representation remained weak. This discovery opens new avenues for enhancing spatial reasoning by imitating human-like cognitive mapping processes.

Implications and Future Directions

The implications of these findings are significant for the development of artificial intelligence systems, particularly those aimed at real-world navigation and interaction. The evident gap between MLLM performance and human capabilities in spatial reasoning tasks suggests that future research should focus on improving the inherent spatial reasoning strategies of these models. Potential directions include:

• Integrating Spatial Cognition: Incorporating mechanisms that mimic human spatial learning, such as using cognitive maps or spatial memory systems, might enhance MLLM performance.
• Task-Specific Training: Fine-tuning models on spatial reasoning tasks could yield better results than currently observed from general-purpose language or vision models.
• Novel Benchmarks: Developing benchmarks that not only test visual-spatial reasoning but also require interactive tasks could push the boundaries of MLLM capabilities.

By highlighting these observations, the paper underscores the current limitations and potential paths forward for advancing visual-spatial intelligence in multimodal AI systems. This research sets the foundation for more robust and context-aware AI models capable of more naturally interacting with the complexities of real-world environments.