Exploring and Improving the Spatial Reasoning Abilities of Large Language Models (2312.01054v1)

Abstract: LLMs represent formidable tools for sequence modeling, boasting an innate capacity for general pattern recognition. Nevertheless, their broader spatial reasoning capabilities, especially applied to numerical trajectory data, remain insufficiently explored. In this paper, we investigate the out-of-the-box performance of ChatGPT-3.5, ChatGPT-4 and Llama 2 7B models when confronted with 3D robotic trajectory data from the CALVIN baseline and associated tasks, including 2D directional and shape labeling. Additionally, we introduce a novel prefix-based prompting mechanism, which yields a 33% improvement on the 3D trajectory data and an increase of up to 10% on SpartQA tasks over zero-shot prompting (with gains for other prompting types as well). The experimentation with 3D trajectory data offers an intriguing glimpse into the manner in which LLMs engage with numerical and spatial information, thus laying a solid foundation for the identification of target areas for future enhancements.

Background and Objectives

LLMs have demonstrated impressive abilities in extrapolating patterns and serving as tools for cross-disciplinary applications. Despite these capabilities, their proficiency in more abstract areas, such as spatial reasoning, is less well-understood. This paper aims to assess the performance of LLMs, specifically ChatGPT versions 3.5 and 4, and Llama 2 7B, in tasks requiring spatial understanding. These tasks involve labeling 2D paths and identifying shapes, as well as labeling 3D robotic trajectories.

Approach and Methodology

To investigate these capabilities, the paper generates datasets for 2D path and shape labeling, using simple directional instructions and shapes like circles. For 3D trajectory labeling, it employs the CALVIN baseline, which contains data on robotic movements. The researchers evaluate the models using zero-shot prompting, In-context Learning (ICL), Chain-of-Thought (CoT) prompting, and propose a new method, Spatial Prefix-Prompting (SPP), which introduces a related spatial problem before the primary query. The paper examines not only how LLMs perform with simple spatial patterns but also the transfer of knowledge from simpler tasks to more complex ones.

Results and Findings

The experiments reveal that LLMs are competent at identifying simple 2D spatial patterns and yield acceptable few-shot identification of directions, especially with ChatGPT-4, which reaches perfect classification rates on short trajectories. However, the performance drops significantly when dealing with more complex 3D trajectories, with even the best models achieving only 80% accuracy after employing SPP on the "cleaned" CALVIN dataset where noise is reduced. CoT prompting showed inconsistent performance and did not always yield improvements, suggesting it may not be as effective for spatial tasks compared to language or mathematical reasoning.

Implications and Future Directions

The Spatial Prefix-Prompting method showed promise, often outperforming other techniques, which indicates that prompting models with simpler, related problems can facilitate better performance on complex spatial tasks. This paper lays the groundwork for future research into enhancing the spatial reasoning abilities of LLMs. Potential applications could extend to areas such as trend analysis or time-series data interpretation. Going forward, the research could benefit from a larger dataset and exploring additional spatial tasks including 3D point-cloud analysis and multi-variable trend forecasting.