Evaluating Spatial Understanding of Large Language Models (2310.14540v3)

Abstract: LLMs show remarkable capabilities across a variety of tasks. Despite the models only seeing text in training, several recent studies suggest that LLM representations implicitly capture aspects of the underlying grounded concepts. Here, we explore LLM representations of a particularly salient kind of grounded knowledge -- spatial relationships. We design natural-language navigation tasks and evaluate the ability of LLMs, in particular GPT-3.5-turbo, GPT-4, and Llama2 series models, to represent and reason about spatial structures. These tasks reveal substantial variability in LLM performance across different spatial structures, including square, hexagonal, and triangular grids, rings, and trees. In extensive error analysis, we find that LLMs' mistakes reflect both spatial and non-spatial factors. These findings suggest that LLMs appear to capture certain aspects of spatial structure implicitly, but room for improvement remains.

The paper "Evaluating Spatial Understanding of LLMs" explores assessing the spatial reasoning capabilities of LLMs, specifically including models like GPT-3.5-turbo, GPT-4, and various LLaMA2 series models. Although these models have been primarily trained on text, the paper investigates whether they implicitly acquire knowledge of spatial relationships through their language-based training.

Key Investigations

1. Evaluation Tasks:
   • The authors design natural language navigation tasks to evaluate the models' ability to understand spatial layouts involving square, hexagonal, and triangular grids, as well as more abstract structures like rings and trees.
   • Unlike simple textual reasoning, these tasks necessitate comprehension of spatial structures to accurately navigate between different locations defined within the prompts.
2. Exploration of Different Structures:
   • A logistic regression analysis was conducted to investigate the models' performance across various grid structures. Findings show that performance varies significantly, with models generally performing better on square grids compared to hexagonal or triangular grids. This suggests a certain bias or familiarity the models might have developed toward structures more commonly encountered in pre-training data.
3. Global vs. Local Map Presentation:
   • The paper differentiates between tasks where LLMs progressively build a local map based on sequential instructions and tasks where the entire spatial map is presented globally at the onset. Results indicate that the global setting is more challenging for most models, likely due to the increased cognitive load from processing the complete map simultaneously.
4. Impact of Data Feeding Order:
   • The order in which spatial data is fed into the models significantly affects performance. For instance, a row-by-row presentation was found to be more effective than random or snake-order presentations. Introducing explicit global coordinates also improved performance in certain snake-order presentations.
5. Inference of Global Map Size:
   • The capability of inferring map dimensions purely from sequential navigation actions was examined. The paper found that performance degraded with increasing side length and area, suggesting limitations in the LLMs’ ability to track larger and more complex navigation paths effectively.
6. Error Analysis:
   • Detailed error analyses were conducted to understand the nature of the mistakes made by LLMs, using both spatial and temporal distance metrics. It was noted that errors tend to cluster around topologically nearby nodes in square grids, supporting the notion of an implicit topology understanding. However, this spatial bias was less evident in hexagonal and triangular grids, where errors seemed driven by a tendency to regress to initial or frequently mentioned positions in the text.
7. Comparison with Human Performance:
   • Human participants were also tested on selected spatial tasks, and although human accuracy was far from perfect, it was notably higher than that of GPT-4, especially in complex or less regularly structured grids.

Conclusion

The paper concludes that while LLMs like GPT-4 exhibit some degree of implicit spatial understanding, this capability is uneven across different spatial structures and task settings. The results imply that the complexity of space representation in text presents significant challenges for current LLMs, despite their advanced language processing abilities. This research contributes to understanding the potential of LLMs to extend beyond traditional language tasks, shedding light on the implicit grounding of concepts within purely textual training environments.