Beyond Lines and Circles: Unveiling the Geometric Reasoning Gap in Large Language Models (2402.03877v3)

Abstract: LLMs demonstrate ever-increasing abilities in mathematical and algorithmic tasks, yet their geometric reasoning skills are underexplored. We investigate LLMs' abilities in constructive geometric problem-solving one of the most fundamental steps in the development of human mathematical reasoning. Our work reveals notable challenges that the state-of-the-art LLMs face in this domain despite many successes in similar areas. LLMs exhibit biases in target variable selection and struggle with 2D spatial relationships, often misrepresenting and hallucinating objects and their placements. To this end, we introduce a framework that formulates an LLMs-based multi-agents system that enhances their existing reasoning potential by conducting an internal dialogue. This work underscores LLMs' current limitations in geometric reasoning and improves geometric reasoning capabilities through self-correction, collaboration, and diverse role specializations.

The paper "Beyond Lines and Circles: Unveiling the Geometric Reasoning Gap in LLMs" presents a comprehensive paper on the limitations of LLMs, particularly in the context of constructive geometric reasoning. While LLMs have shown remarkable performances in various mathematical and algorithmic domains, their capabilities in geometric reasoning remain underexplored and problematic.

Key Findings:

1. Geometric Reasoning Challenges:
   • The paper identifies significant challenges for LLMs in solving constructive geometry problems, a foundational aspect of human mathematical reasoning.
   • LLMs demonstrate biases in selecting target variables and often fail to maintain the correct 2D spatial reasoning, leading to errors such as misrepresentation and hallucinations of objects.
2. Framework Introduction:
   • The authors propose a multi-agent framework designed to enhance LLMs' reasoning abilities by engaging them in self-corrective dialogue, thus addressing some of their inherent limitations.
   • This framework includes four LLM-based agents, each with specialized roles (reasoners, solvers, tool users), engaging in a collaborative discussion to solve geometric construction tasks.
3. Numerical Limitations and Solutions:
   • Through analysis, it is evident that LLMs specialized in mathematical domains do not necessarily perform well in constructive geometry.
   • The paper highlights a novel simulacra-based system that combines tool usage, instruction following, and geometric problem-solving. This approach improves upon separate agents' strengths for more effective solutions.
   • Techniques like variable renaming and adaptive prompt selection are introduced, leading to improved problem-solving by mitigating biases and focusing attention on relevant information.
4. Prompt Engineering and Multi-Agent Systems:
   • The paper explores prompt engineering strategies, such as adaptive-shot mechanisms, which leverage past examples to guide current problem-solving processes, showing improvements over static methods.
   • Multi-agent configurations, where agents with different domain specializations communicate, illustrate a significant advancement in solution accuracy over single-agent models.
5. Experimental Validation:
   • Extensive experimentation on the Euclidea dataset—a set of progressively challenging geometric problems—demonstrates the proposed framework's effectiveness.
   • The use of simulacra is shown to surpass traditional non-collaborative methods in solving geometric problems, highlighting its potential applicability in enhancing LLM capabilities.
   • The incorporation of a visual relations prompt significantly aids in improving the spatial reasoning of text-only LLMs.

Conclusion:

The paper concludes by acknowledging the complexities of applying LLMs to constructive geometry, noting that while profound improvements have been achieved through the multi-agent and dialogue-based approach, further development towards genuinely intelligent and spatially aware LLMs is required. The research lays foundational work that opens new avenues for tackling geometric reasoning through interaction and collaborative problem-solving within LLM frameworks.