Solving Math Word Problems by Combining Language Models With Symbolic Solvers (2304.09102v1)

Abstract: Automatically generating high-quality step-by-step solutions to math word problems has many applications in education. Recently, combining LLMs with external tools to perform complex reasoning and calculation has emerged as a promising direction for solving math word problems, but prior approaches such as Program-Aided LLM (PAL) are biased towards simple procedural problems and less effective for problems that require declarative reasoning. We propose an approach that combines an LLM that can incrementally formalize word problems as a set of variables and equations with an external symbolic solver that can solve the equations. Our approach achieves comparable accuracy to the original PAL on the GSM8K benchmark of math word problems and outperforms PAL by an absolute 20% on ALGEBRA, a new dataset of more challenging word problems extracted from Algebra textbooks. Our work highlights the benefits of using declarative and incremental representations when interfacing with an external tool for solving complex math word problems. Our data and prompts are publicly available at https://github.com/joyheyueya/declarative-math-word-problem.

Solving Math Word Problems by Combining LLMs With Symbolic Solvers

The paper authored by He-Yueya, Poesia, Wang, and Goodman presents a novel method that integrates LLMs with symbolic solvers to tackle mathematical word problems. This approach is particularly significant within the field of AI-driven educational tools, given the intrinsic challenge of deriving step-by-step solutions for math word problems, which require intricate reasoning and often complex arithmetic operations.

Key Contributions

The primary contribution of this paper is the development of a declarative formalization strategy, where LLMs incrementally represent word problems as sets of variables and equations. These representations are then solved using an external symbolic solver, such as SymPy in this paper. This method contrasts with earlier approaches, like the Program-Aided LLM (PAL), which emphasizes procedural reasoning—often proving less effective for problems demanding declarative reasoning.

Crucially, the proposed approach improves accuracy on challenging datasets. On the Algebra dataset, the method achieves an absolute 20% improvement over PAL. The Algebra dataset, an assembly of complex problems sourced from textbooks, serves to rigorously test the robustness of any word problem-solving strategy.

Methodology

The authors employ few-shot prompting to steer LLMs towards constructing incremental declarative solutions. This contrasts with chain-of-thought (CoT) prompting, which focuses on intermediate reasoning steps rather than mathematical declarations. The approach effectively leverages LLMs to generate formalized declarations while delegating computational tasks to external solvers, thereby mitigating arithmetic errors—a common pitfall with standalone LLM computations.

The paper systematically evaluates various prompting approaches, including both original eight-shot and optimized three-shot variants, in conjunction with added declarative principles. The inclusion of guiding principles notably enhances the LLM's performance in generating coherent mathematical formalisms.

Experimental Results

The experimental results showcase the superiority of combining LLMs with symbolic solvers. Detailed evaluations reveal that the declarative approach not only performs on par with PAL on standard benchmarks but distinctly excels on Algebra. The findings emphasize the efficacy of declarative reasoning facilitated by incremental formalization—a capability not always well-handled by procedural solutions.

Implications and Future Directions

The implications of this research are substantial for AI in educational contexts. It underscores the potential for AI systems, specifically those interfacing LLMs with symbolic solvers, to act as sophisticated instructional aids. Such systems could personalize learning experiences by offering step-by-step guidance tailored to a student's comprehension level.

The paper opens pathways for future exploration into hybrid models integrating LLMs with various specialized computational tools to enhance reasoning, learning, and problem-solving capabilities across diverse domains. Potential advancements in symbolic computation could further refine the accuracy and applicability of these systems, contributing to their proliferation in pedagogical settings.

In conclusion, this paper reflects an important stride towards better understanding the synergistic interface between linguistic comprehension and mathematical reasoning in AI systems. The proposed approach lays groundwork for further research and development, encouraging the ongoing refinement of AI models capable of addressing intricate mathematical tasks within educational landscapes.