MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning (2402.17231v3)
Abstract: Tool-augmented LLMs (TALMs) are known to enhance the skillset of LLMs, thereby, leading to their improved reasoning abilities across many tasks. While, TALMs have been successfully employed in different question-answering benchmarks, their efficacy on complex mathematical reasoning benchmarks, and the potential complementary benefits offered by tools for knowledge retrieval and mathematical equation solving are open research questions. In this work, we present MathSensei, a tool-augmented LLM for mathematical reasoning. We study the complementary benefits of the tools - knowledge retriever (Bing Web Search), program generator + executor (Python), and symbolic equation solver (Wolfram-Alpha API) through evaluations on mathematical reasoning datasets. We perform exhaustive ablations on MATH, a popular dataset for evaluating mathematical reasoning on diverse mathematical disciplines. We also conduct experiments involving well-known tool planners to study the impact of tool sequencing on the model performance. MathSensei achieves 13.5% better accuracy over gpt-3.5-turbo with Chain-of-Thought on the MATH dataset. We further observe that TALMs are not as effective for simpler math word problems (in GSM-8K), and the benefit increases as the complexity and required knowledge increases (progressively over AQuA, MMLU-Math, and higher level complex questions in MATH). The code and data are available at https://github.com/Debrup-61/MathSensei.
- Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
- Skills-in-context prompting: Unlocking compositionality in large language models.
- Reconcile: Round-table conference improves reasoning via consensus among diverse llms.
- Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588.
- Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
- Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
- A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level. Proceedings of the National Academy of Sciences, 119(32).
- Improving factuality and reasoning in language models through multiagent debate.
- Complexity-based prompting for multi-step reasoning.
- Pal: Program-aided language models.
- Learning to program with natural language.
- Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
- Measuring mathematical problem solving with the math dataset. NeurIPS.
- Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards reasoning in large language models: A survey.
- Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585–597.
- Encouraging divergent thinking in large language models through multi-agent debate.
- Program induction by rationale generation : Learning to solve and explain algebraic word problems.
- Chameleon: Plug-and-play compositional reasoning with large language models. arXiv preprint arXiv:2304.09842.
- A survey of deep learning for mathematical reasoning.
- Self-refine: Iterative refinement with self-feedback.
- A diverse corpus for evaluating and developing english math word problem solvers.
- OpenAI. 2023. Gpt-4 technical report.
- Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014.
- Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Toolformer: Language models can teach themselves to use tools.
- Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
- Challenging big-bench tasks and whether chain-of-thought can solve them.
- Fever: a large-scale dataset for fact extraction and verification.
- Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
- Olagpt: Empowering llms with human-like problem-solving abilities.
- Runzhe Yang and Karthik Narasimhan. 2023. The Socratic Method for Self-Discovery in Large Language Models. https://princeton-nlp.github.io/SocraticAI/.
- Hotpotqa: A dataset for diverse, explainable multi-hop question answering.
- Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601.
- React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.
- A comprehensive capability analysis of gpt-3 and gpt-3.5 series models.
- Progressive-hint prompting improves reasoning in large language models.
- Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification.
- Debrup Das (3 papers)
- Debopriyo Banerjee (8 papers)
- Somak Aditya (25 papers)
- Ashish Kulkarni (8 papers)