2000 character limit reached
Arithmetic Reasoning with LLM: Prolog Generation & Permutation (2405.17893v1)
Published 28 May 2024 in cs.CL and cs.AI
Abstract: Instructing LLMs to solve elementary school math problems has shown great success using Chain of Thought (CoT). However, the CoT approach relies on an LLM to generate a sequence of arithmetic calculations which can be prone to cascaded calculation errors. We hypothesize that an LLM should focus on extracting predicates and generating symbolic formulas from the math problem description so that the underlying calculation can be done via an external code interpreter. We investigate using LLM to generate Prolog programs to solve mathematical questions. Experimental results show that our Prolog-based arithmetic problem-solving outperforms CoT generation in the GSM8K benchmark across three distinct LLMs. In addition, given the insensitive ordering of predicates and symbolic formulas in Prolog, we propose to permute the ground truth predicates for more robust LLM training via data augmentation.
- Falcon-40B: an open large language model with state-of-the-art performance.
- Neural module networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 39–48.
- Graph of thoughts: Solving elaborate problems with large language models.
- Ivan Bratko. 2012. Prolog programming for Artificial Intelligence. Addison-Wesley.
- Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
- Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.
- Palm: Scaling language modeling with pathways.
- W. F. Clocksin and C. S. Mellish. 2003. Programming in Prolog. Springer-Verlag.
- Training verifiers to solve math word problems.
- Michael A. Covington. 2002. Natural language processing for Prolog programmers. Prentice Hall.
- Pal: Program-aided language models.
- Tora: A tool-integrated reasoning agent for mathematical problem solving.
- Neural module networks for reasoning over text. In International Conference on Learning Representations.
- Lora: Low-rank adaptation of large language models.
- Large language models can self-improve.
- Drew Hudson and Christopher D Manning. 2019. Learning by abstraction: The neural state machine. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
- Mistral 7b.
- Encouraging divergent thinking in large language models through multi-agent debate.
- Jieyi Long. 2023. Large language model guided tree-of-thought.
- Faithful chain-of-thought reasoning.
- Lila: A unified benchmark for mathematical reasoning.
- Learning a natural language interface with neural programmer. In International Conference on Learning Representations.
- Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning.
- Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning.
- Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446.
- Code llama: Open foundation models for code.
- Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761.
- Synthetic prompting: Generating chain-of-thought demonstrations for large language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 30706–30775. PMLR.
- Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.
- Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239.
- Llama 2: Open foundation and fine-tuned chat models.
- Emergent abilities of large language models.
- Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
- Neuro-symbolic integration brings causal and reliable reasoning proofs.
- Xlnet: Generalized autoregressive pretraining for language understanding.
- Tree of thoughts: Deliberate problem solving with large language models.
- Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825.
- Cumulative reasoning with large language models.
- Least-to-most prompting enables complex reasoning in large language models.
- Solving math word problems via cooperative reasoning induced language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics.
- Xiaocheng Yang (11 papers)
- Bingsen Chen (2 papers)
- Yik-Cheung Tam (8 papers)