2000 character limit reached
Get an A in Math: Progressive Rectification Prompting (2312.06867v1)
Published 11 Dec 2023 in cs.CL
Abstract: Chain-of-Thought (CoT) prompting methods have enabled LLMs to generate reasoning paths and solve math word problems (MWPs). However, they are sensitive to mistakes in the paths, as any mistake can result in an incorrect answer. We propose a novel method named Progressive Rectification Prompting (PRP) to improve average accuracy on eight MWP datasets from 77.3 to 90.5. Given an initial answer from CoT, PRP iterates a verify-then-rectify process to progressively identify incorrect answers and rectify the reasoning paths. With the most likely correct answer, the LLM predicts a masked numerical value in the question; if the prediction does not match the masked value, the answer is likely incorrect. Then the LLM is prompted to re-generate the reasoning path hinted with a set of incorrect answers to prevent itself from repeating previous mistakes. PRP achieves the best performance compared against the CoT methods. Our implementation is made publicly available at https://wzy6642.github.io/prp.github.io/.
- Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. arXiv preprint arXiv:2211.12588.
- Training Verifiers to Solve Math Word Problems. CoRR, abs/2110.14168.
- Evans, J. S. 2003. In two minds: dual-process accounts of reasoning. Trends in Cognitive Sciences, 7(10): 454–459.
- Gall, M. D.; et al. 1990. Tools for Learning: A Guide to Teaching Study Skills. ERIC.
- Learning to Solve Arithmetic Word Problems with Verb Categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 523–533. Doha, Qatar: Association for Computational Linguistics.
- Large Language Models are Zero-Shot Reasoners. In Koyejo, S.; Mohamed, S.; Agarwal, A.; Belgrave, D.; Cho, K.; and Oh, A., eds., Advances in Neural Information Processing Systems, volume 35, 22199–22213. Curran Associates, Inc.
- Parsing Algebraic Word Problems into Equations. Transactions of the Association for Computational Linguistics, 3: 585–597.
- Learning to Automatically Solve Algebra Word Problems. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 271–281. Baltimore, Maryland: Association for Computational Linguistics.
- MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solving. In Findings of the Association for Computational Linguistics: NAACL 2022, 997–1009. Seattle, United States: Association for Computational Linguistics.
- Generalizing Math Word Problem Solvers via Solution Diversification. In AAAI.
- Learning To Use Formulas To Solve Simple Arithmetic Problems. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2144–2153. Berlin, Germany: Association for Computational Linguistics.
- Are NLP Models really able to Solve Simple Math Word Problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2080–2094. Online: Association for Computational Linguistics.
- Solving General Arithmetic Word Problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 1743–1752. Lisbon, Portugal: Association for Computational Linguistics.
- Reasoning about Quantities in Natural Language. Transactions of the Association for Computational Linguistics, 3: 1–13.
- Generate & Rank: A Multi-task Framework for Math Word Problems. In Findings of the Association for Computational Linguistics: EMNLP 2021, 2269–2279. Punta Cana, Dominican Republic: Association for Computational Linguistics.
- Large Language Models Can Be Easily Distracted by Irrelevant Context. In Proceedings of the 40th International Conference on Machine Learning.
- Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2609–2634. Toronto, Canada: Association for Computational Linguistics.
- Template-Based Math Word Problem Solvers with Recursive Neural Networks. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’19/IAAI’19/EAAI’19. AAAI Press. ISBN 978-1-57735-809-1.
- Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations.
- Deep Neural Solver for Math Word Problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 845–854. Copenhagen, Denmark: Association for Computational Linguistics.
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Koyejo, S.; Mohamed, S.; Agarwal, A.; Belgrave, D.; Cho, K.; and Oh, A., eds., Advances in Neural Information Processing Systems, volume 35, 24824–24837. Curran Associates, Inc.
- Large Language Models are reasoners with Self-Verification. arXiv preprint arXiv:2212.09561.
- IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 8276–8288. Singapore: Association for Computational Linguistics.
- Auto-Instruct: Automatic Instruction Generation and Ranking for Black-Box Language Models. In Bouamor, H.; Pino, J.; and Bali, K., eds., Findings of the Association for Computational Linguistics: EMNLP 2023, 9850–9867. Singapore: Association for Computational Linguistics.
- Automatic Chain of Thought Prompting in Large Language Models. In The Eleventh International Conference on Learning Representations (ICLR 2023).
- Progressive-Hint Prompting Improves Reasoning in Large Language Models. arXiv:2304.09797.
- Learn to Solve Algebra Word Problems Using Quadratic Programming. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 817–822. Lisbon, Portugal: Association for Computational Linguistics.