Paraphrase and Solve: Exploring and Exploiting the Impact of Surface Form on Mathematical Reasoning in Large Language Models (2404.11500v1)
Abstract: This paper studies the relationship between the surface form of a mathematical problem and its solvability by LLMs. We find that subtle alterations in the surface form can significantly impact the answer distribution and the solve rate, exposing the LLM's lack of robustness and sensitivity to the surface form in reasoning through complex problems. To improve mathematical reasoning performance, we propose Self-Consistency-over-Paraphrases (SCoP), which diversifies reasoning paths from specific surface forms of the problem. We evaluate our approach on four mathematics reasoning benchmarks over three LLMs and show that SCoP improves mathematical reasoning performance over vanilla self-consistency, particularly for problems initially deemed unsolvable. Finally, we provide additional experiments and discussion regarding problem difficulty and surface forms, including cross-model difficulty agreement and paraphrasing transferability, and Variance of Variations (VOV) for LLM evaluation.
- Graph of thoughts: Solving elaborate problems with large language models.
- Rahul Bhagat and Eduard Hovy. 2013. What Is a Paraphrase? Computational Linguistics, 39(3):463–472.
- Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
- Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
- Demystifying prompts in language models via perplexity estimation.
- Measuring massive multitask language understanding. In International Conference on Learning Representations.
- Measuring mathematical problem solving with the math dataset. NeurIPS.
- Daniel Kahneman. 2011. Thinking, fast and slow. macmillan.
- Large language models are zero-shot reasoners.
- Holistic evaluation of language models.
- Let’s verify step by step. arXiv preprint arXiv:2305.20050.
- Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167, Vancouver, Canada. Association for Computational Linguistics.
- Chameleon: Plug-and-play compositional reasoning with large language models.
- Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- OpenAI. 2023. Gpt-4 technical report.
- Training language models to follow instructions with human feedback.
- Reasoning with language model prompting: A survey. arXiv preprint arXiv:2212.09597.
- Scaling language models: Methods, analysis and insights from training gopher.
- Autoprompt: Eliciting knowledge from language models with automatically generated prompts.
- Joint prompt optimization of stacked llms using variational inference.
- Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research.
- Paraphrasing interventions and problem-solving accuracy: Do generative procedures help english language learners with math difficulties? Learning Disabilities Research & Practice, 34(2):68–84.
- Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
- Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
- Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
- Tree of thoughts: Deliberate problem solving with large language models.
- The unreliability of explanations in few-shot prompting for textual reasoning. Advances in neural information processing systems, 35:30378–30392.
- Large language models are human-level prompt engineers.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.