Case-Based or Rule-Based: How Do Transformers Do the Math? (2402.17709v2)
Abstract: Despite the impressive performance in a variety of complex tasks, modern LLMs still have trouble dealing with some math problems that are simple and intuitive for humans, such as addition. While we can easily learn basic rules of addition and apply them to new problems of any length, LLMs struggle to do the same. Instead, they may rely on similar cases seen in the training corpus for help. We define these two different reasoning mechanisms as "rule-based reasoning" and "case-based reasoning". Since rule-based reasoning is essential for acquiring systematic generalization ability, we aim to explore exactly whether transformers use rule-based or case-based reasoning for math problems. Through carefully designed intervention experiments on five math tasks, we confirm that transformers are performing case-based reasoning, no matter whether scratchpad is used, which aligns with the previous observations that transformers use subgraph matching/shortcut learning to reason. To mitigate such problems, we propose a Rule-Following Fine-Tuning (RFFT) technique to teach transformers to perform rule-based reasoning. Specifically, we provide explicit rules in the input and then instruct transformers to recite and follow the rules step by step. Through RFFT, we successfully enable LLMs fine-tuned on 1-5 digit addition to generalize to up to 12-digit addition with over 95% accuracy, which is over 40% higher than scratchpad. The significant improvement demonstrates that teaching LLMs to use rules explicitly helps them learn rule-based reasoning and generalize better in length.
- Generalization on the unseen, logic reasoning and degree curriculum, 2023.
- What learning algorithm is in-context learning? investigations with linear models, 2023.
- Exploring length generalization in large language models, 2022.
- Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models, 2023.
- Language models are few-shot learners, 2020.
- Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
- Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023.
- Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers, 2023.
- Faith and fate: Limits of transformers on compositionality, 2023.
- Towards revealing the mystery behind chain of thought: A theoretical perspective, 2023.
- What can transformers learn in-context? a case study of simple function classes, 2023.
- Towards a mechanistic interpretation of multi-step reasoning capabilities of language models, 2023.
- The impact of positional encoding on length generalization in transformers, 2023.
- Decomposed prompting: A modular approach for solving complex tasks, 2023.
- Large language models are zero-shot reasoners, 2023.
- Humans in humans out: On gpt converging toward common sense in both success and failure, 2023.
- Teaching arithmetic to small transformers, 2023.
- Loogle: Can long-context language models understand long contexts? arXiv preprint arXiv:2311.04939, 2023.
- Towards understanding grokking: An effective theory of representation learning, 2022.
- Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251, 2023.
- Progress measures for grokking via mechanistic interpretability, 2023.
- Show your work: Scratchpads for intermediate computation with language models, 2021.
- OpenAI. Introducing chatgpt, 2022. https://openai.com/blog/chatgpt.
- OpenAI. Gpt-4 technical report, 2023.
- Grokking: Generalization beyond overfitting on small algorithmic datasets, 2022.
- Limitations of language models in arithmetic and symbolic induction, 2022.
- Is chatgpt a general-purpose natural language processing task solver?, 2023.
- Language models are unsupervised multitask learners. 2019. URL https://api.semanticscholar.org/CorpusID:160025533.
- Positional description matters for transformers arithmetic, 2023.
- Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
- Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022.
- Llama: Open and efficient foundation language models, 2023a.
- Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
- Transformers learn in-context by gradient descent, 2023.
- Emergent abilities of large language models, 2022.
- Chain-of-thought prompting elicits reasoning in large language models, 2023.
- Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks, 2023.
- It ain’t that bad: Understanding the mysterious performance drop in ood generalization for generative transformer models, 2023.
- Explaining the complex task reasoning of large language models with template-content structure, 2023.
- Are transformers universal approximators of sequence-to-sequence functions?, 2020.
- Counterfactual memorization in neural language models, 2023.
- The clock and the pizza: Two stories in mechanistic explanation of neural networks, 2023.
- Least-to-most prompting enables complex reasoning in large language models, 2023a.
- Teaching algorithmic reasoning via in-context learning, 2022.
- What algorithms can transformers learn? a study in length generalization, 2023b.
- Transformers can achieve length generalization but not robustly, 2024.
- Large language models can learn rules, 2023.
- Yi Hu (129 papers)
- Xiaojuan Tang (5 papers)
- Haotong Yang (11 papers)
- Muhan Zhang (89 papers)