Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Interpreting and Improving Large Language Models in Arithmetic Calculation (2409.01659v1)

Published 3 Sep 2024 in cs.CL

Abstract: LLMs have demonstrated remarkable potential across numerous applications and have shown an emergent ability to tackle complex reasoning tasks, such as mathematical computations. However, even for the simplest arithmetic calculations, the intrinsic mechanisms behind LLMs remain mysterious, making it challenging to ensure reliability. In this work, we delve into uncovering a specific mechanism by which LLMs execute calculations. Through comprehensive experiments, we find that LLMs frequently involve a small fraction (< 5%) of attention heads, which play a pivotal role in focusing on operands and operators during calculation processes. Subsequently, the information from these operands is processed through multi-layer perceptrons (MLPs), progressively leading to the final solution. These pivotal heads/MLPs, though identified on a specific dataset, exhibit transferability across different datasets and even distinct tasks. This insight prompted us to investigate the potential benefits of selectively fine-tuning these essential heads/MLPs to boost the LLMs' computational performance. We empirically find that such precise tuning can yield notable enhancements on mathematical prowess, without compromising the performance on non-mathematical tasks. Our work serves as a preliminary exploration into the arithmetic calculation abilities inherent in LLMs, laying a solid foundation to reveal more intricate mathematical tasks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems, 2022.
  2. Hidden progress in deep learning: Sgd learns parities near the computational limit. arXiv preprint arXiv:2207.08799, 2022.
  3. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp.  610–623, 2021.
  4. An interpretability illusion for BERT. CoRR, abs/2104.07143, 2021.
  5. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
  6. Palm: Scaling language modeling with pathways. CoRR, abs/2204.02311, 2022.
  7. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  8. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pp.  4171–4186. Association for Computational Linguistics, 2019.
  9. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https://transformer-circuits.pub/2021/framework/index.html.
  10. Causal analysis of syntactic agreement mechanisms in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp.  1828–1843, 2021.
  11. Mathematical capabilities of chatgpt. CoRR, abs/2301.13867, 2023.
  12. Causal abstractions of neural networks. In Advances in Neural Information Processing Systems, pp.  9574–9586, 2021.
  13. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In EMNLP, pp.  30–45, 2022.
  14. Dissecting recall of factual associations in auto-regressive language models. CoRR, abs/2304.14767, 2023.
  15. Localizing model behavior with path patching. CoRR, abs/2304.05969, 2023.
  16. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. CoRR, abs/2305.00586, 2023.
  17. Measuring massive multitask language understanding. ArXiv, abs/2009.03300, 2020.
  18. Natural language descriptions of deep visual features. In International Conference on Learning Representations, 2021.
  19. Learning to solve arithmetic word problems with verb categorization. In EMNLP, pp.  523–533. ACL, 2014.
  20. How well do computers solve math word problems? large-scale dataset construction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers, 2016.
  21. Large language models can self-improve. arXiv preprint arXiv:2210.11610, 2022.
  22. Mathprompter: Mathematical reasoning using large language models. In ACL, pp.  37–42, 2023.
  23. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
  24. Learning to reason deductively: Math word problem solving as complex relation extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp.  5944–5955, 2022.
  25. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, 2022.
  26. Parsing algebraic word problems into equations. Trans. Assoc. Comput. Linguistics, 3:585–597, 2015.
  27. Learning to automatically solve algebra word problems. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 1: Long Papers, pp.  271–281, 2014.
  28. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  5315–5333. Association for Computational Linguistics, July 2023.
  29. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. CoRR, abs/2307.09458, 2023.
  30. Post-hoc interpretability for neural nlp: A survey. ACM Computing Surveys, 55(8):1–42, 2022.
  31. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, 2022.
  32. Compositional explanations of neurons. In Advances in Neural Information Processing Systems, volume 33, pp.  17153–17163, 2020.
  33. A mechanistic interpretability analysis of grokking, 2022.
  34. Learning math reasoning from self-sampled correct and partially-correct solutions. In The Eleventh International Conference on Learning Representations, 2023.
  35. Investigating the limitations of the transformers with simple arithmetic tasks. CoRR, abs/2102.13019, 2021.
  36. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464):447–453, 2019.
  37. Zoom in: An introduction to circuits. In Distill, 2023.
  38. Do language models exhibit the same cognitive biases in problem solving as human learners? CoRR, abs/2401.18070, 2024.
  39. Task-specific skill localization in fine-tuned language models. In ICML, volume 202 of Proceedings of Machine Learning Research, pp.  27011–27033, 2023.
  40. Are NLP models really able to solve simple math word problems? In NAACL-HLT, pp.  2080–2094. Association for Computational Linguistics, 2021.
  41. Pearl, J. Direct and indirect effects. In UAI ’01: Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence, University of Washington, Seattle, Washington, USA, August 2-5, 2001, pp.  411–420, 2001.
  42. Pearl, J. Causality. Cambridge university press, 2009.
  43. Limitations of language models in arithmetic and symbolic induction. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp.  9285–9298. Association for Computational Linguistics, 2023.
  44. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  45. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp.  464–483, 2023.
  46. Mathematical discoveries from program search with large language models. Nat., 625(7995):468–475, 2024.
  47. Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206–215, 2019.
  48. Complex sequential question answering: Towards learning to converse over linked question answer pairs with a knowledge graph. In AAAI, pp.  705–713. AAAI Press, 2018.
  49. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations, 2019.
  50. Understanding arithmetic reasoning in language models using causal mediation analysis. CoRR, abs/2305.15054, 2023.
  51. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
  52. Stanford alpaca: An instruction-following llama model. 2023.
  53. Representing numbers in NLP: a survey and a vision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp.  644–656, 2021.
  54. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023a.
  55. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
  56. Solving math word problems with process- and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022.
  57. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems, volume 33, pp.  12388–12401, 2020.
  58. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In International Conference on Learning Representations, 2023a.
  59. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023b.
  60. Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pp.  845–854, 2017.
  61. Emergent abilities of large language models. ArXiv, abs/2206.07682, 2022a.
  62. Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022, 2022b.
  63. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022c.
  64. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp.  11–20. Association for Computational Linguistics, 2019.
  65. Interpretability at scale: Identifying causal mechanisms in alpaca. CoRR, abs/2305.08809, 2023.
  66. Language models are super mario: Absorbing abilities from homologous models as a free lunch. ArXiv, abs/2311.03099, 2023.
  67. STar: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, 2022.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now