2000 character limit reached
Quantifying In-Context Reasoning Effects and Memorization Effects in LLMs (2405.11880v1)
Published 20 May 2024 in cs.LG, cs.AI, cs.CL, and cs.CV
Abstract: In this study, we propose an axiomatic system to define and quantify the precise memorization and in-context reasoning effects used by the LLM for language generation. These effects are formulated as non-linear interactions between tokens/words encoded by the LLM. Specifically, the axiomatic system enables us to categorize the memorization effects into foundational memorization effects and chaotic memorization effects, and further classify in-context reasoning effects into enhanced inference patterns, eliminated inference patterns, and reversed inference patterns. Besides, the decomposed effects satisfy the sparsity property and the universal matching property, which mathematically guarantee that the LLM's confidence score can be faithfully decomposed into the memorization effects and in-context reasoning effects. Experiments show that the clear disentanglement of memorization effects and in-context reasoning effects enables a straightforward examination of detailed inference patterns encoded by LLMs.
- Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319, 2019.
- Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
- The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium (USENIX security 19), pages 267–284, 2019.
- Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650, 2021.
- Quantifying memorization across neural language models. In International Conference on Learning Representations, 2022.
- Defining and extracting generalizable interaction primitives from dnns. In International Conference on Learning Representations, 2024.
- Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
- What neural networks memorize and why: Discovering the long tail via influence estimation. In Advances in Neural Information Processing Systems, volume 33, pages 2881–2891, 2020.
- Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361, 2021.
- Defining and quantifying and-or interactions for faithful and concise explanation of dnns. arXiv preprint arXiv:2304.13312, 2023a.
- Does a neural network really encode symbolic concepts? In International Conference on Machine Learning, pages 20452–20469. PMLR, 2023b.
- Logiqa: a challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3622–3628, 2021.
- How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven. Transactions of the Association for Computational Linguistics, 11:652–670, 2023.
- Adversarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901, 2020.
- Defining and quantifying the emergence of sparse concepts in dnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20280–20289, 2023a.
- Can we faithfully represent absence states to compute shapley values on a dnn? In International Conference on Learning Representations, 2023b.
- Where we have arrived in proving the emergence of sparse symbolic concepts in ai models. In International Conference on Learning Representations, 2024.
- Can the inference logic of large language models be disentangled into symbolic concepts? arXiv preprint arXiv:2304.01083, 2023.
- Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of NAACL-HLT, pages 4149–4158, 2019.
- The language interpretability tool: Extensible, interactive visualizations and analysis for nlp models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 107–118, 2020.
- Diagnosing the first-order logical reasoning ability through logicnli. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3738–3747, 2021.
- Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274–38290, 2022.
- Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
- Emergent abilities of large language models. Transactions on Machine Learning Research, 2022a.
- Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022b.
- Reclor: A reading comprehension dataset requiring logical reasoning. In International Conference on Learning Representations, 2019.
- Counterfactual memorization in neural language models. In Advances in Neural Information Processing Systems, volume 36, 2024.
- Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
- Explaining how a neural network play the go game and let people learn. arXiv preprint arXiv:2310.09838, 2023.
- Explaining generalization power of a dnn using interactive concepts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17105–17113, 2024.