A Multi-Perspective Analysis of Memorization in Large Language Models (2405.11577v4)
Abstract: LLMs, trained on massive corpora with billions of parameters, show unprecedented performance in various fields. Though surprised by their excellent performances, researchers also noticed some special behaviors of those LLMs. One of those behaviors is memorization, in which LLMs can generate the same content used to train them. Though previous research has discussed memorization, the memorization of LLMs still lacks explanation, especially the cause of memorization and the dynamics of generating them. In this research, we comprehensively discussed memorization from various perspectives and extended the discussion scope to not only just the memorized content but also less and unmemorized content. Through various studies, we found that: (1) Through experiments, we revealed the relation of memorization between model size, continuation size, and context size. Further, we showed how unmemorized sentences transition to memorized sentences. (2) Through embedding analysis, we showed the distribution and decoding dynamics across model size in embedding space for sentences with different memorization scores. The n-gram statistics analysis presents d (3) An analysis over n-gram and entropy decoding dynamics discovered a boundary effect when the model starts to generate memorized sentences or unmemorized sentences. (4)We trained a Transformer model to predict the memorization of different models, showing that it is possible to predict memorizations by context.
- Exploring the limits of large scale pre-training.
- Guillaume Alain and Yoshua Bengio. 2018. Understanding intermediate layers using linear classifier probes.
- A review on language models as knowledge bases.
- Palm 2 technical report.
- Emergent and predictable memorization in large language models.
- Pythia: A suite for analyzing large language models across training and scaling.
- Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
- Quantifying memorization across neural language models.
- Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650. USENIX Association.
- Bert: Pre-training of deep bidirectional transformers for language understanding.
- Vitaly Feldman. 2020. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, STOC 2020, page 954–959, New York, NY, USA. Association for Computing Machinery.
- The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027.
- Sok: Memorization in general-purpose large language models.
- Preventing generation of verbatim memorization in language models gives a false sense of privacy. In Proceedings of the 16th International Natural Language Generation Conference, pages 28–53, Prague, Czechia. Association for Computational Linguistics.
- Scaling laws for neural language models.
- Can language models learn from explanations in context? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 537–563, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Are emergent abilities in large language models just in-context learning?
- Show your work: Scratchpads for intermediate computation with language models.
- Gpt-4 technical report.
- Language models as knowledge bases?
- Exploring the limits of transfer learning with a unified text-to-text transformer.
- NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Singapore. Association for Computational Linguistics.
- Are emergent abilities of large language models a mirage? In Thirty-seventh Conference on Neural Information Processing Systems.
- Localizing paragraph memorization in language models.
- Zhifan Sun and Antonio Valerio Miceli-Barone. 2024. Scaling behavior of machine translation with large language models under prompt injection attacks. In Proceedings of the First edition of the Workshop on the Scaling Behavior of Large Language Models (SCALE-LLM 2024), pages 9–23, St. Julian’s, Malta. Association for Computational Linguistics.
- Neural network studies. 1. comparison of overfitting and overtraining. Journal of Chemical Information and Modeling, 35(5):826–833.
- Pablo Villalobos. 2023. Scaling laws literature review. Accessed: 2024-04-29.
- Emergent abilities of large language models.
- A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, 4(2):100211.
- Understanding deep learning requires rethinking generalization.
- Bowen Chen (50 papers)
- Namgi Han (6 papers)
- Yusuke Miyao (34 papers)