Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Cracking Factual Knowledge: A Comprehensive Analysis of Degenerate Knowledge Neurons in Large Language Models (2402.13731v2)

Published 21 Feb 2024 in cs.CL and cs.AI

Abstract: LLMs store extensive factual knowledge, but the underlying mechanisms remain unclear. Previous research suggests that factual knowledge is stored within multi-layer perceptron weights, and some storage units exhibit degeneracy, referred to as Degenerate Knowledge Neurons (DKNs). Despite the novelty and unique properties of this concept, it has not been rigorously defined or systematically studied. We first consider the connection weight patterns of MLP neurons and define DKNs from both structural and functional aspects. Based on this, we introduce the Neurological Topology Clustering method, which allows the formation of DKNs in any numbers and structures, leading to a more accurate DKN acquisition. Furthermore, inspired by cognitive science, we explore the relationship between DKNs and the robustness, evolvability, and complexity of LLMs. Our execution of 34 experiments under 6 settings demonstrates the connection between DKNs and these three properties. The code will be available soon.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. The k-means algorithm: A comprehensive survey and performance evaluation. Electronics, 9(8):1295.
  2. Zeyuan Allen-Zhu and Yuanzhi Li. 2023. Physics of language models: Part 3.2, knowledge manipulation. ArXiv preprint, abs/2309.14402.
  3. Serguei Barannikov. 1994. The framed morse complex and its invariants. Advances in Soviet Mathematics, 21:93–116.
  4. Gunnar Carlsson. 2009. Topology and data. Bulletin of the American Mathematical Society, 46(2):255–308.
  5. What input errors do you experience? typing and pointing errors of mobile web users. International journal of human-computer studies, 68(3):138–157.
  6. Journey to the center of the knowledge neurons: Discoveries of language-independent knowledge neurons and degenerate knowledge neurons. In Proceedings of the AAAI Conference on Artificial Intelligence.
  7. Stability of persistence diagrams. In Proceedings of the twenty-first annual symposium on Computational geometry, pages 263–271.
  8. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493–8502, Dublin, Ireland. Association for Computational Linguistics.
  9. Simba: An efficient tool for approximating rips-filtration persistence via sim plicial ba tch collapse. Journal of Experimental Algorithmics (JEA), 24:1–16.
  10. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257–273.
  11. Gerald M Edelman and Joseph A Gally. 2001. Degeneracy and complexity in biological systems. Proceedings of the National Academy of Sciences, 98(24):13763–13768.
  12. Persistent homology-a survey. Contemporary mathematics, 453(26):257–282.
  13. Herbert Edelsbrunner and John L Harer. 2022. Computational topology: an introduction. American Mathematical Society.
  14. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pages 226–231.
  15. A model-based approach for robustness testing. In Testing of Communicating Systems: 17th IFIP TC6/WG 6.1 International Conference, TestCom 2005, Montreal, Canada, May 31-June, 2005. Proceedings 17, pages 333–348. Springer.
  16. Dissecting recall of factual associations in auto-regressive language models.
  17. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  18. A survey on automated fact-checking. Transactions of the Association for Computational Linguistics, 10:178–206.
  19. Model complexity of deep learning: A survey.
  20. Multilingual LAMA: Investigating knowledge in multilingual pretrained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3250–3258, Online. Association for Computational Linguistics.
  21. Michael Kerber and Raghvendra Sharathkumar. 2013. Approximate čech complex in low and high dimensions. In International Symposium on Algorithms and Computation, pages 666–676. Springer.
  22. Marc Kirschner and John Gerhart. 1998. Evolvability. Proceedings of the National Academy of Sciences, 95(15):8420–8427.
  23. A rigorous study of integrated gradients method and extensions to internal neuron attributions. In International Conference on Machine Learning, pages 14485–14508. PMLR.
  24. Mourad Mars. 2022. From word embeddings to pre-trained language models: A state-of-the-art walkthrough. Applied Sciences, 12(17):8805.
  25. Paul H Mason. 2015. Degeneracy: Demystifying and destigmatizing a core concept in systems biology. Complexity, 20(3):12–21.
  26. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems.
  27. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations.
  28. Fionn Murtagh and Pedro Contreras. 2012. Algorithms for hierarchical clustering: an overview. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 2(1):86–97.
  29. Gpt-4 technical report.
  30. A roadmap for the computation of persistent homology. EPJ Data Science, 6:1–38.
  31. How context affects language models’ factual predictions. In Automated Knowledge Base Construction.
  32. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.
  33. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  34. Llama 2: Open foundation and fine-tuned chat models.
  35. On the use of size functions for shape analysis. Biological cybernetics, 70(2):99–107.
  36. James Whitacre and Axel Bender. 2010. Degeneracy: a design principle for achieving robustness and evolvability. Journal of theoretical biology, 263(1):143–153.
  37. James M Whitacre. 2010. Degeneracy: a link between evolvability, robustness and complexity in biological systems. Theoretical Biology and Medical Modelling, 7:1–17.
  38. Zeyuan Allen Zhu and Yuanzhi Li. 2023. Physics of language models: Part 3.1, knowledge storage and extraction. ArXiv preprint, abs/2309.14316.
  39. Afra Zomorodian and Gunnar Carlsson. 2004. Computing persistent homology. In Proceedings of the twentieth annual symposium on Computational geometry, pages 347–356.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Yuheng Chen (16 papers)
  2. Pengfei Cao (39 papers)
  3. Yubo Chen (58 papers)
  4. Yining Wang (91 papers)
  5. Shengping Liu (21 papers)
  6. Kang Liu (207 papers)
  7. Jun Zhao (469 papers)
Citations (1)
View on Semantic Scholar