2000 character limit reached
Can We Understand Plasticity Through Neural Collapse?
Published 3 Apr 2024 in cs.LG and cs.AI | (2404.02719v1)
Abstract: This paper explores the connection between two recently identified phenomena in deep learning: plasticity loss and neural collapse. We analyze their correlation in different scenarios, revealing a significant association during the initial training phase on the first task. Additionally, we introduce a regularization approach to mitigate neural collapse, demonstrating its effectiveness in alleviating plasticity loss in this specific setting.
- Critical learning periods in deep neural networks, 2019.
- On warm-starting neural network training, 2020.
- Loss of plasticity in deep continual learning, 2023.
- In search of robust measures of generalization. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 11723–11733. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/86d7c8a08b4aaa1bc7c599473f5dddda-Paper.pdf.
- Neural collapse under mse loss: Proximity to and dynamics on the central path, 2022.
- Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=ZkC8wKoLbQ7.
- Understanding plasticity in neural networks. ArXiv, abs/2303.01486, 2023. URL https://api.semanticscholar.org/CorpusID:257280044.
- The primacy bias in deep reinforcement learning. In International Conference on Machine Learning, 2022. URL https://api.semanticscholar.org/CorpusID:248811264.
- Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020. doi: 10.1073/pnas.2015509117. URL https://www.pnas.org/doi/abs/10.1073/pnas.2015509117.
- What Is Neural Plasticity?, pp. 1–15. Springer International Publishing, Cham, 2017. ISBN 978-3-319-62817-2. doi: 10.1007/978-3-319-62817-2˙1. URL https://doi.org/10.1007/978-3-319-62817-2_1.
- Invariant collaborative filtering to popularity distribution shift. Proceedings of the ACM Web Conference 2023, 2023. URL https://api.semanticscholar.org/CorpusID:256808660.
- A geometric analysis of neural collapse with unconstrained features, 2021.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.