Disentangling Linear Mode-Connectivity (2312.09832v1)
Abstract: Linear mode-connectivity (LMC) (or lack thereof) is one of the intriguing characteristics of neural network loss landscapes. While empirically well established, it unfortunately still lacks a proper theoretical understanding. Even worse, although empirical data points are abound, a systematic study of when networks exhibit LMC is largely missing in the literature. In this work we aim to close this gap. We explore how LMC is affected by three factors: (1) architecture (sparsity, weight-sharing), (2) training strategy (optimization setup) as well as (3) the underlying dataset. We place particular emphasis on minimal but non-trivial settings, removing as much unnecessary complexity as possible. We believe that our insights can guide future theoretical works on uncovering the inner workings of LMC.
- Git Re-Basin: Merging Models modulo Permutation Symmetries, March 2023. URL http://arxiv.org/abs/2209.04836.
- Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling. In Proceedings of the 38th International Conference on Machine Learning, pages 769–779. PMLR, July 2021. URL https://proceedings.mlr.press/v139/benton21a.html.
- Random initialisations performing above chance and how to find them, November 2022. URL http://arxiv.org/abs/2209.07509.
- Towards Democratizing Joint-Embedding Self-Supervised Learning, March 2023. URL http://arxiv.org/abs/2303.01986.
- On the Relationship between Self-Attention and Convolutional Layers. 2020. URL https://arxiv.org/abs/1911.03584.
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021. URL http://arxiv.org/abs/2010.11929.
- Essentially No Barriers in Neural Network Energy Landscape. In Proceedings of the 35th International Conference on Machine Learning, pages 1309–1318. PMLR, July 2018. URL https://proceedings.mlr.press/v80/draxler18a.html.
- The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks, July 2022. URL https://arxiv.org/abs/2110.06296.
- Linear Mode Connectivity and the Lottery Ticket Hypothesis. In Proceedings of the 37th International Conference on Machine Learning, pages 3259–3269. PMLR, November 2020. URL https://proceedings.mlr.press/v119/frankle20a.html.
- Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper/2018/hash/be3087e74e9100d4bc4c6268cdbe8456-Abstract.html.
- Linear Connectivity Reveals Generalization Strategies, January 2023. URL http://arxiv.org/abs/2205.12411.
- FFCV: Accelerating Training by Removing Data Bottlenecks, June 2023. URL http://arxiv.org/abs/2306.12517.
- Mechanistic Lens on Mode Connectivity. 2023. URL https://arxiv.org/abs/2211.08422.
- Behnam Neyshabur. Towards Learning Convolutions from Scratch, July 2020. URL http://arxiv.org/abs/2007.13657.
- What is being transferred in transfer learning? arXiv:2008.11687 [cs, stat], January 2021. URL http://arxiv.org/abs/2008.11687.
- Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances. In Proceedings of the 38th International Conference on Machine Learning, pages 9722–9732. PMLR, July 2021. URL https://proceedings.mlr.press/v139/simsek21a.html.
- What can linear interpolation of neural network loss landscapes tell us?, February 2022. URL http://arxiv.org/abs/2106.16004.
- Symmetries, Flat Minima And The Conserved Quantities Of Gradient Flow. 2023. URL https://arxiv.org/abs/2210.17216.
- Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity, July 2023. URL http://arxiv.org/abs/2307.08286.