Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
120 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Layer-wise Linear Mode Connectivity (2307.06966v3)

Published 13 Jul 2023 in cs.LG

Abstract: Averaging neural network parameters is an intuitive method for fusing the knowledge of two independent models. It is most prominently used in federated learning. If models are averaged at the end of training, this can only lead to a good performing model if the loss surface of interest is very particular, i.e., the loss in the midpoint between the two models needs to be sufficiently low. This is impossible to guarantee for the non-convex losses of state-of-the-art networks. For averaging models trained on vastly different datasets, it was proposed to average only the parameters of particular layers or combinations of layers, resulting in better performing models. To get a better understanding of the effect of layer-wise averaging, we analyse the performance of the models that result from averaging single layers, or groups of layers. Based on our empirical and theoretical investigation, we introduce a novel notion of the layer-wise linear connectivity, and show that deep networks do not have layer-wise barriers between them.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Git re-basin: Merging models modulo permutation symmetries. In International Conference on Learning Representations, 2022.
  2. A modern look at the relationship between sharpness and generalization. In International Conference on Machine Learning. PMLR, 2023a.
  3. Sgd with large step sizes learns sparse features. In International Conference on Machine Learning. PMLR, 2023b.
  4. Federated learning with personalization layers. arXiv preprint arXiv:1912.00818, 2019.
  5. Revisiting model stitching to compare neural representations. In Advances in neural information processing systems, volume 34, pp.  225–236, 2021.
  6. Pythia: A suite for analyzing large language models across training and scaling, 2023.
  7. The intriguing role of module criticality in the generalization of deep networks. In International Conference on Learning Representations, 2020.
  8. On the importance and applicability of pre-training for federated learning. In International Conference on Learning Representations, 2022.
  9. Clip itself is a strong fine-tuner: Achieving 85.7% and 88.0% top-1 accuracy with vit-b and vit-l on imagenet. arXiv preprint arXiv:2212.06138, 2022.
  10. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pp. 1309–1318. PMLR, 2018.
  11. The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Representations, 2022.
  12. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2020.
  13. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757, 2019.
  14. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259–3269. PMLR, 2020.
  15. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in neural information processing systems, volume 31, 2018.
  16. How much data are augmentations worth? an investigation into scaling laws, invariance, and implicit regularization. In ICLR, 2023.
  17. Editing models with task arithmetic. In International Conference on Learning Representations, 2022.
  18. Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pp.  876–885. Association For Uncertainty in Artificial Intelligence (AUAI), 2018.
  19. Arthur Jacot. Bottleneck structure in learned features: Low-dimension vs regularity tradeoff. arXiv preprint arXiv:2305.19008, 2023.
  20. Repair: Renormalizing permuted activations for interpolation repair. In International Conference on Learning Representations, 2022.
  21. Fedbn: Federated learning on non-iid features via local batch normalization. In International Conference on Learning Representations, 2021.
  22. Towards explaining the regularization effect of initial large learning rate in training neural networks. Advances in Neural Information Processing Systems, 32, 2019.
  23. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523, 2020.
  24. Bad global minima exist and sgd can reach them. Advances in Neural Information Processing Systems, 33:8543–8552, 2020.
  25. Mechanistic mode connectivity. In International Conference on Machine Learning, pp. 22965–23004. PMLR, 2023.
  26. Layer-wised model aggregation for personalized federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10092–10101, 2022.
  27. Can neural network memorization be localized? In International Conference on Machine Learning, 2023.
  28. Normalization layers are all that sharpness-aware minimization needs. arXiv preprint arXiv:2306.04226, 2023.
  29. Fedbabu: Toward enhanced representation for federated image classification. In International Conference on Learning Representations, 2021.
  30. Task arithmetic in the tangent space: Improved editing of pre-trained models. arXiv preprint arXiv:2305.12827, 2023.
  31. Relative flatness and generalization. In Advances in neural information processing systems, volume 34, pp.  18420–18432, 2021.
  32. Federated learning with partial model personalization. In International Conference on Machine Learning, pp. 17716–17758. PMLR, 2022.
  33. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in neural information processing systems, volume 30, 2017.
  34. Revisiting adapters with adversarial training. In International Conference on Learning Representations, 2023.
  35. The effects of mild over-parameterization on the optimization landscape of shallow relu neural networks. In Conference on Learning Theory, pp.  3889–3934. PMLR, 2021.
  36. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In International Conference on Machine Learning, pp. 9722–9732. PMLR, 2021.
  37. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045–22055, 2020.
  38. Robustness may be at odds with accuracy. In International Conference on Learning Representations, 2018.
  39. What can linear interpolation of neural network loss landscapes tell us? In International Conference on Machine Learning, pp. 22325–22341. PMLR, 2022.
  40. Federated learning with matched averaging. In International Conference on Learning Representations, 2020.
  41. Learning neural network subspaces. In International Conference on Machine Learning, pp. 11217–11227. PMLR, 2021.
  42. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 23965–23998. PMLR, 2022.
  43. How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective. In Advances in Neural Information Processing Systems, volume 31, 2018.
  44. A walk with sgd. arXiv preprint arXiv:1802.08770, 2018.
  45. Robustness and generalization. Machine learning, 86:391–423, 2012.
  46. Zongwei Zhou Yixiong Chen, Alan Yuille. Which layer is learning faster? a systematic exploration of layer-wise convergence rate for deep neural networks. In International Conference on Learning Representations, 2023.
  47. On convexity and linear mode connectivity in neural networks. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022.
  48. Are all layers created equal? Journal of Machine Learning Research, 23(67):1–28, 2022.
  49. Going beyond linear mode connectivity: The layerwise linear feature connectivity. In Advances in neural information processing systems, 2023.
  50. Decentralized sgd and average-direction sam are asymptotically equivalent. In International Conference on Machine Learning, 2023.
Citations (10)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now