Papers
Topics
Authors
Recent
Search
2000 character limit reached

Vanishing Feature: Diagnosing Model Merging and Beyond

Published 5 Feb 2024 in cs.LG and cs.AI | (2402.05966v4)

Abstract: Model merging offers an efficient way to combine pre-trained neural networks but often suffers from inconsistent performance, especially when merging models with different initializations. We identify the vanishing feature'' phenomenon, where input-induced features diminish during propagation through the merged model, degrading performance. Through theoretical and empirical analysis, we reveal that this phenomenon underpins challenges like variance collapse and explains techniques like permutation-based merging, post-merging normalization, etc. We show that existing normalization strategies can be enhanced by precisely targeting the vanishing feature issue. Leveraging these insights, we propose thePreserve-First Merging'' (PFM) strategy, which focuses on preserving early-layer features, enabling the merged models, for the first time, to outperform the original models in advanced settings without post-training. Furthermore, we demonstrate that the vanishing feature phenomenon extends to other contexts, such as model pruning. Applying post-pruning normalization to mitigate the issue significantly improves one-shot pruning performance at high sparsity, offering a simple and effective post-pruning solution. The code is available at https://github.com/XingyuQu/VF.

Authors (2)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Critical learning periods in deep networks. In International Conference on Learning Representations, 2018.
  2. Layerwise linear mode connectivity. arXiv preprint arXiv:2307.06966, 2023.
  3. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.04836, 2022.
  4. Why do we need weight decay in modern deep learning? arXiv preprint arXiv:2310.04415, 2023a.
  5. Sgd with large step sizes learns sparse features. In International Conference on Machine Learning, pp.  903–925. PMLR, 2023b.
  6. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  7. Loss surface simplexes for mode connecting volumes and fast ensembling. In International Conference on Machine Learning, pp.  769–779. PMLR, 2021.
  8. Random initialisations performing above chance and how to find them. arXiv preprint arXiv:2209.07509, 2022.
  9. Mode combinability: Exploring convex combinations of permutation aligned models. arXiv preprint arXiv:2308.11511, 2023.
  10. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.  248–255. Ieee, 2009.
  11. Deng, L. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6):141–142, 2012.
  12. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pp.  1309–1318. PMLR, 2018.
  13. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2110.06296, 2021.
  14. Proving linear mode connectivity of neural networks via optimal transport. arXiv preprint arXiv:2310.19103, 2023.
  15. Fisher, R. A. Theory of statistical estimation. In Mathematical proceedings of the Cambridge philosophical society, volume 22, pp.  700–725. Cambridge University Press, 1925.
  16. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33:5850–5861, 2020.
  17. Frankle, J. Revisiting” qualitatively characterizing neural network optimization problems”. arXiv preprint arXiv:2012.06898, 2020.
  18. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018.
  19. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp.  3259–3269. PMLR, 2020.
  20. Topology and geometry of half-rectified network optimization. arXiv preprint arXiv:1611.01540, 2016.
  21. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018.
  22. Qualitatively characterizing neural network optimization problems. arXiv preprint arXiv:1412.6544, 2014.
  23. Using mode connectivity for loss landscape analysis. arXiv preprint arXiv:1806.06977, 2018.
  24. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
  25. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.  1026–1034, 2015.
  26. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  27. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022.
  28. Transformer fusion with optimal transport. arXiv preprint arXiv:2310.05719, 2023.
  29. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp.  448–456. pmlr, 2015.
  30. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018.
  31. Repair: Renormalizing permuted activations for interpolation repair. arXiv preprint arXiv:2211.08403, 2022.
  32. Linear connectivity reveals generalization strategies. arXiv preprint arXiv:2205.12411, 2022.
  33. Learning multiple layers of features from tiny images. 2009.
  34. Deep learning. nature, 521(7553):436–444, 2015.
  35. Eagleeye: Fast sub-net evaluation for efficient neural network pruning. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pp.  639–654. Springer, 2020.
  36. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710, 2016.
  37. Mechanistic mode connectivity. In International Conference on Machine Learning, pp.  22965–23004. PMLR, 2023.
  38. Torchvision the machine-vision package of torch. In Proceedings of the 18th ACM international conference on Multimedia, pp.  1485–1488, 2010.
  39. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.  1273–1282. PMLR, 2017.
  40. Special properties of gradient descent with large learning rates. In International Conference on Machine Learning, pp.  25082–25104. PMLR, 2023.
  41. Uniform convergence may be unable to explain generalization in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
  42. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512–523, 2020.
  43. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
  44. Unmasking the lottery ticket hypothesis: What’s encoded in a winning ticket’s mask? arXiv preprint arXiv:2210.03044, 2022.
  45. Deep networks on toroids: removing symmetries reveals the structure of flat regions in the landscape geometry. In International Conference on Machine Learning, pp.  17759–17781. PMLR, 2022.
  46. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  47. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems, 33:18098–18109, 2020.
  48. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045–22055, 2020.
  49. Optimizing mode connectivity via neuron alignment. Advances in Neural Information Processing Systems, 33:15300–15311, 2020.
  50. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  51. Plateau in monotonic linear interpolation–a” biased” view of loss landscape for deep networks. arXiv preprint arXiv:2210.01019, 2022.
  52. Learning neural network subspaces. In International Conference on Machine Learning, pp.  11217–11227. PMLR, 2021.
  53. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp.  23965–23998. PMLR, 2022.
  54. Revisiting permutation symmetry for merging models between different datasets. arXiv preprint arXiv:2306.05641, 2023.
  55. Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers. arXiv preprint arXiv:1802.00124, 2018.
  56. On convexity and linear mode connectivity in neural networks. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022.
  57. Fixup initialization: Residual learning without normalization. arXiv preprint arXiv:1901.09321, 2019.
  58. Going beyond linear mode connectivity: The layerwise linear feature connectivity. arXiv preprint arXiv:2307.08286, 2023.
  59. Sparse model soups: A recipe for improved pruning via model averaging. arXiv preprint arXiv:2306.16788, 2023.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Tweets

Sign up for free to view the 2 tweets with 0 likes about this paper.

Sign Up for Free