Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Improving Convergence and Generalization Using Parameter Symmetries (2305.13404v3)

Published 22 May 2023 in cs.LG and math.OC

Abstract: In many neural networks, different values of the parameters may result in the same loss value. Parameter space symmetries are loss-invariant transformations that change the model parameters. Teleportation applies such transformations to accelerate optimization. However, the exact mechanism behind this algorithm's success is not well understood. In this paper, we show that teleportation not only speeds up optimization in the short-term, but gives overall faster time to convergence. Additionally, teleporting to minima with different curvatures improves generalization, which suggests a connection between the curvature of the minimum and generalization ability. Finally, we show that integrating teleportation into a wide range of optimization algorithms and optimization-based meta-learning improves convergence. Our results showcase the versatility of teleportation and demonstrate the potential of incorporating symmetry in optimization.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Osmar Aléssio. Formulas for second curvature, third curvature, normal curvature, first geodesic curvature and first geodesic torsion of implicit curve in n-dimensions. Computer Aided Geometric Design, 29(3-4):189–201, 2012.
  2. A modern look at the relationship between sharpness and generalization. International Conference on Machine Learning, 2023.
  3. Learning to learn by gradient descent by gradient descent. Advances in Neural Information Processing Systems, 29, 2016.
  4. The representation theory of neural networks. Mathematics, 9(24), 2021. ISSN 2227-7390.
  5. Neural teleportation. Mathematics, 11(2):480, 2023.
  6. Symmetry-invariant optimization in deep networks. arXiv preprint arXiv:1511.01754, 2015.
  7. Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems, 3:747–769, 2021.
  8. Entropy-sgd: Biasing gradient descent into wide valleys. International Conference on Learning Representations, 2017.
  9. Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6):141–142, 2012.
  10. Flat minima generalize for low-rank matrix recovery. arXiv preprint arXiv:2203.03756, 2022.
  11. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pp.  1019–1028. PMLR, 2017.
  12. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305, 2020.
  13. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Neural Information Processing Systems, 2018.
  14. The role of permutation invariance in linear mode connectivity of neural networks. International Conference on Learning Representations, 2022.
  15. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp.  1126–1135. PMLR, 2017.
  16. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
  17. Quiver neural networks. arXiv preprint arXiv:2207.12773, 2022.
  18. Universal approximation and model compression for radial neural networks. arXiv preprint arXiv:2107.02550v2, 2022.
  19. Functional dimension of feedforward relu neural networks. arXiv preprint arXiv:2209.04036, 2022.
  20. Flat minima. Neural computation, 9(1):1–42, 1997.
  21. Averaging weights leads to wider optima and better generalization. Conference on Uncertainty in Artificial Intelligence, 2018.
  22. On large-batch training for deep learning: Generalization gap and sharp minima. International Conference on Learning Representations, 2017.
  23. Fisher sam: Information geometry and sharpness aware minimisation. In International Conference on Machine Learning, pp.  11148–11161. PMLR, 2022.
  24. Learning multiple layers of features from tiny images. 2009.
  25. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. In International Conference on Learning Representations, 2021.
  26. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pp.  5905–5914. PMLR, 2021.
  27. John M Lee. Introduction to Smooth Manifolds. Graduate Texts in Mathematics, vol 218. Springer, New York, NY, 2013.
  28. Sgdr: Stochastic gradient descent with warm restarts. International Conference on Learning Representations, 2017.
  29. 𝒢𝒢\mathcal{G}caligraphic_G-SGD: Optimizing relu neural networks in its positively scale-invariant space. International Conference on Learning Representations, 2019.
  30. Path-SGD: Path-normalized optimization in deep neural networks. In Advances in Neural Information Processing Systems, 2015.
  31. Relative flatness and generalization. 35th Conference on Neural Information Processing Systems, 2021.
  32. How you start matters for generalization. arXiv preprint arXiv:2206.08558, 2022.
  33. Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint arXiv:1706.04454, 2017.
  34. Aleksandr Mikhailovich Shelekhov. On the curvatures of a curve in n-dimensional euclidean space. Russian Mathematics, 65(11):46–58, 2021.
  35. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In International Conference on Machine Learning, pp.  9722–9732. PMLR, 2021.
  36. Sebastian U. Stich. Unified optimal analysis of the (stochastic) gradient method. CoRR, 2019.
  37. Understanding the dynamics of gradient flow in overparameterized linear models. In International Conference on Machine Learning, pp.  10153–10161. PMLR, 2021.
  38. Twan Van Laarhoven. L2 regularization versus batch and weight normalization. Advances in Neural Information Processing Systems, 2017.
  39. Towards understanding generalization of deep learning: Perspective of loss landscapes. arXiv preprint arXiv:1706.10239, 2017.
  40. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
  41. Symmetry teleportation for accelerated optimization. Advances in Neural Information Processing Systems, 2022.
  42. Symmetries, flat minima, and the conserved quantities of gradient flow. International Conference on Learning Representations, 2023.
  43. Towards theoretically understanding why sgd generalizes better than adam in deep learning. Advances in Neural Information Processing Systems, 33:21285–21296, 2020.
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets