Papers
Topics
Authors
Recent
Search
2000 character limit reached

G-TRACER: Expected Sharpness Optimization

Published 24 Jun 2023 in stat.ML and cs.LG | (2306.13914v1)

Abstract: We propose a new regularization scheme for the optimization of deep learning architectures, G-TRACER ("Geometric TRACE Ratio"), which promotes generalization by seeking flat minima, and has a sound theoretical basis as an approximation to a natural-gradient descent based optimization of a generalized Bayes objective. By augmenting the loss function with a TRACER, curvature-regularized optimizers (eg SGD-TRACER and Adam-TRACER) are simple to implement as modifications to existing optimizers and don't require extensive tuning. We show that the method converges to a neighborhood (depending on the regularization strength) of a local minimum of the unregularized objective, and demonstrate competitive performance on a number of benchmark computer vision and NLP datasets, with a particular focus on challenging low signal-to-noise ratio problems.

Authors (2)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (27)
  1. On the convergence of sgd with biased gradients, 2021.
  2. S. Amari. Natural gradient works efficiently in learning. Neural Computation, 10(2):251–276, 1998.
  3. T. W. Anderson. An introduction to multivariate statistical analysis. Wiley Series in Probability and Statistics, 2003.
  4. A general framework for updating belief distributions. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(5):1103–1130, feb 2016. doi: 10.1111/rssb.12158. URL https://doi.org/10.1111%2Frssb.12158.
  5. Optimization methods for large-scale machine learning. 2016. doi: 10.48550/ARXIV.1606.04838. URL https://arxiv.org/abs/1606.04838.
  6. Backpack: Packing more into backprop, 2020.
  7. Sharp minima can generalize for deep nets. CoRR, abs/1703.04933, 2017. URL http://arxiv.org/abs/1703.04933.
  8. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):2121–2159, 2011. URL http://jmlr.org/papers/v12/duchi11a.html.
  9. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. 2017. doi: 10.48550/ARXIV.1703.11008. URL https://arxiv.org/abs/1703.11008.
  10. Sharpness-aware minimization for efficiently improving generalization. CoRR, abs/2010.01412, 2020. URL https://arxiv.org/abs/2010.01412.
  11. Geoffrey E. Hinton and Drew van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Annual Conference on Computational Learning Theory, COLT ’93, pp.  5–13, New York, NY, USA, 1993. Association for Computing Machinery. ISBN 0897916115. doi: 10.1145/168304.168306. URL https://doi.org/10.1145/168304.168306.
  12. Flat Minima. Neural Computation, 9(1):1–42, 01 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.1.1. URL https://doi.org/10.1162/neco.1997.9.1.1.
  13. Understanding generalization through visualizations. CoRR, abs/1906.03291, 2019. URL http://arxiv.org/abs/1906.03291.
  14. Categorical reparameterization with Gumbel-Softmax, 2016. URL https://arxiv.org/abs/1611.01144.
  15. The Bayesian learning rule, 2021. URL https://arxiv.org/abs/2107.04562.
  16. Fast and scalable Bayesian deep learning by weight-perturbation in adam. 2018. doi: 10.48550/ARXIV.1806.04854. URL https://arxiv.org/abs/1806.04854.
  17. Fisher SAM: Information geometry and sharpness aware minimisation. 2022. doi: 10.48550/ARXIV.2206.04920. URL https://arxiv.org/abs/2206.04920.
  18. Adam: A method for stochastic optimization, 2014. URL https://arxiv.org/abs/1412.6980.
  19. Limitations of the empirical fisher approximation. CoRR, abs/1905.12558, 2019. URL http://arxiv.org/abs/1905.12558.
  20. (not) bounding the true error. In T. Dietterich, S. Becker, and Z. Ghahramani (eds.), Advances in Neural Information Processing Systems, volume 14. MIT Press, 2001. URL https://proceedings.neurips.cc/paper/2001/file/98c7242894844ecd6ec94af67ac8247d-Paper.pdf.
  21. James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21(146):1–76, 2020. URL http://jmlr.org/papers/v21/17-678.html.
  22. David A. McAllester. Some PAC-Bayesian theorems. 1999. URL https://doi.org/10.1023/A:1007618624809.
  23. Empirical analysis of the Hessian of over-parameterized neural networks. CoRR, abs/1706.04454, 2017. URL http://arxiv.org/abs/1706.04454.
  24. Implicit regularization of SGD via thermophoresis. In Advances in Neural Information Processing Systems, Machine Learning for Physical Sciences Workshop, 2020.
  25. Bayesian deep learning and a probabilistic perspective of generalization. CoRR, abs/2002.08791, 2020. URL https://arxiv.org/abs/2002.08791.
  26. ADAHESSIAN: an adaptive second order optimizer for machine learning. CoRR, abs/2006.00719, 2020. URL https://arxiv.org/abs/2006.00719.
  27. Noisy natural gradient as variational inference. CoRR, abs/1712.02390, 2017. URL http://arxiv.org/abs/1712.02390.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up