Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Stochastic Gradient Descent with Preconditioned Polyak Step-size (2310.02093v1)

Published 3 Oct 2023 in cs.LG and math.OC

Abstract: Stochastic Gradient Descent (SGD) is one of the many iterative optimization methods that are widely used in solving machine learning problems. These methods display valuable properties and attract researchers and industrial machine learning engineers with their simplicity. However, one of the weaknesses of this type of methods is the necessity to tune learning rate (step-size) for every loss function and dataset combination to solve an optimization problem and get an efficient performance in a given time budget. Stochastic Gradient Descent with Polyak Step-size (SPS) is a method that offers an update rule that alleviates the need of fine-tuning the learning rate of an optimizer. In this paper, we propose an extension of SPS that employs preconditioning techniques, such as Hutchinson's method, Adam, and AdaGrad, to improve its performance on badly scaled and/or ill-conditioned datasets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (31)
  1. An estimator for the diagonal of a matrix. Applied Numerical Mathematics, 57(11):1214–1229, 2007. Numerical Algorithms, Parallelism and Applications (2).
  2. Training neural networks for and by interpolation. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 799–809. PMLR, 9 2020.
  3. Subgradient methods. lecture notes of EE392o, Stanford University, Autumn Quarter, 2004:2004–2005, 2003.
  4. Bruce Christianson. Automatic Hessians by reverse accumulation. IMA Journal of Numerical Analysis, 12(2):135–150, 1992.
  5. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):2121–2159, 2011.
  6. Function value learning: Adaptive learning rates based on the polyak stepsize and function splitting in erm. arXiv preprint arXiv:2307.14528, 2023.
  7. Cutting some slack for sgd with adaptive polyak stepsizes. arXiv preprint arXiv:2202.12328, 2022.
  8. Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics-Simulation and Computation, 18(3):1059–1076, 1989.
  9. Doubly adaptive scaled algorithm for machine learning using second-order information. In International Conference on Learning Representations, 2022.
  10. Adaptive sgd with polyak stepsize and line-search: Robust convergence and variance reduction. arXiv preprint arXiv:2308.06058, 2023.
  11. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diego, CA, USA, 2015.
  12. Guanghui Lan. An optimal method for stochastic composite optimization. Mathematical Programming, 133:365–397, 2012.
  13. SP2 : A second order stochastic polyak method. In The Eleventh International Conference on Learning Representations, 2023.
  14. On the convergence of stochastic gradient descent with adaptive stepsizes. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89, pages 983–992. PMLR, 4 2019.
  15. Stochastic polyak step-size for sgd: An adaptive learning rate for fast convergence. In International Conference on Artificial Intelligence and Statistics, pages 1306–1314. PMLR, 2021.
  16. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
  17. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization, 19(4):1574–1609, 2009.
  18. Dynamics of sgd with stochastic polyak stepsizes: Truly adaptive variants and convergence to exact solution. In S Koyejo, S Mohamed, A Agarwal, D Belgrave, K Cho, and A Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 26943–26954. Curran Associates, Inc., 2022.
  19. Boris T Polyak. Introduction to optimization. New York, Optimization Software,, 1987.
  20. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838–855, 1992.
  21. Boris Teodorovich Polyak. Minimization of unsmooth functionals. USSR Computational Mathematics and Mathematical Physics, 9:14–29, 1969.
  22. Boris Teodorovich Polyak. A new method of stochastic approximation type. Avtomatika i Telemekhanika, 51(7):98–107, 1990.
  23. On the convergence of adam and beyond. In International Conference on Learning Representations, 2018.
  24. A stochastic approximation method. The Annals of Mathematical Statistics, 22:400–407, 1951.
  25. Stochastic gradient methods with preconditioned updates. arXiv preprint arXiv:2206.00285, 2022.
  26. A stochastic proximal polyak step size. arXiv preprint arXiv:2301.04935, 2023.
  27. Momo: Momentum models for adaptive learning rates. arXiv preprint arXiv:2305.07583, 2023.
  28. AI-SARAH: Adaptive and implicit stochastic recursive gradient methods. Transactions on Machine Learning Research, 2023.
  29. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26–31, 2012.
  30. Adagrad stepsizes: Sharp convergence over nonconvex landscapes. The Journal of Machine Learning Research, 21(1):9047–9076, 2020.
  31. Adahessian: An adaptive second order optimizer for machine learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35:10665–10673, 5 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Farshed Abdukhakimov (4 papers)
  2. Chulu Xiang (3 papers)
  3. Dmitry Kamzolov (27 papers)
  4. Martin Takáč (145 papers)
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now