2000 character limit reached
Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad
Published 5 Mar 2024 in cs.LG, cs.AI, and math.OC | (2403.02648v4)
Abstract: Adaptive methods are extremely popular in machine learning as they make learning rate tuning less expensive. This paper introduces a novel optimization algorithm named KATE, which presents a scale-invariant adaptation of the well-known AdaGrad algorithm. We prove the scale-invariance of KATE for the case of Generalized Linear Models. Moreover, for general smooth non-convex problems, we establish a convergence rate of for KATE, matching the best-known ones for AdaGrad and Adam. We also compare KATE to other state-of-the-art adaptive algorithms Adam and AdaGrad in numerical experiments with different problems, including complex machine learning tasks like image classification and text classification on real data. The results indicate that KATE consistently outperforms AdaGrad and matches/surpasses the performance of Adam in all considered scenarios.
- A. Agresti. Foundations of linear and generalized linear models. John Wiley & Sons, 2015.
- Lower bounds for non-convex stochastic optimization. Mathematical Programming, 199(1-2):165–214, 2023.
- Lower bounds for finding stationary points i. Mathematical Programming, 184(1-2):71–120, 2020.
- A. Defazio and K. Mishchenko. Learning-rate-free learning by D-adaptation. arXiv preprint arXiv:2301.07733, 2023.
- A simple convergence proof of adam and adagrad. arXiv preprint arXiv:2003.02395, 2020.
- BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
- Stochastic mirror descent: Convergence analysis and adaptive variants via the mirror stochastic polyak stepsize. arXiv preprint arXiv:2110.15412, 2021.
- Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
- The power of adaptivity in sgd: Self-tuning step sizes with unbounded gradients and affine variance. In Conference on Learning Theory, pages 313–355. PMLR, 2022.
- E. L. Frome. The analysis of rates using poisson regression models. Biometrics, pages 665–674, 1983.
- Stochastic polyak stepsize with a moving target. arXiv preprint arXiv:2106.11851, 2021.
- Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
- S. Horváth and P. Richtárik. A better alternative to error feedback for communication-efficient distributed learning. arXiv preprint arXiv:2006.11077, 2020.
- Applied logistic regression, volume 398. John Wiley & Sons, 2013.
- D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. 2009.
- Sp2: A second order stochastic polyak method. arXiv preprint arXiv:2207.08171, 2022.
- X. Li and F. Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. In The 22nd international conference on artificial intelligence and statistics, pages 983–992. PMLR, 2019.
- On the convergence of adagrad on ℝdsuperscriptℝ𝑑\mathbb{R}^{d}blackboard_R start_POSTSUPERSCRIPT italic_d end_POSTSUPERSCRIPT: Beyond convexity, non-asymptotic rate and acceleration. arXiv preprint arXiv:2209.14827, 2022.
- Stochastic polyak step-size for sgd: An adaptive learning rate for fast convergence. In International Conference on Artificial Intelligence and Statistics, pages 1306–1314. PMLR, 2021.
- H. B. McMahan and M. Streeter. Adaptive bound optimization for online convex optimization. arXiv preprint arXiv:1002.4908, 2010.
- K. Mishchenko and A. Defazio. Prodigy: An expeditiously adaptive parameter-free learner. arXiv preprint arXiv:2306.06101, 2023.
- Generalized linear models. Journal of the Royal Statistical Society Series A: Statistics in Society, 135(3):370–384, 1972.
- A. M. Oberman and M. Prazeres. Stochastic gradient descent with polyak’s learning rate. arXiv preprint arXiv:1903.08688, 2019.
- Dynamics of sgd with stochastic polyak stepsizes: Truly adaptive variants and convergence to exact solution. Advances in Neural Information Processing Systems, 35:26943–26954, 2022.
- B. T. Polyak. Minimization of unsmooth functionals. USSR Computational Mathematics and Mathematical Physics, 9(3):14–29, 1969.
- On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237, 2019.
- H. Robbins and S. Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400–407, 1951.
- CARER: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3687–3697, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1404. URL https://www.aclweb.org/anthology/D18-1404.
- S. Shalev-Shwartz and S. Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
- AI-SARAH: Adaptive and implicit stochastic recursive gradient methods. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=WoXJFsJ6Zw.
- T. Tieleman and G. Hinton. Rmsprop: Divide the gradient by a running average of its recent magnitude. coursera: Neural networks for machine learning. COURSERA Neural Networks Mach. Learn, 17, 2012.
- Adagrad stepsizes: Sharp convergence over nonconvex landscapes. The Journal of Machine Learning Research, 21(1):9047–9076, 2020.
- Linear convergence of adaptive stochastic gradient descent. In International conference on artificial intelligence and statistics, pages 1475–1485. PMLR, 2020.
Paper Prompts
Sign up for free to create and run prompts on this paper.