Papers
Topics
Authors
Recent
Search
2000 character limit reached

Nonsmooth nonconvex stochastic heavy ball

Published 26 Apr 2023 in math.OC | (2304.13328v3)

Abstract: Motivated by the conspicuous use of momentum-based algorithms in deep learning, we study a nonsmooth nonconvex stochastic heavy ball method and show its convergence. Our approach builds upon semialgebraic (definable) assumptions commonly met in practical situations and combines a nonsmooth calculus with a differential inclusion method. Additionally, we provide general conditions for the sample distribution to ensure the convergence of the objective function. Our results are general enough to justify the use of subgradient sampling in modern implementations that heuristically apply rules of differential calculus on nonsmooth functions, such as backpropagation or implicit differentiation. As for the stochastic subgradient method, our analysis highlights that subgradient sampling can make the stochastic heavy ball method converge to artificial critical points. Thanks to the semialgebraic setting, we address this concern showing that these artifacts are almost surely avoided when initializations are randomized, leading the method to converge to Clarke critical points.

Authors (1)
  1. Tam Le 
Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. Mathematics and its Applications. Springer Netherlands (1988)
  2. Grundlehren der mathematischen Wissenschaften. Springer Berlin, Heidelberg (2012)
  3. In: H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, R. Garnett (eds.) Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc. (2019)
  4. In: H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, R. Garnett (eds.) Advances in Neural Information Processing Systems, vol. 32, pp. 690–701. Curran Associates, Inc. (2019)
  5. Benaïm, M.: Dynamics of stochastic approximation algorithms. In: Seminaire de probabilites XXXIII, pp. 1–68. Springer (1999)
  6. SIAM Journal on Control and Optimization 44(1), 328–348 (2005). DOI 10.1137/S0363012904439301
  7. In: H.D. III, A. Singh (eds.) Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 119, pp. 810–821. PMLR (2020)
  8. Set-Valued and Variational Analysis 30(3), 1117–1147 (2022). DOI 10.1007/s11228-022-00638-z
  9. arXiv preprint arXiv:2112.05482 (2021)
  10. arXiv preprint arXiv:2202.13744 (2022)
  11. In: M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, J.W. Vaughan (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 13,537–13,549. Curran Associates, Inc. (2021)
  12. In: H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin (eds.) Advances in Neural Information Processing Systems, vol. 33, pp. 10,809–10,819. Curran Associates, Inc. (2020)
  13. Mathematical Programming 188(1), 19–51 (2021). DOI 10.1007/s10107-020-01501-5
  14. Journal of the European Mathematical Society (2022). DOI 10.4171/JEMS/1285
  15. Journal of Machine Learning Research 22(134), 1–31 (2021)
  16. Clarke, F.H.: Optimization and Nonsmooth Analysis. Classics in Applied Mathematics. Society for Industrial and Applied Mathematics (1990)
  17. Duke Mathematical Journal 156(2), 311–348 (2011). DOI 10.1215/00127094-2010-213
  18. Coste, M.: An introduction to o-minimal geometry. Institut de Recherche Mathématique de Rennes (1999)
  19. Foundations of Computational Mathematics 20(1), 119–154 (2020). DOI 10.1007/s10208-018-09409-5
  20. Duke Math. J. 84(2), 497–540 (1996). DOI 10.1215/S0012-7094-96-08416-1
  21. Cybernetics and Systems Analysis 34(2), 196–215 (1998)
  22. In: S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, R. Garnett (eds.) Advances in Neural Information Processing Systems, vol. 31. Curran Associates, Inc. (2018)
  23. Electronic Journal of Statistics 12(1), 461–529 (2018)
  24. CoRR abs/1412.6980 (2014)
  25. In: F. Pereira, C. Burges, L. Bottou, K. Weinberger (eds.) Advances in Neural Information Processing Systems, vol. 25. Curran Associates, Inc. (2012)
  26. Nature 521(7553), 436–444 (2015). DOI 10.1038/nature14539
  27. arXiv: Optimization and Control (2018)
  28. Norkin, V.: Generalized-differentiable functions. Cybernetics and Systems Analysis 16, 10–12 (1980). DOI 10.1007/BF01099354
  29. Norkin, V.: Stochastic generalized-differentiable functions in the problem of nonconvex nonsmooth stochastic optimization. Cybernetics and Systems Analysis 22, 804–809 (1986). DOI 10.1007/BF01068698
  30. In: R.P. Adams, V. Gogate (eds.) Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, Proceedings of Machine Learning Research, vol. 115, pp. 356–366. PMLR (2020)
  31. Polyak, B.T.: Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics 4(5), 1–17 (1964). DOI https://doi.org/10.1016/0041-5553(64)90137-5
  32. Ruszczyński, A.: Convergence of a stochastic subgradient method with averaging for nonsmooth nonconvex constrained optimization. Optimization Letters 14(7), 1615–1625 (2020). DOI 10.1007/s11590-020-01537-8
  33. In: M. Belkin, S. Kpotufe (eds.) Proceedings of Thirty Fourth Conference on Learning Theory, Proceedings of Machine Learning Research, vol. 134, pp. 3935–3971. PMLR (2021)
  34. Springer (2012). DOI 10.1007/978-1-4614-1897-9
  35. In: S. Dasgupta, D. McAllester (eds.) Proceedings of the 30th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 28, pp. 1139–1147. PMLR, Atlanta, Georgia, USA (2013)
  36. arXiv: Optimization and Control (2016)
Citations (10)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up