Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Optimization on manifolds: A symplectic approach (2107.11231v2)

Published 23 Jul 2021 in cond-mat.stat-mech, math.OC, and stat.ML

Abstract: Optimization tasks are crucial in statistical machine learning. Recently, there has been great interest in leveraging tools from dynamical systems to derive accelerated and robust optimization methods via suitable discretizations of continuous-time systems. However, these ideas have mostly been limited to Euclidean spaces and unconstrained settings, or to Riemannian gradient flows. In this work, we propose a dissipative extension of Dirac's theory of constrained Hamiltonian systems as a general framework for solving optimization problems over smooth manifolds, including problems with nonlinear constraints. We develop geometric/symplectic numerical integrators on manifolds that are "rate-matching," i.e., preserve the continuous-time rates of convergence. In particular, we introduce a dissipative RATTLE integrator able to achieve optimal convergence rate locally. Our class of (accelerated) algorithms are not only simple and efficient but also applicable to a broad range of contexts.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. P. A. M. Dirac, “Generalized Hamiltonian dynamics,” Canadian J. of Math. 2 (1950) 129–148.
  2. P. A. M. Dirac, Lectures on Quantum Mechanics. Dover Publications, 2001.
  3. M. Henneaux and C. Tetelboim, Quantization of Gauge Systems. Princeton University Press, 1994.
  4. Princeton University Press, 2008.
  5. N. Boumal, An Introduction to Optimization on Smooth Manifolds. Cambridge University Press, 2023.
  6. C. Criscitiello and N. Boumal, “An accelerated first-order method for non-convex optimization on manifolds,” Found. Comput. Math. (2022) .
  7. H. Zhang and S. Sra, “First-order methods for geodesically convex optimization,” Conf. Learning Theory (2016) 1617–1638.
  8. J. Townsend, N. Koep, and S. Weichwald, “Pymanopt: A Python toolbox for optimization on manifolds using automatic differentiation,” J. Mach. Learn. Res. 17 (2016) 1–5.
  9. K. Ahn and S. Sra, “From Nesterov’s estimate sequence to Riemannian acceleration,” Conf. Learning Theory (2020) 84–118.
  10. F. Alimisis, A. Orvieto, G. Becigneul, and A. Lucchi, “Momentum improves optimization on Riemannian manifolds,” AISTATS 130 (2021) 1351–1359.
  11. R. Berndt, An Introduction to Symplectic Geometry. American Mathematical Society, 2000.
  12. A. Wibisono, A. C. Wilson, and M. I. Jordan, “A variational perspective on accelerated methods in optimization,” Proc. Nat. Acad. Sci. 113 no. 47, (2016) E7351–E7358.
  13. G. França, J. Sulam, D. P. Robinson, and R. Vidal, “Conformal symplectic and relativistic optimization,” J. Stat. Mech. 2020 no. 12, (2020) 124008.
  14. M. Betancourt, M. I. Jordan, and A. Wilson, “On symplectic optimization,” arXiv:1802.03653 [stat.CO].
  15. M. Muehlebach and M. I. Jordan, “Optimization with momentum: dynamical, control-theoretic, and symplectic perspectives,” J. Mach. Learn. Res. 22 no. 73, (2021) 1–50.
  16. A. Bravetti, M. L. Daza-Torres, H. Flores-Arguedas, and M. Betancourt, “Optimization algorithms inspired by the geometry of dissipative systems,” arXiv:1912.02928 [math.OC].
  17. G. França, D. P. Robinson, and R. Vidal, “Gradient flows and proximal splitting methods: A unified view on accelerated and stochastic optimization,” Phys. Rev. E 103 (2021) 053304.
  18. G. França, D. P. Robinson, and R. Vidal, “A nonsmooth dynamical systems perspective on accelerated extensions of ADMM,” IEEE Trans. Automatic Control 68 no. 5, (2023) 2966–2978.
  19. G. Benettin and A. Giorgilli, “On the Hamiltonian interpolation of near-to-the-identity symplectic mappings with application to symplectic integration algorithms,” J. Stat. Phys. 74 (1994) 1117–1143.
  20. R. I. McLachlan and G. R. W. Quispel, “Splitting methods,” Acta Numer. 11 (2002) 341.
  21. B. Leimkuhler and S. Reich, Simulating Hamiltonian Dynamics. Cambridge University Press, 2004.
  22. R. I. McLachlan, G. Quispel, and W. Reinout, “Geometric integrators for ODEs,” J. Phys. A: Math. and Gen. 39 no. 19, (2006) 5251.
  23. Springer, 2010.
  24. G. França, M. I. Jordan, and R. Vidal, “On dissipative symplectic integration with applications to gradient-based optimization,” J. Stat. Mech. 2021 no. 4, (2021) 043402.
  25. M. J. Gotay, J. M. Nester, and G. Hinds, “Presymplectic manifolds and the Dirac–Bergmann theory of constraints,” J. Math. Phys. no. 19, (1978) 2388.
  26. J. E. Marsden and M. West, “Discrete mechanics and variational integrators,” Acta Numer. 10 (2001) 357–514.
  27. Y. Nesterov, Lectures on convex optimization. Springer, 2018.
  28. H. C. Andersen, “Rattle: A “velocity” version of the SHAKE algorithm for molecular dynamics calculations,” J. Comput. Phys. 52 no. 1, (1983) 24–34.
  29. B. J. Leimkuhler and R. D. Skeel, “Symplectic numerical integrators in constrained Hamiltonian systems,” J. Comput. Phys. 112 (1994) 117–125.
  30. S. Reich, “Symplectic integration of constrained Hamiltonian systems by composition methods,” SIAM J. Numer. Anal. 32 no. 3, (1996) 475–491.
  31. B. Leimkuhler and C. Matthews, “Efficient molecular dynamics using geodesic integration and solvent–solute splitting,” Proc. Royal Soc. A: Math., Phys. and Eng. Sci. 472 no. 2189, (2016) 20160138.
  32. J. Baik, E. Collins-Woodfin, P. L. Doussal, and H. Wu, “Spherical spin glass model with external field,” J. Stat. Phys. 183 no. 31, (2021) .
  33. Springer, 1989.
  34. J. E. Marsden and T. S. Ratiu, Introduction to Mechanics and Symmetry: A Basic Exposition of Classical Mechanical Systems. Springer, 2010.
  35. S. M. Carroll, Spacetime and Geometry: An Introduction to General Relativity. Cambridge University Press, 2019.
  36. IOP Publishing, 2003.
  37. H. Marthinsen and B. Owren, “Geometric integration of non-autonomous Hamiltonian problems,” Adv. Comput. Math. 42 (2016) 313–332.
  38. M. Asorey, J. F. Cariñena, and L. A. Ibort, “Generalized canonical transformations for time-dependent systems,” J. Math. Phys. 24 no. 12, (1983) 2745–2750.
  39. A. C. Hansen, “A theoretical framework for backward error analysis on manifolds,” J. Geom. Mech. 3 no. 1, (2011) 81–111.
  40. S. Wiggins, Introduction to Applied Nonlinear Dynamical Systems and Chaos. Springer, 2000.
  41. P. Hartman, “A lemma in the theory of structural stability of differential equations,” Proc. Amer. Math. Soc. 11 (1960) 610–620.
  42. A. Barp, A. Kennedy, and M. Girolami, “Hamiltonian Monte Carlo on symmetric and homogeneous spaces via symplectic reduction,” arXiv:1903.02699 [stat.CO]. arXiv:1903.02699v2 [stat.CO].
  43. J. C. Gower and G. B. Dijksterhuis, Procrustes Problems. Oxford University Press, 2004.
  44. G. Wahba, “A least squares estimate of satellite attitude,” SIAM Review 7 no. 3, (1965) 409–409.
Citations (11)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now