2000 character limit reached
Zeroth-order Riemannian Averaging Stochastic Approximation Algorithms
Published 25 Sep 2023 in math.OC, cs.LG, and stat.ML | (2309.14506v1)
Abstract: We present Zeroth-order Riemannian Averaging Stochastic Approximation (\texttt{Zo-RASA}) algorithms for stochastic optimization on Riemannian manifolds. We show that \texttt{Zo-RASA} achieves optimal sample complexities for generating $\epsilon$-approximation first-order stationary solutions using only one-sample or constant-order batches in each iteration. Our approach employs Riemannian moving-average stochastic gradient estimators, and a novel Riemannian-Lyapunov analysis technique for convergence analysis. We improve the algorithm's practicality by using retractions and vector transport, instead of exponential mappings and parallel transports, thereby reducing per-iteration complexity. Additionally, we introduce a novel geometric condition, satisfied by manifolds with bounded second fundamental form, which enables new error bounds for approximating parallel transport with vector transport.
- P.-A. Absil and J. Malick. Projection-like retractions on matrix manifolds. SIAM Journal on Optimization, 22(1):135–158, 2012.
- Optimization algorithms on matrix manifolds. Princeton University Press, 2008.
- Stochastic multilevel composition optimization algorithms with level-independent convergence rates. SIAM Journal on Optimization, 32(2):519–544, 2022.
- S. Bonnabel. Stochastic gradient descent on Riemannian manifolds. IEEE Transactions on Automatic Control, 58(9):2217–2229, 2013.
- N. Boumal. An introduction to optimization on smooth manifolds. Cambridge University Press, 2023.
- Manopt, a Matlab toolbox for optimization on manifolds. Journal of Machine Learning Research, 15(42):1455–1459, 2014. URL https://www.manopt.org.
- A course in metric geometry, volume 33. American Mathematical Society, 2022.
- Proximal gradient method for nonsmooth optimization over the Stiefel manifold. SIAM Journal on Optimization, 30(1):210–239, 2020.
- M. P. Do Carmo. Riemannian geometry, volume 6. Springer, 1992.
- The geometry of algorithms with orthogonality constraints. SIAM journal on Matrix Analysis and Applications, 20(2):303–353, 1998.
- K. A. Gallivan and P. Absil. Note on the convex hull of the Stiefel manifold. Technical note, 2010.
- E. S. Gawlik and M. Leok. High-order retractions on matrix manifolds using projected polynomials. SIAM Journal on Matrix Analysis and Applications, 39(2):801–828, 2018.
- S. Ghadimi and W. B. Powell. Stochastic search for a parametric cost function approximation: Energy storage with rolling forecasts. arXiv preprint arXiv:2204.07317, 2022.
- A single timescale stochastic approximation method for nested stochastic optimization. SIAM Journal on Optimization, 30(1):960–979, 2020.
- A. Han and J. Gao. Riemannian stochastic recursive momentum method for non-convex optimization. In IJCAI, 2020.
- W. Huang and K. Wei. Riemannian proximal gradient methods. Mathematical Programming, 194(1-2):371–413, 2022.
- A Broyden class of quasi-Newton methods for Riemannian optimization. SIAM Journal on Optimization, 25(3):1660–1685, 2015.
- G. Lan. First-order and Stochastic Optimization Methods for Machine Learning. Springer Nature, 2020.
- Stochastic zeroth-order Riemannian derivative estimation and optimization. Mathematics of Operations Research, 48(2):1183–1211, 2022.
- Tracking and regret bounds for online zeroth-order Euclidean and Riemannian optimization. SIAM Journal on Optimization, 32(2):445–469, 2022.
- Characteristic classes. Number 76. Princeton university press, 1974.
- Y. Nesterov. Lectures on convex optimization, volume 137. Springer, 2018.
- B. T. Polyak. Comparison of the convergence rates for single-step and multi-step optimization algorithms in the presence of noise. Engineering Cybernetics, 15(1):6–10, 1977.
- A. Ruszczyński. A linearization method for nonsmooth stochastic programming problems. Mathematics of Operations Research, 12(1):32–49, 1987.
- A. Ruszczynski. A stochastic subgradient method for nonsmooth nonconvex multilevel composition optimization. SIAM Journal on Control and Optimization, 59(3):2301–2320, 2021.
- A. Ruszczynski and W. Syski. Stochastic approximation method with gradient averaging for unconstrained problems. IEEE Transactions on Automatic Control, 28(12):1097–1105, 1983.
- H. Sato. Riemannian conjugate gradient methods: General framework and specific algorithms with convergence analyses. SIAM Journal on Optimization, 32(4):2690–2717, 2022.
- Riemannian stochastic variance reduced gradient algorithm with retraction and vector transport. SIAM Journal on Optimization, 29(2):1444–1472, 2019.
- K. Scheinberg. Finite difference gradient approximation: To randomize or not? INFORMS Journal on Computing, 34(5):2384–2388, 2022.
- Matrix exponentiated gradient updates for on-line learning and Bregman projection. Journal of Machine Learning Research, 6(Jun):995–1018, 2005.
- L. W. Tu. An Introduction to Manifolds. Springer, 2011.
- T. Wang. On sharp stochastic zeroth-order Hessian estimators over Riemannian manifolds. Information and Inference: A Journal of the IMA, 12(2):787–813, 2023.
- From the Greene–Wu convolution to gradient estimation over Riemannian manifolds. arXiv:2108.07406, 2021.
- Online optimization over Riemannian manifolds. Journal of Machine Learning Research, 24(84):1–67, 2023.
- L. Xiao. Dual averaging method for regularized stochastic learning and online optimization. Advances in Neural Information Processing Systems, 22, 2009.
- Riemannian SVRG: Fast stochastic optimization on Riemannian manifolds. NeurIPS, 29, 2016.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
