Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds (1605.07147v2)

Abstract: We study optimization of finite sums of geodesically smooth functions on Riemannian manifolds. Although variance reduction techniques for optimizing finite-sums have witnessed tremendous attention in the recent years, existing work is limited to vector space problems. We introduce Riemannian SVRG (RSVRG), a new variance reduced Riemannian optimization method. We analyze RSVRG for both geodesically convex and nonconvex (smooth) functions. Our analysis reveals that RSVRG inherits advantages of the usual SVRG method, but with factors depending on curvature of the manifold that influence its convergence. To our knowledge, RSVRG is the first provably fast stochastic Riemannian method. Moreover, our paper presents the first non-asymptotic complexity analysis (novel even for the batch setting) for nonconvex Riemannian optimization. Our results have several implications; for instance, they offer a Riemannian perspective on variance reduced PCA, which promises a short, transparent convergence analysis.

• The paper introduces Riemannian SVRG, a variance-reduced stochastic method that adapts SVRG to Riemannian manifolds, achieving linear convergence for strongly geodesically convex functions.
• The paper provides a non-asymptotic complexity analysis that quantifies the impact of manifold curvature through a geometric constant for both convex and nonconvex functions.
• The method demonstrates practical advantages in applications like PCA and covariance matrix computations, enabling efficient optimization in large-scale machine learning problems.

Fast Stochastic Optimization on Riemannian Manifolds: A Summary

The paper "Fast Stochastic Optimization on Riemannian Manifolds" addresses the challenge of optimizing finite sums of geodesically smooth functions on Riemannian manifolds, which is a fundamental problem in machine learning. The paper introduces Riemannian SVRG (Rsvrg), a variance-reduced optimization method tailored to the geometry of Riemannian manifolds, and provides an analytic framework extending known variance reduction techniques from Euclidean spaces to this more complex setting.

Key Contributions

1. Algorithm Development: The core contribution is the introduction of Riemannian SVRG, a stochastic gradient method based on the popular SVRG algorithm. This method accounts for manifold curvature through operations like exponential maps and parallel transport, offering a significant computational advantage over conventional methods with linear convergence rates for strongly geodesically convex functions.
2. Analytic Insights: The paper is notable for providing the first non-asymptotic complexity analysis for nonlinear Riemannian excitation, treating cases of both convex and nonconvex functions. The analysis delineates dependencies on manifold curvature, capturing curvature effects through the geometric constant $\zeta$. For geodesically strongly convex functions, Riemannian SVRG achieves linear convergence, whereas for nonconvex or gradient-dominated functions, it offers notable improvement over basic stochastic methods.
3. Applications and Practical Implications: Riemannian SVRG's utility is underscored in applications like variance reduced Principal Component Analysis (PCA) and the computation of the Riemannian centroid of covariance matrices. Especially in computing leading eigenvectors, the analysis supports faster convergence theories and practical implementations which resonate in contemporary data science applications.

Implications and Future Directions

The introduction of Riemannian SVRG is impactful for several reasons. Practically, it enables the application of stochastic gradient algorithms to large-scale problems where manifold considerations are critical, such as low-rank matrix approximation and geometric deep learning. Theoretically, it adds a new dimension to understanding the dynamics of stochastic optimization beyond Euclidean settings, providing tools and insights that could be generalized to other geometric optimization challenges.

Looking forward, an important avenue for future research is the systematic exploration of retraction and vector transport's impact on the algorithm's performance. While this paper sticks to precise Riemannian computations, many practical implementations hinge on efficient approximations of these operations, and understanding their interplay with convergence and complexity results would be invaluable.

In conclusion, this paper provides a well-structured advancement in Riemannian optimization frameworks, equipping researchers with tools to handle a broader class of optimization problems more efficiently. The detailed theoretical groundwork is poised to pave the way for cutting-edge algorithmic innovations in machine learning and beyond.