---
title: Stochastic Gradient Descent on Riemannian Manifolds
url: https://www.emergentmind.com/topics/stochastic-gradient-descent-on-riemannian-manifolds
type: topic
---

# Stochastic Gradient Descent on Riemannian Manifolds

Stochastic Gradient Descent on Riemannian Manifolds is an extension of classical stochastic gradient techniques to the setting where the optimization variable is constrained to lie on a non-Euclidean, smooth manifold endowed with a Riemannian metric. Such problems arise in diverse applications, including matrix factorization with orthogonality constraints, optimization over symmetric positive-definite matrices, policy evaluation in reinforcement learning, and adversarial robustness in deep networks. The core challenge is to generalize stochastic approximation and descent methods in a way that fully leverages and respects manifold geometry, nonlinearity, and curvature.

## 1. Algorithmic Foundations

The canonical stochastic gradient descent (SGD) iteration on a Riemannian manifold \((\mathcal{M},g)\) is defined for the objective
\[
f(x) = \mathbb{E}_{\xi\sim\mathcal{D}}[Q(x,\xi)],
\]
where \(x \in \mathcal{M}\), and \(Q(x,\xi)\) is the sample loss. At each step \(t\), given a stochastic estimate \(H(x_t,\xi_t)\in T_{x_t}\mathcal{M}\) of the Riemannian gradient, the iterate is updated by moving along a geodesic or, more commonly in practice, via a retraction \(R_{x_t}\):
\[
x_{t+1} = R_{x_t}\left(-\gamma_t H(x_t, \xi_t)\right).
\]
The retraction \(R_{x_t}:T_{x_t}\mathcal{M}\to\mathcal{M}\) is a smooth mapping satisfying \(R_{x_t}(0) = x_t\), \(DR_{x_t}(0) = \mathrm{Id}_{T_{x_t}\mathcal{M}}\). In cases where the exponential map is computationally tractable, it may be used directly; otherwise, a suitable retraction provides a numerically efficient alternative [1111.5280, 2312.07990].

For batch or mini-batch stochasticity, the update becomes
\[
x_{t+1} = R_{x_t}\left(-\gamma_t\, \frac{1}{b}\sum_{i=1}^{b} H(x_t, \xi_{t,i})\right),
\]
where \(\{\xi_{t,i}\}\) are IID samples.

## 2. Geometric Assumptions and Convergence Theory

Standard convergence analysis for Riemannian SGD relies on the following geometric and analytical requirements:
- **Manifold regularity:** \(\mathcal{M}\) is connected, complete (typically, Hadamard if global nonpositive curvature desired), and equipped with a well-defined injectivity radius.
- **Retraction quality:** \(R_x\) is at least a first-order retraction, with higher-order retractions affording improved error bounds (notably for weak-approximation order in diffusion limits) [2402.03467].
- **Objective regularity:** The cost \(f\) is geodesically \(L\)-smooth, i.e., \(\|\operatorname{grad}f(x) - \Gamma_y^x \operatorname{grad}f(y)\|_x \leq L\, d(x,y)\).
- **Stochastic gradient model:** Unbiasedness and bounded second moments, \(\mathbb{E}[H(x,\xi)] = \operatorname{grad}f(x)\), \(\mathbb{E}\| H(x,\xi) - \operatorname{grad}f(x)\|^2 \leq \sigma^2\), extending naturally to mini-batch gradients with variance decaying as \(1/b\) [2312.07990].

Under standard Robbins-Monro step-size conditions (\(\sum \gamma_t = \infty\), \(\sum \gamma_t^2< \infty\)), almost sure convergence to a critical point (in the sense \(\|\operatorname{grad}f(x_t)\| \to 0\)) is guaranteed on compact manifolds or under suitable growth controls (for Hadamard settings, possibly with step-size normalization by local geometry) [1111.5280, 2312.07990].

The convergence rate for geodesically convex, \(L\)-smooth objectives and constant mini-batch size \(b\) is
\[
\min_{0\leq t < K} \mathbb{E}\|\operatorname{grad} f(x_t)\|^2 \leq O\!\left( \frac{1}{K} + \frac{\sigma^2}{b} \right)
\]
with \(K\) the iteration count. For polynomial decaying step sizes (e.g., \(\gamma_t = O(1/\sqrt{t})\)), rates are \(O(\log K / \sqrt{K})\), mirroring Euclidean results [2312.07990].

## 3. Architectural Variants and Acceleration Techniques

Beyond vanilla Riemannian SGD, a suite of variance-reduced and accelerated schemes have been developed:

- **Riemannian SVRG (R-SVRG):** Organizes SGD into epochs, correcting each stochastic gradient via a full-gradient anchor and parallel-transported corrections. This reduces variance and accelerates convergence for finite-sum objectives. In the geodesically strongly convex case, R-SVRG achieves a linear rate, and for nonconvex objectives, an \(O(n + n^{2/3}/\epsilon)\) gradient complexity for an \(\epsilon\)-stationary point [1605.07147, 1702.05594].

- **Riemannian Stochastic Hybrid Gradient (R-SHG):** Blends R-SGD, R-SVRG, and stochastic recursive gradients, with time-varying coefficients. This offers single-loop variance reduction with \(O(1/\sqrt{K})\) convergence under decaying steps and improved asymptotics under fixed-step regimes [2109.04289].

- **Variance-reduced saddle escaping:** In nonconvex settings, methods such as perturbed Riemannian SRG inject isotropic noise in the tangent space and use recursive gradients for saddle point escape, with optimal (up to log factors) second-order convergence complexities in both finite-sum and online settings [2010.12191].

- **Averaging and Polyak–Ruppert extensions:** Retraction-based iterative averaging improves the convergence rate of RSGD for strongly convex problems to \(O(1/n)\) for the averaged iterate, matching the optimal asymptotic rate and distribution [1802.09128].

- **Learning-rate-free RSGD:** Recent advances have introduced adaptive step-size Riemannian SGD procedures (e.g., RDoG), which use geometric quantities to automatically adjust learning rates on the fly, removing the need for meticulous hyperparameter tuning and still attaining optimal (up to logarithms) \(O(1/\sqrt{T})\) rates [2406.02296].

## 4. Extensions: Non-Smooth, Composite, and Decentralized Optimization

- **Nonsmooth/tame objectives:** SGD with retraction remains convergent for locally Lipschitz, Whitney \(C^k\)–stratifiable objectives (e.g., those arising from ReLU, batch-norm, or composite regularization) under essentially the same step-size policies as in the smooth setting, ensuring almost sure convergence to Clarke-stationary points [2302.00709].

- **Composition/nested objectives:** Riemannian Stochastic Composition Gradient Descent (R-SCGD) addresses situations where the objective function is a nested composition of expectations. A key innovation is tracking inner expectations via auxiliary sequences, yielding unbiased gradients and \(O(1/\epsilon^2)\) iteration complexity [2207.09350].

- **Decentralized/distributed settings:** Methods have been developed for networks of agents each optimizing local objectives under consensus constraints, crucial for federated and distributed learning. Convergence rates up to \(O(1/T)\) in consensus error and \(O(\log T/\sqrt{T})\) in global optimality have been achieved, for both intrinsic (exp and log) and consensus-algorithmic approaches, over general manifolds including the Stiefel and Grassmann cases [2102.07091, 2603.17096, 2405.16900].

## 5. Practical Considerations and Applications

Implementation requires careful construction of stochastic Riemannian gradients—generally, by projecting the Euclidean gradient onto the tangent space via the metric, followed by retraction. On matrix manifolds (Stiefel, Grassmann, SPD), explicit closed-form retractions and projections are exploited for computational efficiency [1902.01144].

Minibatch size selection introduces a bias-variance tradeoff: increasing batch size reduces the gradient estimator variance (\(\sim\sigma^2/b\)), resulting in faster convergence per iteration but at higher per-iteration cost. Theory predicts strict monotonic decrease and convexity of the total iteration count as a function of batch size, with a critical optimal batch size that minimizes overall cost [2312.07990].

Empirical work demonstrates robust convergence across statistical manifold learning, decentralized covariance estimation, streaming PCA, robust distributional-robust DNN training, and policy evaluation for reinforcement learning [2010.06097, 2405.16900, 2207.09350, 1111.5280].

## 6. Stochastic Riemannian Optimization in Adversarial and Minimax Settings

Riemannian stochastic gradient descent forms the basis for gradient-based solvers in manifold-constrained minimax problems, where one seeks, for example,
\[
\min_{x\in\mathcal{M}}\, \max_{y\in\mathcal{Y}}\, \mathbb{E}_\xi[ f(x, y; \xi) ].
\]
Dedicated algorithms such as Riemannian stochastic gradient descent ascent (RSGDA) have been developed with rigorous sample complexity bounds. In Geodesically-Nonconvex, Strongly-Concave (GNSC) settings, RSGDA achieves sample complexity \(O(\kappa^4\epsilon^{-4})\) for \(\epsilon\)-stationarity, with further acceleration to \(\widetilde{O}(\kappa^4\epsilon^{-3})\) via momentum-based variance reduction (e.g., STORM-like updates with vector transport) [2010.06097].

Key to the analysis are geometric Lipschitz constants governing regularity of the manifold gradients and the effects of retraction, as well as Lyapunov-type descent arguments adapted to the Riemannian context.

## 7. Diffusion Approximations and Continuous-Time Limits

In the small-step regime, stochastic Riemannian SGD can be rigorously approximated by a diffusion process—a Riemannian stochastic modified flow (RSMF)—with drift and covariance structure determined by the geometry, retraction, and gradient noise. Under appropriate regularity of the retraction and manifold curvature, the weak error of the Euler discretization enjoys an order of \(O(\eta^2)\) [2402.03467]. These insights connect discrete SGD dynamics to well-posed stochastic differential equations on manifolds, clarifying algorithmic scaling and the effect of geometric structures on noise propagation.

Additionally, for infinite-dimensional settings such as Wasserstein space, Riemannian SGD and SVRG flows have been derived and analyzed at the level of Fokker–Planck equations, matching the Euclidean continuous-time rates and clarifying connections to Langevin sampling and geometric MCMC [2401.13530].

---

**References**:
- [1111.5280] Bonnabel, "Stochastic gradient descent on Riemannian manifolds"
- [2312.07990] "Convergence of Riemannian Stochastic Gradient Descent on Hadamard Manifold"
- [1605.07147] "Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds"
- [1802.09128] "Averaging Stochastic Gradient Descent on Riemannian Manifolds"
- [2010.12191] "Escape saddle points faster on manifolds via perturbed Riemannian stochastic recursive gradient"
- [2010.06097] "Gradient Descent Ascent for Minimax Problems on Riemannian Manifolds"
- [2207.09350] "Riemannian Stochastic Gradient Method for Nested Composition Optimization"
- [2402.03467] "Stochastic Modified Flows for Riemannian Stochastic Gradient Descent"
- [2102.07091] "Decentralized Riemannian Gradient Descent on the Stiefel Manifold"
- [2405.16900] "Distributed Riemannian Stochastic Gradient Tracking Algorithm on the Stiefel Manifold"
- [2406.02296] "Learning-Rate-Free Stochastic Optimization over Riemannian Manifolds"
- [1902.01144] "Riemannian adaptive stochastic gradient algorithms on matrix manifolds"
- [2302.00709] "Riemannian Stochastic Approximation for Minimizing Tame Nonsmooth Objective Functions"
- [2401.13530] "Continuous-time Riemannian SGD and SVRG Flows on Wasserstein Probabilistic Space"
- [2109.04289] "Riemannian Stochastic Hybrid Gradient Algorithm for Nonconvex Optimization"
- [2603.17096] "Intrinsic Decentralized Stochastic Riemannian Optimization on Manifolds with Bounded Sectional Curvature"
- [1702.05594] "Riemannian stochastic variance reduced gradient algorithm with retraction and vector transport"

Source: https://www.emergentmind.com/topics/stochastic-gradient-descent-on-riemannian-manifolds