---
title: Riemannian Gradient Descent
url: https://www.emergentmind.com/topics/riemannian-gradient-descent
type: topic
---

# Riemannian Gradient Descent

Riemannian gradient descent is a class of first-order optimization methods that generalize standard gradient descent to smooth manifolds endowed with Riemannian structure, enabling unconstrained or constrained optimization where the feasible set forms a manifold rather than a linear space. By employing the geometry of the manifold—specifically, the tangent spaces, the Riemannian metric, and retraction or exponential maps—these methods define update rules intrinsically compatible with manifold constraints, and are essential across manifold machine learning, inverse problems, robust statistics, geometric deep learning, quantum computation, and control.

## 1. Basic Formulation and Principles

Let $(M,g)$ be a smooth Riemannian manifold with metric $g$ and tangent bundle $TM$. For a differentiable $f:M\to\mathbb{R}$, the Riemannian gradient $\grad f(x)\in T_xM$ at $x\in M$ is defined via
$$
D f(x)[v] = \langle \grad f(x), v\rangle_x, \quad \forall v\in T_xM.
$$
A classical Riemannian gradient descent (RGD) iteration with step size $\eta_k>0$ takes the form
$$
x_{k+1} = \Retr_{x_k}(-\eta_k \grad f(x_k)),
$$
where $\Retr_{x_k}:T_{x_k}M\to M$ is a retraction map, usually chosen as the Riemannian exponential map when computationally feasible. In practice, retractions are selected for computational efficiency or numerical stability and approximate the exponential map to first (or higher) order.

The update is analogous to Euclidean GD but replaces vector addition with tangent vector moves mapped to the manifold by a geometric mechanism. The step $\Retr_{x_k}(-\eta_k \grad f(x_k))$ is guaranteed to lie on $M$, preserving any manifold constraints such as orthonormality, fixed rank, or determinant, and almost always outperforms Euclidean projection in poorly conditioned or highly curved settings [2403.10429].

## 2. Geometric Structures and Computational Ingredients

The efficacy of Riemannian gradient descent rests on several geometric components:

- **Tangent Space and Projections**: Each $x\in M$ has a tangent space $T_x M$; the Riemannian gradient is an intrinsic vector field and typically constructed by projecting (if $M$ is embedded) the Euclidean gradient onto $T_x M$ [2305.02543, 2606.02328].
- **Riemannian Metric**: The inner product $\langle \cdot,\cdot\rangle_x$ defines gradient directions and step sizes. For embedded submanifolds, the metric is usually induced from the ambient Euclidean or Hermitian space, but structure-adapted metrics may offer superior conditioning or preconditioning properties [2305.02543].
- **Retraction and Exponential Map**: Retraction $\Retr_x$ is a smooth local map satisfying $\Retr_x(0)=x$ and $D\Retr_x(0) = \operatorname{Id}$; common choices are the exact exponential map, QR/polar-based retractions for Stiefel/Grassmannian, SVD truncation for low-rank matrix manifolds, or rowwise normalizations in product-of-spheres models [1805.08207, 2606.02328, 2403.11726].
- **Parallel Transport and Vector Transport**: For non-Euclidean manifolds, it is sometimes necessary to compare tangent vectors at different points using parallel or isometric vector transport, especially in momentum or variance-reduced variants [2409.11181].

These ingredients yield update rules that fully respect both the manifold structure and the problem's symmetries and invariances.

## 3. Convergence Theory, Rates, and Curvature Dependence

Theoretical properties of Riemannian gradient descent reflect a close analogy to Euclidean optimization, with modifications incorporating curvature and geodesic convexity:

- **Convexity and Smoothness**: Geodesic (or $g$-) convexity generalizes Euclidean convexity; $f$ is $g$-convex if it is convex along all geodesics. $L$-smoothness and $\mu$-strong convexity are defined via geodesic distances [2403.10429, 2504.16724].
- **Convergence Rates**: Sublinear rates $O(1/k)$ for $L$-smooth $g$-convex $f$, and linear rates $O(\exp(-k\mu/L))$ for strongly $g$-convex $f$, hold up to curvature-dependent scaling. The convergence rates degrade gracefully via geometric constants (e.g., $\zeta_R = O(1 + R\sqrt{-\kappa_{\min}})$ for sectional curvature lower-bounded by $\kappa_{\min}$), and are provably robust when the manifold has bounded (possibly nonzero) curvature [2403.10429].
- **Iterate Boundedness**: For provable guarantees, one must control that iterates remain in a convex ball around the minimizer; curvature-induced constants inflate this radius compared to the Euclidean case [2403.10429].
- **Adaptive and Inexact Variants**: Adaptive step-size selection can be based on local Lipschitz estimates computed from parallel transports along geodesics, allowing larger steps in regions of low curvature or smoothness [2504.16724]. Inexact Riemannian gradient methods with controlled absolute or relative errors in the directional vector preserve stationarity and convergence guarantees under mild summability or boundedness conditions [2409.11181].
- **Stochastic and Minimax Settings**: Riemannian stochastic gradient descent (RSGD) and its convergence properties—e.g., variance reduction and weak-error SDE approximations—have been established (e.g. on Hadamard manifolds or for minimax optimization) with batch-size/variance trade-offs structurally homologous to Euclidean results, but with curvature- and geometry-dependent constants [2312.07990, 2402.03467, 2010.06097].

## 4. Algorithmic Forms, Enhancements, and Representative Examples

Many algorithmic instantiations of Riemannian GD are available, tailored to specific manifold models and problem classes:

| Manifold / Structure              | Retraction or Update Rule                 | Application Domain                             |
|-----------------------------------|------------------------------------------|------------------------------------------------|
| Stiefel/Grassmann $(X^T X = I)$  | QR/polar, Exp map                        | PCA, quantum chemistry [2603.15870], statistics |
| Fixed-rank / partial isometry     | SVD truncation, QR/sphere projection     | Deep learning QKV [2606.02328], low-rank recovery [2305.02543] |
| Positive definite Hermitian       | $X^{1/2} \exp(X^{-1/2}\xi X^{-1/2}) X^{1/2}$ | Covariance averaging, matrix control [1904.02844] |
| Hyperbolic space $H^n$            | Ambient projection + hyperboloid Exp      | Barycenters, hierarchical representations [1805.08207] |
| Product-of-spheres $S^2\times\cdots$ | Row normalization                        | Area-preserving mapping, geom-registration [2403.11726] |
| Rational transfer functions       | Orthographic (subspace) retraction        | Model order reduction (IRKA) [2311.02031]       |

Optimizations may leverage momentum, preconditioning [2305.02543], adaptive step selection [2504.16724], or variance reduction for stochastic settings [2010.06097]. When the ambient dimension or curvature is high, randomized subspace approximations or quasi-Riemannian projections (e.g., one-mode tangent projections for tensors) provide computational tractability without sacrificing geometric fidelity [2401.15925, 2512.13401].

### Selected Algorithmic Themes

- **Preconditioned Riemannian GD**: Diagonal or geometric preconditioners are constructed by local adaptation to the norm or energy of the gradient, yielding linear convergence under restricted isometry (matrix recovery) or dramatically improving wall-clock performance at large scale [2305.02543].
- **Stochastic RGDs and Riemannian SDEs**: RSGD is weakly approximated by deterministic geodesic flows at $O(\eta)$ and by a second-order diffusion (Riemannian stochastic modified flow, RSMF) at $O(\eta^2)$, capturing fluctuations of stochastic or minibatch gradients [2402.03467].
- **Manifold-specific curvature exploitation**: In negative curvature (Hadamard) manifolds, strict convexity of squared distance yields unique minimizers and strong convergence for averaging (e.g., Fréchet mean) [2312.07990, 1805.08207]. Adaptive methods in nonnegative curvature enjoy global O$(1/\sqrt{k})$ best-iterate rates even without precise knowledge of global smoothness [2504.16724].
- **Quantum and Infinite-Dimensional Settings**: Gradient flows on quantum state manifolds employ group-theoretic projections and unitary retractions; infinite-dimensional Hartree–Fock problems are solved via Stiefel manifold optimization in Sobolev space with physically-motivated preconditioning [2603.15870, 2512.13401].

## 5. Application Domains and Empirical Results

Riemannian gradient descent is used in a broad array of high-impact domains:

- **Low-rank Matrix and Tensor Recovery**: Accurate and robust estimation in signal processing, imaging, and completion, with global-linear or nearly dimension-free convergence from random initialization under weak isometry conditions [2012.15467, 2305.02543, 2401.15925].
- **Deep Learning and Robust Optimization**: Training of neural networks with orthogonality, low-rank, or parameter-sharing constraints, where RGD variants offer improved geometric fidelity and adversarial robustness compared to projected Euclidean methods [2606.02328, 2010.06097].
- **Matrix Manifolds in Control and Model Reduction**: Optimization over $\mathcal{H}_2$-manifolds or positive definite matrices for control, reduced-order modeling, and system identification [2311.02031, 1904.02844].
- **Manifold Averaging and Statistics**: Computation of Karcher or Fréchet means in non-Euclidean geometry (e.g., SPD, hyperbolic), with unique minimizers and globally convergent RGD schemes [1904.02844, 1805.08207].
- **Quantum Algorithms**: Ground-state preparation, where the structure-exploiting RGD delivers favorable scaling with problem size and allows scalable approximations via random subspaces [2512.13401].
- **Computational Anatomy and Differential Geometry**: Spherical area-preserving parameterizations and brain-surface registration, leveraging power-manifold RGD and guaranteed global convergence [2403.11726].

Empirically, RGD methods consistently outperform naive projection-based approaches, especially in high curvature or constraint-dense settings, and exhibit improved sample and iteration complexity in adversarial, distributionally robust, and high-dimensional problems [2010.06097, 2606.02328]. Preconditioned and adaptive RGD are orders of magnitude faster in practical large-scale inference and machine learning [2305.02543].

## 6. Extensions: Minimax, Stochastic, and Inexact Models

Beyond unconstrained minimization, Riemannian GD has been adapted to:

- **Minimax Optimization on Manifolds**: For problems $\min_{x\in\mathcal{M}}\max_{y\in\mathcal{Y}} f(x,y)$ where $f$ may be geodesically nonconvex in $x$ but strongly concave in $y$, Riemannian Gradient Descent Ascent (RGDA) and its stochastic and accelerated variants achieve sample complexities matching Euclidean GDA up to curvature-dependent condition numbers [2010.06097]. Momentum and variance-reduced techniques—such as STORM—yield accelerated rates $\tilde{O}(\kappa^4 \epsilon^{-3})$ for $\epsilon$-stationarity.
- **Stochastic and Batch-Size Trade-Offs**: On Hadamard manifolds, the iteration and sample complexities trade off with batch size similarly to Euclidean SGD, but curvature and unique geodesicity are key to establishing convexity and rate bounds [2312.07990].
- **Inexact and Extragradient Methods**: Riemannian inexact GD methods control for gradient inaccuracy via normed balls (C1) or cones (C2) around the true gradient, preserving convergence under KL conditions and supporting sharpness-aware minimization and extragradient variants [2409.11181]. Numerical evidence suggests that controlled inexactness does not degrade convergence in common machine learning models.

## 7. Representative Pseudocode and Workflow

A unified pseudocode pattern encapsulates most Riemannian GD algorithms [2403.10429, 2606.02328, 2305.02543, 1805.08207]:

```python
# Example: generic Riemannian gradient descent iteration
x_k = initial_point_on_manifold()
while not converged:
    v_k = compute_Riemannian_gradient(f, x_k)         # project ambient gradient to T_{x_k}M
    t_k = choose_step_size(v_k, x_k)                  # fixed, adaptive, or line search
    x_kplus1 = retraction(x_k, -t_k * v_k)            # e.g., Exp map, SVD, QR, normalization
    x_k = x_kplus1
```

Variants for specific manifolds substitute in efficient geometry-specific formulas for computing gradients, projections, and retraction, and may incorporate stochasticity, adaptive step schemes, or preconditioning.

## References

- "Convergence and Trade-Offs in Riemannian Gradient Descent and Riemannian Proximal Point" [2403.10429]
- "Riemannian Gradient Descent for Low-Rank Architectures" [2606.02328]
- "A Preconditioned Riemannian Gradient Descent Algorithm for Low-Rank Matrix Recovery" [2305.02543]
- "Gradient descent in hyperbolic space" [1805.08207]
- "Adaptive Gradient Descent on Riemannian Manifolds with Nonnegative Curvature" [2504.16724]
- "Inexact Riemannian Gradient Descent Method for Nonconvex Optimization" [2409.11181]
- "Riemannian gradient descent for Hartree-Fock theory" [2603.15870]
- "Riemannian gradient descent-based quantum algorithms for ground state preparation with guarantees" [2512.13401]
- "Gradient Descent Ascent for Minimax Problems on Riemannian Manifolds" [2010.06097]
- "Convergence of Riemannian Stochastic Gradient Descent on Hadamard Manifold" [2312.07990]
- "Stochastic Modified Flows for Riemannian Stochastic Gradient Descent" [2402.03467]
- "Application of gradient descent algorithms based on geodesic distances" [1904.02844]
- "IRKA is a Riemannian Gradient Descent Method" [2311.02031]
- "Riemannian gradient descent for spherical area-preserving mappings" [2403.11726]
- "Fast Global Convergence for Low-rank Matrix Recovery via Riemannian Gradient Descent with Random Initialization" [2012.15467]
- "A Single-Mode Quasi Riemannian Gradient Descent Algorithm for Low-Rank Tensor Recovery" [2401.15925]

Source: https://www.emergentmind.com/topics/riemannian-gradient-descent