---
title: Riemannian/Projected Gradient Descent
url: https://www.emergentmind.com/topics/riemannian-projected-gradient-descent
type: topic
---

# Riemannian/Projected Gradient Descent

Riemannian and projected gradient descent (GD) are fundamental algorithmic paradigms for optimization over manifolds, submanifolds, and nonlinear constraint sets. Their core principle is to generalize classical steepest-descent to spaces with nontrivial geometry by replacing Euclidean operations with their Riemannian or geometric analogues, ensuring that the optimization trajectory remains on the feasible set and that the update directions account for curvature and constraint-induced structure.

## 1. Geometric Foundations and Algorithmic Variants

Riemannian gradient descent (RGD) operates on smooth manifolds $\mathcal{M}$ endowed with a Riemannian metric $\langle\cdot,\cdot\rangle_x$ on each tangent space $T_x \mathcal{M}$. At each iteration, the algorithm computes the Riemannian gradient $\operatorname{grad} f(x) \in T_x \mathcal{M}$, which satisfies
\[
\langle \operatorname{grad} f(x), v \rangle_x = Df(x)[v],\quad \forall v \in T_x \mathcal{M}.
\]
The basic geometric update is the exponential map (or an efficient retraction $\mathcal{R}_x$):
\[
x_{k+1} = \operatorname{Exp}_x(-\eta_k \operatorname{grad} f(x_k)), \quad\text{or}\quad x_{k+1} = \mathcal{R}_x(-\eta_k \operatorname{grad} f(x_k)).
\]
Projected gradient descent (PGD) and Projected Riemannian Gradient Descent (PRGD) generalize this scheme to embedded submanifolds or constraint sets, relying on orthogonal projections or nearest-point projections after a tangent-space step. If $M \subset \mathbb{R}^n$ is a compact $C^2$ submanifold, the projection is
\[
\operatorname{Proj}_M(u) = \arg\min_{y \in M} \|y - u\|,
\]
and the projected update is
\[
x_{k+1} = \operatorname{Proj}_M(x_k - \eta_k \nabla f(x_k)).
\]
For submanifolds, the Riemannian gradient may be implemented as the metric projection of the Euclidean gradient onto the tangent space $T_xM$,
\[
\operatorname{grad} f(x) = P_{T_xM}(\nabla f(x)),
\]
with subsequent manifold retraction or nearest-point projection.

### Table: Core Variants

| Method         | Feasible Set        | Descent Step                              | Return to Manifold          |
| -------------- | ------------------ | ----------------------------------------- | -------------------------- |
| RGD            | Riemannian $\mathcal{M}$ | $x_{k+1} = \operatorname{Exp}_{x_k}(-\eta_k \operatorname{grad} f(x_k))$   | Exponential / Retraction   |
| PGD            | $\mathbb{R}^n$, $C\subset\mathbb{R}^n$ (convex set or submanifold) | $x_{k+1} = \operatorname{Proj}_C(x_k - \eta_k \nabla f(x_k))$    | Metric projection |
| PRGD           | Submanifold $M\subset\R^n$   | $x_{k+1} = \operatorname{Proj}_M(x_k - \eta_k \nabla f(x_k))$     | Manifold projection       |

## 2. Differential Structure, Projections, and Natural Metrics

A unifying principle is the replacement of Euclidean notions with those induced by the manifold's metric or a suitable pull-back metric, as in natural gradient descent (NGD). For parameterized constraints or ansatzes of the form $L(\theta) = \bar L(f(\theta))$ with $\theta \in X \subseteq \mathbb{R}^n$, $f: X \to Y$, one can induce a Riemannian metric on $X$ by pull-back of a reference metric $G_Y$ on $Y$:
\[
(G_X^f)_{ij}(\theta) = \frac{\partial y^\alpha}{\partial \theta^i} [G_Y(y)]_{\alpha\beta} \frac{\partial y^\beta}{\partial \theta^j}.
\]
The generalized NGD update is then
\[
\theta_{t+1} = \theta_t - \eta (G_X^f(\theta_t))^{-1}\nabla_\theta L(\theta_t).
\]
In constrained scenarios, projection onto the admissible tangent space arises either via explicit orthogonal projection or implicitly via the induced metric and the Jacobian of the embedding $f_*$. If the full ambient tangent space is not spanned, the projected update in $Y$ takes the form
\[
d y \propto -P_{\operatorname{im} f_*} G_Y^{-1}\nabla_y \bar L(y)
\]
with $P_{\operatorname{im} f_*}$ denoting the orthogonal projection onto the image subspace, ensuring that the iterate remains on the submanifold $f(X)\subseteq Y$ [2210.02764].

## 3. Convergence Properties and Algorithmic Guarantees

### Local Linear and Sublinear Rates

Under standard Riemannian smoothness and convexity assumptions (e.g., geodesic convexity, sectional curvature bounds), RGD and its projected variants enjoy well-characterized convergence rates:
- For $L$-smooth, $\mu$-strongly geodesically convex functions, RGD with constant step-size $\eta=1/L$ achieves linear convergence:
  \[
  f(x_k) - f(x^*) \leq (1-c)^k,
  \]
  where $c = \Theta(\mu/L)$ and the rates incorporate curvature-dependent constants such as $\zeta_{r}$ [2403.10429].
- In the absence of strong convexity, sublinear $O(1/k)$ rates in function value gap are attainable.
- In nonconvex settings, under $L$-smoothness and the Kurdyka–Łojasiewicz property, global convergence of RGD (and IRGD) can be established, with convergence rates depending on the local KL exponent [2409.11181].

Projected Riemannian GD and decentralized algorithms on compact submanifolds attain comparable rates, with $O(1/\sqrt{K})$ and $O(1/K)$ rates to stationarity for standard and gradient-tracking schemes, respectively [2304.08241]. On Hadamard manifolds with nonpositive curvature, decentralized projected RGD achieves minimax-optimal dynamic regret of order $O(\sqrt{T(1+P_T)}/\sqrt{1-\sigma_2(W)})$, where $P_T$ is path variation and $\sigma_2$ the spectral gap of the consensus matrix [2410.05128].

### Trade-offs and Implementation

The explicit choice of metric (e.g., Fisher, Sobolev, pull-back via function-space structure) directly impacts both convergence rate and numerical stability, often dramatically accelerating optimization in ill-conditioned, constraint-rich, or physically structured models [2210.02764, 2512.11339]. Step-size schedules can be constant, diminishing, or adaptive, and line searches or Armijo-type rules are frequently employed to guarantee monotonic descent or enforce stability constraints [2311.02031].

## 4. Practical Implementations and Applications

Riemannian and projected GD frameworks have wide-ranging applications:

- **Quantum information**: RGD on the Stiefel or unitary group manifolds is used in quantum process tomography and ground-state preparation, enabling physical trace-preservation and unitarity constraints to be exactly enforced, with Riemannian retractions (e.g., Cayley transforms, matrix exponentials) maintaining feasibility at each step [2404.18840, 2512.13401].
- **Matrix manifolds**: Low-rank matrix optimization is treated by RGD on fixed-rank and partial isometry manifolds, with tangent-space projections and SVD-based retractions yielding local linear or accelerated rates [2606.02328, 2211.16236].
- **Variational ansätze and physical constraints**: Pull-back metrics (Sobolev, energy-based) are used to precondition optimization in neural variational Monte Carlo and variational quantum eigensolvers [2512.11339].
- **Decentralized optimization**: DPRGD and its variants are implementable in multi-agent architectures, with theoretical variance reduction and dynamic regret bounds [2304.08241, 2410.05128].

## 5. Extensions: Inexactness, Adaptivity, and Accelerated Methods

### Inexact and Stochastic RGD

In practical high-dimensional or derivative-free scenarios, only approximate or stochastic gradients are available. The IRGD algorithm replaces the exact gradient with $g_k$ satisfying either absolute or relative inexactness:
\[
\|g_k - \operatorname{grad} f(x_k)\| \leq \epsilon_k \quad \text{(absolute)}, \qquad \|g_k - \operatorname{grad} f(x_k)\| \leq \nu\|\operatorname{grad} f(x_k)\| \quad \text{(relative)},
\]
with appropriate step-size control ensuring convergence to stationary points or, under KL, to a limit point [2409.11181]. RSAM and Riemannian extragradient methods are encompassed as special cases.

### Adaptive Step-Size

Adaptive RGD modulates the step-size $\alpha_k$ based on local curvature and observed gradient variation, achieving competitive or superior per-iteration progress relative to Armijo backtracking, particularly on nonnegatively curved manifolds [2504.16724].

### Acceleration and Line-Search

Manifold analogues of Nesterov acceleration (e.g., NARG with orthographic retraction) achieve optimal local linear rates given knowledge of spectrum or via adaptive-restart schemes, matching or outperforming Euclidean first-order methods on low-rank matrix problems [2211.16236].

## 6. Consensus, Minimax Problems, and Product Manifold Optimization

Alternating projected/Riemannian strategies are natural for nonconvex—concave minimax or saddle-point problems on product manifolds. Algorithms such as ARPGDA combine Riemannian descent in manifold variables (e.g., Stiefel) with Euclidean projected ascent in convex variables (e.g., simplex), achieving $\mathcal{O}(\epsilon^{-3})$ rates to stationarity, with rigorous potential-based convergence analysis [2210.16076]. For geodesic nonconvex—strongly-concave minmax problems, deterministic and stochastic Riemannian GD–ascent schemes reach $O(\kappa^2\epsilon^{-2})$ and $O(\kappa^4\epsilon^{-4})$ stationarity sample complexity, improved by STORM-style variance reduction [2010.06097].

## 7. Illustrative Examples

- **SPD matrices**: On the manifold $P(n)$ of Hermitian positive-definite matrices, the affine-invariant RGD follows the geodesic-exponential update, with the Riemannian gradient coinciding with the matrix-congruence gradient and per-iteration cost $O(n^3)$ [1904.02844].
- **Hyperbolic space**: In the hyperboloid model of $\mathbb{H}^n$, gradient descent is implemented by projection of the ambient Minkowski gradient, followed by an exact exponential-map update along geodesics, outperforming Poincaré-ball retraction methods [1805.08207].

## References

Selected papers providing key technical details and convergence results:

- "Generalization to the Natural Gradient Descent" [2210.02764]
- "Application of gradient descent algorithms based on geodesic distances" [1904.02844]
- "Decentralized projected Riemannian gradient method for smooth optimization on compact submanifolds" [2304.08241]
- "Inexact Riemannian Gradient Descent Method for Nonconvex Optimization" [2409.11181]
- "Fast gradient method for Low-Rank Matrix Estimation" [2211.16236]
- "Decentralized Online Riemannian Optimization with Dynamic Environments" [2410.05128]
- "Riemannian gradient descent for Hartree-Fock theory" [2603.15870]
- "Adaptive Gradient Descent on Riemannian Manifolds with Nonnegative Curvature" [2504.16724]
- "Projected Sobolev Natural Gradient Descent for Neural Variational Monte Carlo Solution of the Gross-Pitaevskii Equation" [2512.11339]
- "An Efficient Alternating Riemannian/Projected Gradient Descent Ascent Algorithm for Fair Principal Component Analysis" [2210.16076]
- "IRKA is a Riemannian Gradient Descent Method" [2311.02031]
- "Riemannian Stein Variational Gradient Descent for Bayesian Inference" [1711.11216]
- "Riemannian Gradient Descent for Low-Rank Architectures" [2606.02328]
- "Riemannian gradient descent-based quantum algorithms for ground state preparation with guarantees" [2512.13401]
- "Convergence and Trade-Offs in Riemannian Gradient Descent and Riemannian Proximal Point" [2403.10429]
- "Gradient Descent Ascent for Minimax Problems on Riemannian Manifolds" [2010.06097]
- "Fast Quantum Process Tomography via Riemannian Gradient Descent" [2404.18840]

Source: https://www.emergentmind.com/topics/riemannian-projected-gradient-descent