---
title: Product Riemannian Gradient Descent
url: https://www.emergentmind.com/topics/product-riemannian-gradient-descent-prgd-algorithm
type: topic
---

# Product Riemannian Gradient Descent

Product Riemannian Gradient Descent (PRGD) refers to a family of first-order optimization algorithms tailored to smooth constraint manifolds, extending gradient descent mechanisms from Euclidean to Riemannian settings. PRGD variants have been developed for both general non-convex smooth optimization on manifolds via perturbative techniques as well as for specific structured problems, notable among them low-rank tensor completion using the tensor-train (TT) format. Two principal forms are: (i) Perturbed Riemannian Gradient Descent for escaping saddle points and ensuring second-order criticality in non-convex manifold optimization [1906.04321]; and (ii) Preconditioned Riemannian Gradient Descent accelerating convergence for TT-format tensor completion [2501.13385]. Both leverage the manifold geometry and structure through retractions, tangent space manipulations, and strategic use of preconditioning and step alternation.

## 1. Riemannian Optimization Setup and Structural Assumptions

The central aim in Riemannian optimization is the minimization of a smooth function $f : \mathcal{M} \to \mathbb{R}$ where $\mathcal{M}$ is a $d$-dimensional Riemannian manifold representing problem-specific constraints (e.g., orthonormality, rank, positivity). Optimization respects manifold geometry and uses tools such as the Riemannian gradient ($\operatorname{grad} f(x) \in T_x\mathcal{M}$) and Riemannian Hessian ($\operatorname{Hess} f(x)$ acting on the tangent space). For well-posed results, standard regularity conditions are assumed:
- Existence of a lower bound for $f$ over $\mathcal{M}$.
- Lipschitz continuity for the pullback-gradient $\nabla \hat f_x(s)$ and pullback-Hessian $\nabla^2 \hat f_x(s)$ over balls in the tangent space $T_x\mathcal{M}$.
- Retraction mapping denoted $\operatorname{Retr}_x(s): T_x\mathcal{M} \to \mathcal{M}$, which must be smooth, satisfy $\operatorname{Retr}_x(0) = x$, and possess an appropriate second-order error term.

In TT-format tensor completion, $\mathcal{M}_r^{\operatorname{tt}}$ is the smooth embedded manifold of tensors of fixed TT-rank $r$, parametrized by cores $G_k \in \mathbb{R}^{r_{k-1} \times d_k \times r_k}$ [2501.13385].

## 2. PRGD Algorithmic Structure and Pseudocode

Two representative PRGD variants have been developed:

### A. Perturbed Riemannian Gradient Descent (Saddle Point Escape) [1906.04321]
PRGD alternates between two types of updates:
- **Manifold step**: If $\|\operatorname{grad} f(x_t)\| > \epsilon$, a Riemannian gradient descent step is taken, retracting back to the manifold.
- **Tangent-space perturbed step**: If $\|\operatorname{grad} f(x_t)\| \leq \epsilon$, a random perturbation within a ball in the tangent space is injected, followed by several gradient steps in $T_{x_t}\mathcal{M}$, then retracts.

Pseudocode for PRGD proceeds as:

```
t ← 0
while t ≤ T do
    if ‖grad f(x_t)‖ > ε then
        x_{t+1} ← TangentialGDstep(x_t, 0, η, b, 1)
        t ← t+1
    else
        sample ξ ∼ Uniform(B_{x_t}(0, r)) in T_{x_t}𝓜
        s₀ ← η ξ
        x_{t+𝒯} ← TangentialGDstep(x_t, s₀, η, b, 𝒯)
        t ← t+𝒯
    end if
end while
```
The `TangentialGDstep` subroutine performs projected gradient descent steps in the tangent space with appropriate step size and retraction.

### B. Preconditioned Riemannian Gradient Descent (TT-format Tensor Completion) [2501.13385]
PRGD here adapts the inner product (metric) dynamically via data-dependent weighting and projects descent direction appropriately:
- At each iteration, ambient gradient $G_\ell = P_\Omega(T_\ell - T^*)$ (sampling operator) is computed.
- Mode-wise weights are constructed to approximate the local geometry.
- A tangent-space direction is determined via projection in the weighted metric, and the step is followed by retraction back to $\mathcal{M}_{r}^{\operatorname{tt}}$ via TT-SVD.
- The update reads:
  $$
  T_{\ell+1}
    = \operatorname{Retr}_{T_\ell}\left(T_\ell - \eta_\ell W_\ell^{-1} \operatorname{grad} f(T_\ell)\right)
  $$

## 3. Retraction and Manifold Operators

Retraction, $\operatorname{Retr}_x(s)$, maps a vector in the tangent space at $x$ back to the manifold, satisfying $\operatorname{Retr}_x(0) = x$ and differential $D\operatorname{Retr}_x(0)$ equals the identity. In proof arguments, the pullback function $\hat f_x(s) = f(\operatorname{Retr}_x(s))$ is used for local analysis.

In TT-format tensor completion, retraction is realized via quasi-optimal TT-SVD, which computes the closest tensor with target TT-rank [2501.13385].

Projection from the ambient space to the tangent space is central. For TT-manifolds, orthogonal projectors $P_T$ and its weighted version $\widetilde P_{T_\ell}$ are used, enabling efficient calculation of Riemannian gradients.

## 4. Convergence Properties and Complexity

### A. Non-Convex Manifold Optimization [1906.04321]
Under assumptions A1–A3, for any twice differentiable $f$, and retraction $\operatorname{Retr}$ meeting mild second-order error constraints, PRGD guarantees approximate second-order criticality: visits to points $x$ with $\|\operatorname{grad} f(x)\| \leq \epsilon$ and $\lambda_{\min}(\operatorname{Hess} f(x)) \geq -\sqrt{\rho \epsilon}$ within
$$
O\left((\log d)^4 / \epsilon^2\right)
$$
gradient queries, matching the Euclidean PGD complexity and retaining low dimensionality dependence.

The key proof steps show:
- RGD steps decrease $f$ deterministically while far from saddle points.
- Perturbed tangent-space steps probabilistically escape saddle regions, leveraging a volume argument and randomization to ensure progress with high probability.
- Mild retraction assumptions suffice; explicit curvature bounds of $\mathcal{M}$ are not required.

### B. TT-format Tensor Completion [2501.13385]
PRGD achieves linear convergence rates (geometric decay) independent of the condition number under tensor incoherence and spikiness assumptions, when started from a sufficiently good initialization and with adequate sample complexity. The contraction factor (e.g. $0.3574$ per iteration) is explicit. Per-step computational complexity aligns with unpreconditioned RGD: gradient formation in $O(|\Omega|)$, preconditioning overhead $O(m|\Omega|)$, and TT-SVD retraction as the dominant cost $O(m d_{\max} r_{\max}^3)$.

## 5. Alternating Step Mechanics and Tangent-Space Perturbations

The PRGD design separates manifold navigation from tangent-space exploration:
- **Manifold steps** ($\operatorname{Retr}_x$): Used when sufficiently far from critical points, guarantee deterministic decrease in $f$.
- **Tangent-space perturbed steps**: Exploit linear structure in $T_x\mathcal{M}$ near potential saddle points, enable isotropic perturbations and direct adaptation of Euclidean PGD analysis.
- Switching between these modes ensures transfer of Euclidean complexity guarantees to the manifold setting, obviating the need for costly Hessian computations.

## 6. Hyperparameter Selection and Example Applications

All PRGD parameters (step size $\eta$, perturbation radius $r$, inner-loop length $\mathcal{T}$) are functions of problem-specific constants: Lipschitz constants from the pullback-gradient and Hessian ($\ell$, $\rho$), user tolerance $\epsilon$, probability parameter $\delta$, and radius $b$. These may require empirical tuning or estimation via adaptive procedures.

**Example: PCA as Manifold Optimization** [1906.04321]
On the sphere $\mathcal{M} = S^{d-1}$ with cost $f(x) = -\frac{1}{2} x^\top A x$, the standard projection retraction yields $L = 2.5 \|A\|$, $\rho = 9 \|A\|$, so $\ell$ can be taken on the order of $\|A\|$. PRGD thus admits concrete instantiation and complexity scaling in high-dimensional data analysis.

**Example: TT-format Tensor Completion** [2501.13385]
Applications include:
- Synthetic tensor completion: PRGD reduces iteration and CPU time by up to two orders of magnitude compared to RGD.
- Hyperspectral image completion: Achieves improved PSNR and up to tenfold convergence acceleration.
- Quantum state tomography: Demonstrates 5–10× reduction in iterations and substantial CPU-time savings, dependent on step-size schedule.

## 7. Comparison with Standard RGD and Euclidean PGD

The following table summarizes complexity and requirements:

| Algorithm         | First-order queries           | Second-order queries               | Dimensionality dependence | Retraction assumptions             |
|-------------------|-----------------------------|------------------------------------|--------------------------|------------------------------------|
| Riemannian GD     | $O(1/\epsilon^2)$           | – (no escape guarantee)            | –                        | Any retraction (A2)                |
| Euclidean PGD     | $O((\log d)^4/\epsilon^2)$  | $O((\log d)^4/\epsilon^2)$         | $(\log d)^4$             | Not applicable                     |
| PRGD              | $O((\log d)^4/\epsilon^2)$  | $O((\log d)^4/\epsilon^2)$         | $(\log d)^4$             | Retraction satisfying A2–A4         |

All algorithms operate with first-order information only. PRGD matches the best-known complexity for saddle point escape in unconstrained Euclidean settings while handling manifold constraints with mild additional conditions on the retraction (and no explicit manifold curvature bounds) [1906.04321], [2501.13385].

## References

- C. Criscitiello & N. Boumal, "Efficiently escaping saddle points on manifolds" [1906.04321]
- S. Jin et al., "How to Escape Saddle Points Efficiently," ICML 2017 & 2019.
- P. Absil, R. Mahony, R. Sepulchre, "Optimization Algorithms on Matrix Manifolds," Princeton, 2008.
- Z. Zhang et al., "Fast and Provable Tensor-Train Format Tensor Completion via Preconditioned Riemannian Gradient Descent" [2501.13385]

Source: https://www.emergentmind.com/topics/product-riemannian-gradient-descent-prgd-algorithm