---
title: Riemannian Preconditioning in Manifold Optimization
url: https://www.emergentmind.com/topics/riemannian-preconditioning
type: topic
---

# Riemannian Preconditioning in Manifold Optimization

Riemannian preconditioning is the systematic design of a Riemannian metric in manifold optimization frameworks to accelerate convergence by embedding local second-order information—typically block-diagonal or structure-aware Hessian approximations—into the manifold geometry. By replacing the standard Euclidean or canonical metric with a task-adapted metric that mimics curvature, Riemannian preconditioning equips gradient-based algorithms with effective scaling properties and Newton-like behavior at significantly reduced computational cost. The approach has been extensively developed for tensor completion, low-rank matrix/tensor recovery, optimization on product quotients and the Stiefel manifold, as well as ground states in infinite-dimensional PDE-constrained systems.

## 1. Principles and Motivations: Preconditioning Through Metric Design

The role of Riemannian preconditioning is to improve optimization within a constraint manifold (often a product or quotient space) by selecting a metric that makes first-order methods behave more like Newton-type algorithms. The preconditioning metric is constructed using curvature information from the objective, such as block-diagonal Hessian approximations, Gram matrices, or left/right covariance factors. This connects the Riemannian gradient step to the solution of an associated Sequential Quadratic Programming (SQP) subproblem, so that steepest descent becomes a quasi-Newton method in the appropriate limit [1405.6055]. In practice, the metric is chosen to closely approximate the local Hessian's dominant blocks while remaining easily invertible to maintain computational efficiency [2101.11108].

Within equality-constrained or quotient frameworks, the metric can be defined as
\[ g_x(\xi,\eta) = \langle\xi, D^2\mathcal L(x, \lambda_x)[\eta]\rangle, \]
where \(\mathcal L\) is the Lagrangian and \(\lambda_x\) the multiplier at \(x\). In product-manifold settings, the metric takes block-diagonal form to capture the structure of factorized models [2306.08873].

## 2. Metric Construction: Block-Diagonal and Structure-Aware Schemes

A central feature of Riemannian preconditioning is the design of the metric based on approximate Hessians. The commonly adopted strategy is to use a block-diagonal metric where each block corresponds to a natural variable (factor) or tangent space component.

For example, in low-rank tensor completion via polyadic decomposition [2101.11108], the product manifold is \(\mathcal M = \mathbb{R}^{n_1 \times r} \times \cdots \times \mathbb{R}^{n_d \times r}\), and the metric is
\[
\langle \xi, \eta \rangle_X = \sum_{i=1}^d \operatorname{tr}[\xi_i^\top G_i(X) \eta_i]
\]
with
\[
G_i(X) = (X^{(-i)})^\top X^{(-i)} + \delta I_r,
\]
where \(X^{(-i)}\) is the Khatri–Rao product of all factor matrices except \(X_i\). This captures the curvature in the \(X_i\)-variable via Gram matrices. The small shift \(\delta\) ensures positive definiteness, robustifying the method against rank-deficient blocks.

More generally, for a product manifold \(M = M_1 \times \cdots \times M_K\), the preconditioned metric takes the form
\[
g_x(\xi, \eta) = \sum_{k=1}^K \langle \xi_k, P_k(x)[\eta_k] \rangle,
\]
where \(P_k(x)\) is designed to approximate the block-diagonal Hessian block for block \(k\) [2306.08873].

## 3. Algorithmic Realizations: Gradient, Conjugate Gradient, and Trust-Region

The incorporation of the preconditioning metric affects all geometric algorithmic primitives:
- **Riemannian gradient:** For block-diagonal metrics, the Riemannian gradient direction for each block \(i\) is given by
  \[
  (\operatorname{grad} f(X))_i = [\partial_{X_i} f] \, G_i(X)^{-1},
  \]
  ensuring that each component is scaled inversely to its local curvature [2101.11108].

- **Retraction:** When the manifold is a vector space, the retraction is trivial (the identity map); otherwise, standard matrix factorizations (e.g., QR or polar decompositions) are used to ensure feasibility after each step [1902.01635].

- **Vector transport:** For most block-metric or ambient-vector-space settings, the vector transport simplifies to the identity map, reducing per-iteration overhead in first-order methods.

- **Line-search and stepsizes:** Exact line-search or structure-aware Barzilai–Borwein rules can be employed, leveraging closed-form polynomials in least-squares settings [2101.11108].

- **Conjugate gradient and higher-order methods:** Preconditioned metrics also enhance Riemannian nonlinear conjugate gradient and trust-region schemes, retaining Newton-like convergence rates under suitable regularity [1506.02159], [1902.01635].

## 4. Theoretical Guarantees: Conditioning and Convergence Rates

Riemannian preconditioning reduces the condition number of the Riemannian Hessian at stationary points, which directly controls the convergence rate of first-order algorithms. Concretely:
- **Local linear convergence:** Let \(\kappa_g\) denote the condition number of the Riemannian Hessian under metric \(g\) at the minimizer \(x^*\). Then Riemannian gradient descent (RGD) with Armijo or exact line-search satisfies
  \[
  f(x^{(t)}) - f(x^*) \leq \left(1 - \frac{c}{\kappa_g}\right) (f(x^{(t-1)}) - f(x^*))
  \]
  for some constant \(c > 0\) [2306.08873].

- **Łojasiewicz-type global convergence:** In analytic settings, the combination of sufficient decrease, Lipschitz continuous gradients, and a Łojasiewicz gradient inequality yields full convergence to a stationary point with sublinear or linear rates depending on the Łojasiewicz exponent [2101.11108].

- **Preconditioner robustness:** The use of the \(\delta\)-shift and Gram matrices ensures invariance to rank-deficient factors, enabling convergence even when the local block Hessians become nearly singular [2101.11108].

## 5. Practical Applications: Empirical Acceleration and Robustness

The practical efficacy of Riemannian preconditioning is well-documented:
- **Low-rank tensor and matrix completion:** Empirical studies show 4×–15× acceleration (in time and memory) for low-rank tensor completion problems compared to unpreconditioned Riemannian GD/CG and state-of-the-art methods such as geomCG [2101.11108]. The metric construction also yields robust performance under overestimated rank, with recovery quality stable even for \( r \gg r_{\text{true}} \).
  
- **Overparameterization tolerance:** Algorithms with structure-aware preconditioning metrics display flat recovery error with respect to overspecified rank parameters, while unpreconditioned schemes often stagnate or slow down [2101.11108].

- **Tensor decompositions:** Similar metric prescriptions for Tucker and tensor ring decompositions yield order-of-magnitude speedups in convergence and lower sample complexity in phase transitions [1605.08257], [2302.14456].

- **Product manifolds and matrix manifolds:** The block-wise metric strategy extends to canonical correlation analysis, SVD, and generalized Stiefel/Grassmann settings, reducing Hessian condition numbers by one to two orders of magnitude and yielding 5–20× reduction in iteration counts [1902.01635], [2306.08873].

## 6. Generalization and Extension to Other Manifold Structures

The design principles underlying Riemannian preconditioning generalize to a wide class of constrained manifold optimization settings:
- **Recipe:** Identify natural coordinate blocks (e.g., factor matrices or core tensors), approximate or compute block Hessians (typically via Gram matrices), apply positive definiteness shifts, and define the metric block-diagonally. This structural approach is applicable to tensor trains, hierarchical Tucker, bi-factored bilinear models, and matrix manifolds (e.g., fixed-rank or orthogonality-constrained cases) [2101.11108], [2306.08873], [1405.6055].
  
- **Theoretical guarantees:** Under standard regularity (smoothness and positive definiteness of block Hessians on tangent spaces), all Riemannian geometric ingredients and first-order convergence theory extend seamlessly to these generalized settings.

- **Adaptivity:** The choice of metric adapts automatically to local curvature and factor degeneracy (through Gram matrices and rank adaptation), making the preconditioners robust to practical pathologies in large-scale optimization.

## 7. Summary Table: Key Elements of Riemannian Preconditioning

| Component                       | Construction/Role                                                             | Source Papers         |
|----------------------------------|-------------------------------------------------------------------------------|----------------------|
| Metric definition                | Block-diagonal from Hessian blocks, Gram matrices, or structure-aware weights | [2101.11108], [2306.08873], [1405.6055] |
| Riemannian gradient              | Blockwise scaling of Euclidean gradient by local Gram or Hessian inverses     | [2101.11108]         |
| Robustness shift                 | Positive definite shift δI for degenerate/ill-posed blocks                    | [2101.11108], [1605.08257]|
| Local convergence rate           | Condition-number controlled, linear to superlinear                            | [2306.08873]         |
| Product/quotient manifold scope  | Extensible to general product/quotient settings                               | [2306.08873], [1405.6055] |
| Empirical speedup and tolerance  |10× acceleration, stability under overparameterization                         | [2101.11108]         |

Riemannian preconditioning thereby provides a principled, structure-exploiting, and computationally scalable approach to manifold optimization, effectively bridging first-order and second-order methods and enabling robust solution of challenging problems in tensor and matrix recovery, signal processing, and beyond [2101.11108], [2306.08873], [1605.08257].

Source: https://www.emergentmind.com/topics/riemannian-preconditioning