---
title: Riemannian Online Gradient Descent
url: https://www.emergentmind.com/topics/riemannian-online-gradient-descent
type: topic
---

# Riemannian Online Gradient Descent

Searching arXiv for relevant papers on Riemannian Online Gradient Descent and closely related variants.
Riemannian Online Gradient Descent (ROGD) generalizes online and stochastic first-order methods to curved spaces by replacing Euclidean additive updates with intrinsic geodesic moves on a manifold. Given a Riemannian manifold \(M\), a sequence of per-round losses \(f_t:M\to\mathbb{R}\), and an iterate \(x_t\in M\), the canonical update advances along the geodesic in the negative Riemannian gradient direction,
\[
x_{t+1}=\operatorname{Exp}_{x_t}\!\big(-\eta_t\,\operatorname{grad} f_t(x_t)\big),
\]
where \(\eta_t>0\) is a learning rate and \(\operatorname{Exp}_{x_t}\) is the exponential map at \(x_t\) [1805.08207]. In the literature, this template appears in adversarial online optimization, stochastic approximation, decentralized optimization, optimistic algorithms, and structured manifold models such as hyperbolic spaces, symmetric positive definite manifolds, and fixed-rank tensor-train manifolds [1111.5280] [2312.07990] [2410.05128] [2308.16004] [2509.11236] [2605.04533].

## 1. Geometric setting and first-order structure

ROGD is formulated on a smooth Riemannian manifold \((M,g)\), where each tangent space \(T_xM\) carries an inner product induced by the metric \(g\), and the geodesic distance is denoted \(d(\cdot,\cdot)\). A central class is the Hadamard manifold: complete, simply connected, and with nonpositive sectional curvature. In this setting, unique geodesics connect any two points, the exponential map is a diffeomorphism, the logarithm map \(\operatorname{Log}_x(y)=\operatorname{Exp}_x^{-1}(y)\) is globally well-defined, and \(d^2(\cdot,\cdot)\) is geodesically convex [2410.05128] [2312.07990].

The Riemannian gradient \(\operatorname{grad} f(x)\in T_xM\) is defined by the metric duality relation
\[
\langle \operatorname{grad} f(x),\xi\rangle_x=df_x(\xi),
\]
for all \(\xi\in T_xM\) [1111.5280]. For geodesically convex functions, the first-order inequality takes the intrinsic form
\[
f(y)\ge f(x)+\big\langle \operatorname{grad} f(x),\operatorname{Log}_x(y)\big\rangle,
\]
which replaces the Euclidean affine lower bound [2410.05128] [2308.16004]. In constrained problems one typically assumes a closed, bounded, geodesically convex feasible set \(\mathcal{X}\subset M\), with geodesic projection
\[
\Pi_{\mathcal{X}}(y)=\arg\min_{x\in\mathcal{X}} d^2(x,y).
\]
On Hadamard manifolds this projection is single-valued and nonexpansive [2410.05128] [2509.11236].

Curvature enters the analysis through comparison inequalities. On Hadamard manifolds with sectional curvatures lower bounded by \(\kappa<0\), a recurring factor is
\[
\zeta(\kappa,c)=\sqrt{|\kappa|}\,c\,\coth\big(\sqrt{|\kappa|}\,c\big),
\]
which quantifies curvature-dependent distortion in triangle inequalities and regret constants [2410.05128]. In analyses that allow positive curvature, one instead restricts the feasible set to a strongly convex ball, typically requiring \(D<\pi/(2\sqrt{\kappa_+})\) when the curvature upper bound satisfies \(\kappa_+>0\) [2509.07779].

## 2. Canonical updates and algorithmic forms

The basic intrinsic update uses the exponential map. In stochastic approximation this is written as
\[
x_{k+1}=\operatorname{Exp}_{x_k}\big(-\alpha_k\,\widehat g_k\big),
\]
where \(\widehat g_k\) is an unbiased stochastic tangent gradient estimate and the step sizes satisfy the Robbins–Monro conditions \(\sum_k \alpha_k=\infty\) and \(\sum_k \alpha_k^2<\infty\) [1111.5280]. When exact geodesics are expensive, a retraction \(R_x:T_xM\to M\) can replace \(\operatorname{Exp}_x\), yielding
\[
x_{k+1}=R_{x_k}\big(-\alpha_k\,\widehat g_k\big),
\]
with first-order local agreement \(d(R_x(tv),\operatorname{Exp}_x(tv))=O(t^2)\) [1111.5280]. Retractions are standard on fixed-rank matrix and tensor manifolds, where truncated SVD, TT-rounding, and TT-SVD are natural choices [2605.04533].

Mini-batch Riemannian stochastic gradient descent (RSGD) replaces the stochastic oracle by a batch average. If
\[
\operatorname{grad} f_{B_k}(x_k)=\frac1b\sum_{i=1}^b G_{\xi_{k,i}}(x_k),
\]
then the estimator remains unbiased and its conditional variance scales as \(\sigma^2/b\), so mini-batching reduces variance by a factor \(1/b\) [2312.07990].

Projected online updates combine a local gradient step with projection onto \(\mathcal{X}\). In decentralized projected Riemannian gradient descent, each agent forms the half-step
\[
y_{i,t+1}=\Pi_{\mathcal{X}}\Big(\operatorname{Exp}_{x_{i,t}}\big(-\eta_t\,\operatorname{grad} f_{i,t}(x_{i,t})\big)\Big),
\]
before a consensus phase [2410.05128]. In optimistic online variants, the gradient is augmented by a parallel-transported prediction term. The Riemannian Online Optimistic Gradient Descent update is
\[
x_{t+1}=\operatorname{Exp}_{x_t}\!\Big(-2\eta\,g_t+\eta\,\Gamma_{x_{t-1}}^{x_t}g_{t-1}\Big),
\]
where \(g_t=\nabla f_t(x_t)\) and \(\Gamma_{x_{t-1}}^{x_t}\) denotes parallel transport along the geodesic from \(x_{t-1}\) to \(x_t\) [2308.16004].

A specialized but structurally important variant appears on the fixed-TT-rank manifold \(M_r\) for matrix product operator quantum state tomography. There the online Riemannian gradient descent iteration consists of tangent-space projection of the Euclidean gradient, a tangent step, an elementwise trimming step for incoherence control, and a TT retraction by TTSVD:
\[
T_t^+=T_t-\eta_t P_{T_t}(G_t),\qquad T_{t+1}=\mathrm{TTSVD}_r(\mathrm{Trim}_{\xi_t}(T_t^+)).
\]
This formulation exploits the manifold geometry of real low-TT-rank coefficient tensors derived from Hermitian matrix product operators [2605.04533].

## 3. Regret, convergence, and step-size regimes

Two analytical traditions coexist. In stochastic optimization, the goal is convergence to critical points or small gradient norms. In adversarial online optimization, the goal is sublinear regret relative to a static or time-varying comparator sequence. The distinction is explicit: some Riemannian SGD analyses provide convergence bounds but do not present adversarial regret guarantees [2312.07990].

Under geodesic \(L\)-smoothness and unbiased mini-batch gradients, RSGD on a Hadamard manifold satisfies
\[
\sum_{k=0}^{K-1}\alpha_k\Big(1-\frac{L\alpha_k}{2}\Big)\mathbb{E}\|\operatorname{grad} f(x_k)\|^2
\le f(x_0)-f_\star+\frac{L\sigma^2}{2b}\sum_{k=0}^{K-1}\alpha_k^2.
\]
For a constant step size \(0<\alpha<2/L\), this yields an average squared-gradient bound of order \(O(1/K+\sigma^2/b)\); with diminishing \(\alpha_k\), one obtains \(\liminf_{k\to\infty}\mathbb{E}\|\operatorname{grad} f(x_k)\|=0\) [2312.07990]. Earlier stochastic approximation results established almost sure convergence of intrinsic and retraction-based updates to critical points under compactness or Hadamard-type stability assumptions [1111.5280].

In online convex optimization, the central quantity is regret. For a static comparator \(x^\ast\), the standard regret is
\[
\mathrm{Regret}_T=\sum_{t=1}^T f_t(x_t)-\min_{x\in\mathcal{X}}\sum_{t=1}^T f_t(x),
\]
while dynamic regret compares against a sequence \(\{u_t\}\) and incorporates the path variation
\[
P_T=\sum_{t=1}^{T-1} d(u_t,u_{t+1}).
\]
For decentralized online Riemannian optimization on Hadamard manifolds, the dynamic regret bound established for both exact and simplified consensus is
\[
\mathcal{O}\!\left(\frac{\sqrt{T(1+P_T)}}{\sqrt{1-\sigma_2(W)}}\right),
\]
where \(\sigma_2(W)\) is the second-largest singular value of the mixing matrix, so the spectral gap \(1-\sigma_2(W)\) governs the network penalty [2410.05128]. When \(n=1\) or \(\sigma_2(W)=0\), this reduces to centralized Riemannian OGD with regret order \(\mathcal{O}(\sqrt{T(1+P_T)})\) [2410.05128].

Optimistic Riemannian algorithms refine the comparator dependence by exploiting temporal regularity in the losses. For R-OOGD, if the losses are \(g\)-convex, \(g\)-\(G\)-Lipschitz, and \(g\)-\(L\)-smooth, then
\[
R_T^{\mathrm{dyn}}
\le \frac{D_0^2+2D_0P_T}{\eta}
+\eta\,\frac{4\zeta_0^2}{\sigma_0}\big(G^2+V_T\big),
\]
where \(V_T=\sum_{t=2}^T \sup_{x\in X}\|\nabla f_t(x)-\nabla f_{t-1}(x)\|^2\), \(\sigma_0=\sigma(K,D_0)\), and \(\zeta_0=\zeta(\kappa,D_0)\) are curvature-dependent constants [2308.16004]. The adaptive meta-expert variant R-AOOGD achieves
\[
R_T^{\mathrm{dyn}}
\le \frac{\zeta_0}{\sqrt{\sigma_0}}\,
\sqrt{(1+V_T+P_T)(1+P_T)},
\]
matching the Euclidean dependence on regularity measures up to curvature constants [2308.16004].

A distinct line of work replaces geodesic convexity by horospherical convexity on Hadamard manifolds. For \(h\)-convex losses, ROGD with fixed \(\eta\) satisfies
\[
\mathrm{Regret}_T
\le \frac{d^2(x_1,x^\ast)}{2\eta}
+\frac{\eta}{2}\sum_{t=1}^T \|\operatorname{grad} f_t(x_t)\|^2,
\]
which yields \(O(\sqrt{T})\) regret under \(L\)-Lipschitzness, with no curvature term in the bound [2509.11236]. For \(\mu\)-strongly \(h\)-convex losses and \(\eta_t=1/(\mu t)\), the regret becomes
\[
\mathrm{Regret}_T\le \frac{L^2}{2\mu}(1+\log T),
\]
again curvature-independent [2509.11236]. This establishes that curvature dependence is not universal; it can disappear when the objective class is strengthened from \(g\)-convexity to \(h\)-convexity [2509.11236].

## 4. Consensus, networks, and decentralized extensions

Decentralized ROGD introduces a second geometric operation beyond gradient descent: manifold consensus. On Hadamard manifolds, the exact consensus step is formulated through the weighted Fréchet mean,
\[
x_{i,t+1}
=\arg\min_{x\in\mathcal{X}} \sum_{j=1}^n w_{ij}\,d^2(x,y_{j,t+1}),
\]
which is unique because the objective is geodesically convex [2410.05128]. The corresponding algorithm is described as the first decentralized online Riemannian optimization algorithm on Hadamard manifolds [2410.05128].

The consensus step admits a linear variance reduction result:
\[
\operatorname{VF}(\{x_{i,t+1}\})
\le \sigma_2^2(W)\,\operatorname{VF}(\{y_{i,t+1}\}),
\]
where \(\operatorname{VF}\) denotes Fréchet variance [2410.05128]. Since weighted Fréchet means require solving an inner minimization problem, a closed-form approximation based on log-exp averaging was proposed:
\[
x_{i,t+1}
=\operatorname{Exp}_{y_{i,t+1}}\!\left(
\gamma_t\sum_{j=1}^n w_{ij}\,\operatorname{Log}_{y_{i,t+1}}(y_{j,t+1})
\right).
\]
With \(\gamma_t=1/(2\zeta(\kappa,2D))\), this also contracts variance linearly and attains the same dynamic regret order, although the contraction factor is weaker than for exact Fréchet averaging [2410.05128].

The network error can be bounded explicitly. For exact consensus,
\[
d(x_{i,t},\bar x_t)\le
\frac{\eta\,\sigma_2(W)\sqrt{n}\,L}{1-\sigma_2(W)},
\]
while the closed-form consensus yields
\[
d(x_{i,t},\bar x_t)\le
\frac{4\,\zeta(\kappa,2D)\,\eta\,\sqrt{n}\,L}{1-\sigma_2(W)},
\]
where \(\bar x_t\) is the Fréchet mean of the network iterates [2410.05128]. These formulas make explicit the role of curvature, Lipschitz continuity, network size, and spectral gap.

Beyond Hadamard manifolds, positive curvature obstructs global convexity of geodesic distance, projection nonexpansiveness, and consensus analysis. A curvature-aware consensus step on manifolds with bounded sectional curvature addresses this by restricting the feasible set to a convex ball and choosing a consensus step size \(s_t\) from curvature-dependent constants \(C_1\), \(C_2\), and distortion terms \(C_3\), \(C_4\) [2509.07779]. The resulting one-step intrinsic averaging update is
\[
x_{i,t+1}=
\operatorname{Exp}_{y_{i,t+1}}\!\Big(
s_t\sum_{j=1}^n w_{ij}\operatorname{Log}_{y_{i,t+1}}(y_{j,t+1})
\Big),
\]
and it enjoys linear variance contraction
\[
\frac1n\sum_{i=1}^n d^2(x_i(s),\bar y)
\le \rho\,
\frac1n\sum_{i=1}^n d^2(y_i,\bar y),
\qquad \rho\in(0,1),
\]
together with \(O(\sqrt{T})\) static regret in both full-information and two-point bandit settings [2509.07779]. This shows that decentralization can be extended beyond nonpositive curvature, but only under explicit convexity-radius and distortion controls [2509.07779].

## 5. Model-specific realizations

In hyperbolic space, the hyperboloid model makes ROGD fully explicit. Hyperbolic \(n\)-space is represented as
\[
\mathbb{H}^n=\{x\in\mathbb{R}^{n+1}:\langle x,x\rangle_L=-1,\ x_0>0\},
\]
with Lorentz inner product
\[
\langle x,y\rangle_L=-x_0y_0+\sum_{i=1}^n x_i y_i.
\]
The tangent projection is \(P_x(u)=u+\langle x,u\rangle_Lx\), the distance is \(d(x,y)=\operatorname{arccosh}(-\langle x,y\rangle_L)\), and the exact exponential map is
\[
\operatorname{Exp}_x(v)=
\cosh(\|v\|_L)\,x+\sinh(\|v\|_L)\,\frac{v}{\|v\|_L}.
\]
This yields exact geodesic gradient steps rather than retraction approximations [1805.08207]. For online Fréchet mean estimation with losses \(f_t(x)=\tfrac12 w_t d(x,y_t)^2\), the gradient satisfies \(\operatorname{grad} f_t(x)=-w_t\operatorname{Log}_x(y_t)\), so the update is a geodesic step toward the incoming sample [1805.08207].

On the manifold of symmetric positive definite matrices, several geometries and applications appear. Under the affine-invariant Riemannian metric,
\[
\langle X_P,Y_P\rangle_P=\operatorname{tr}(X_P^T P^{-1}Y_P P^{-1}),
\]
the manifold \(S_{++}^d\) is Hadamard and the exponential map is
\[
\operatorname{Exp}_P(X_P)=
P^{1/2}\exp(P^{-1/2}X_PP^{-1/2})P^{1/2}
\]
[2312.07990]. This geometry supports RSGD for the Riemannian centroid problem, nonlinear gossip updates that move neighboring covariance matrices toward each other along the Fisher geodesic, and online Tyler’s \(M\)-estimation and online Fréchet mean computation under curvature-independent \(h\)-convex analyses [1111.5280] [2509.11236]. Empirically, increasing the batch size improves RSGD performance on the SPD centroid problem, and the number of steps required for convergence is a convex monotone decreasing function of batch size [2312.07990].

On fixed-rank tensor-train manifolds, online Riemannian gradient descent has been specialized to quantum state tomography with matrix product operators. The key structural result is that Hermiticity of the MPO cores implies that the coefficient tensor under the Pauli or generalized Gell-Mann basis is real and low-TT-rank, making the reconstruction problem equivalent to noisy low-rank tensor completion on the manifold \(M_r\) of fixed TT rank [2605.04533]. With proper initialization, the resulting oRGD converges linearly to the target MPO, the number of distinct measurement settings scales quadratically with the system size, and the per-iteration computational costs are \(O(nd^2r_{\max}^2)\) for contraction and tangent projection and \(O(nd^2r_{\max}^3)\) for TT retraction [2605.04533].

ROGD also appears in game-theoretic and regression settings. R-OGDA extends optimistic descent-ascent to zero-sum \(g\)-convex-concave games on product manifolds and achieves average-iterate \(1/T\), best-iterate \(1/\sqrt{T}\), and linear last-iterate convergence rates under strong \(g\)-convexity-strong concavity [2308.16004]. Online geodesic regression and robust geometry-aware PCA provide further examples in which intrinsic updates operate directly on spheres, Grassmannians, or SPD products [2308.16004].

## 6. Variants, misconceptions, and open problems

A common misconception is that “online” and “stochastic” are interchangeable in the Riemannian setting. They are not. Stochastic analyses typically assume i.i.d. sampling, unbiased gradient oracles, and convergence criteria based on expected gradient norms or variational measures \(V_k(x)\); they do not automatically imply adversarial regret guarantees [2312.07990] [1111.5280]. By contrast, online convex optimization studies regret against static or dynamic comparators under adversarial sequences of losses [2410.05128] [2308.16004] [2509.11236].

A second misconception is that exact geodesic updates are invariably too costly to use. This is false in some important manifolds. In the hyperboloid model of hyperbolic space, the exponential and logarithm maps have closed forms and the tutorial evidence shows that exact hyperboloid exponential updates can outperform Poincaré-ball retraction updates in Fréchet mean problems [1805.08207]. At the same time, retractions remain essential in matrix and tensor manifolds, where truncated SVD, QR-based updates, or TT-rounding are computationally preferable [1111.5280] [2605.04533].

A third misconception is that curvature-dependent regret degradation is unavoidable. Geodesically convex analyses on manifolds with nonzero curvature often introduce curvature factors such as \(\zeta(\kappa,D)\) or \(\sigma(K,D)\) through comparison geometry [2410.05128] [2308.16004]. However, under horospherical convexity on Hadamard manifolds, ROGD admits \(O(\sqrt{T})\) and \(O(\log T)\) regret bounds with no curvature term in the final rates [2509.11236]. This does not eliminate curvature from the geometry; rather, it replaces the function class by the stronger notion of \(h\)-convexity [2509.11236].

The present theory remains assumption-heavy. Hadamard geometry is often imposed to guarantee uniqueness of projections and Fréchet means, geodesic convexity and Lipschitz continuity are standard prerequisites for regret analysis, and positive-curvature settings require bounded convex balls with radius below the convexity threshold [2410.05128] [2509.07779]. In MPO-based tomography, Hermiticity and trace-one handling are addressed, but positivity is not enforced; incorporating PSD constraints would change the geometry and remains future work [2605.04533]. In decentralized dynamic regret, tuned step sizes depend on the path variation \(P_T\), and designing adaptive policies without oracle knowledge remains open [2410.05128].

The stated extensions are correspondingly geometric and algorithmic: directed or time-varying graphs, coupled constraints, bandit feedback, acceleration, strong convexity and smoothness, adaptive consensus weights, second-order methods on Hadamard manifolds, tighter curvature dependence beyond Hadamard manifolds, and retraction-transport generalizations with explicit constants [2410.05128] [2509.07779]. This suggests that Riemannian Online Gradient Descent is best understood not as a single algorithm but as a geometric design pattern: update in a tangent space using a metric-compatible gradient, return to the manifold by exponential map or retraction, and control curvature either through comparison inequalities, Busemann-function envelopes, or model-specific structure.

Source: https://www.emergentmind.com/topics/riemannian-online-gradient-descent