---
title: Recurrent Manifold Optimization (RMO)
url: https://www.emergentmind.com/topics/recurrent-manifold-optimization-rmo
type: topic
---

# Recurrent Manifold Optimization (RMO)

Recurrent Manifold Optimization (RMO) denotes a class of methods in which recurrent or iterative refinement is performed with explicit respect for non-Euclidean geometry. In recent cross-modal registration, RBE-Flow formulates dense flow estimation as a closed-loop recurrent Bayesian estimation problem on learned feature manifolds, with a Recurrent Manifold Optimization block that produces both local updates and uncertainty estimates [2606.30492]. In geometric optimization for deep learning, RMO also refers to training recurrent networks whose hidden-to-hidden weights remain on the Stiefel manifold, thereby preserving orthogonality during learning [2302.08210]. In manifold-valued sequence modeling, the SPD-SRU formulates recurrence directly on \(SPD(n)\) and defines learning as a constrained optimization problem over orthogonal actions and simplex-constrained weights [1805.11204]. These usages suggest that RMO is best understood as an umbrella term for recurrent procedures that embed optimization steps, state evolution, or parameter learning in a Riemannian or manifold-constrained setting.

## 1. Terminological scope and recurrent formulations

A common source of ambiguity is that the acronym “RMO” is used in more than one way. Li et al. use RMO to mean “Riemannian manifold optimization” in advanced wireless communications, where optimization is performed directly on manifolds such as the complex sphere, Stiefel manifold, Grassmann manifold, and Hermitian positive-definite manifold [2602.08225]. By contrast, RBE-Flow uses “Recurrent Manifold Optimization” for a specific iterative module inside a recurrent Bayesian estimator on learned feature manifolds [2606.30492]. The SPD-SRU exposition labels its learning problem “Recurrent Manifold Optimization” because the recurrent cell and its optimization operate in a manifold-valued setting, specifically \(SPD(n)\) with weighted Fréchet means and orthogonal group actions [1805.11204]. The survey of geometric optimization for deep learning uses the term for training recurrent networks whose recurrent weight matrix \(W\) is constrained by \(W^T W = I_m\), so that optimization proceeds on \(\St(m,m)\) rather than in Euclidean space [2302.08210].

| Setting | Manifold or constraint | Recurrent role |
|---|---|---|
| Orthogonal RNN training | \(W \in \St(m,m)\), \(W^T W = I_m\) | Hidden-to-hidden transition is manifold-constrained |
| SPD-SRU | \(SPD(n)\), \(g \in O(n)\), simplex-normalized weights | Hidden state and moving means are manifold-valued |
| RBE-Flow | Learned feature manifolds, dense flow with covariance | Iterative refinement and Bayesian fusion are closed-loop |
| Wireless RMO | Complex sphere, Stiefel, Grassmann, HPD, circle manifolds | Not recurrent in the strict RNN sense; supplies the geometric optimization backbone |

This suggests that “recurrent” can refer either to temporal sequence processing, as in recurrent neural networks and SPD-SRU, or to iterative closed-loop refinement, as in RBE-Flow. It also suggests that the invariant element across these formulations is not a single algorithm but a shared commitment to manifold-respecting updates.

## 2. Geometric foundations and algorithmic primitives

The geometric basis is the standard constrained problem
\[
\min_{x \in \mathcal M} f(x),
\]
where \(\mathcal M\) is a Riemannian submanifold and \(F\) is the restriction of \(f\) to \(\mathcal M\) [2602.08225]. A real manifold \(\mathcal M\) of dimension \(d\) is a topological space locally homeomorphic to \(\mathbb R^d\) and endowed with a smooth structure. If \(\mathcal M\) is embedded in \(\mathbb R^n\) or \(\mathbb C^n\), each tangent space \(T_x \mathcal M\) can be equipped with the inner product induced by the ambient Euclidean metric, yielding a Riemannian manifold \((\mathcal M,g)\) [2602.08225].

For embedded manifolds, the Riemannian gradient is the orthogonal projection of the Euclidean gradient onto the tangent space:
\[
\operatorname{grad} f(x) = P_x(\nabla f(x)).
\]
On the sphere \(\mathbb S^{n-1}\), the tangent space is
\[
T_x \mathbb S^{n-1} = \{\xi \in \mathbb R^n : x^T \xi = 0\},
\]
with projector
\[
P_x(u)=u-(x^T u)x.
\]
The Riemannian Hessian is
\[
\operatorname{Hess} f(x)[\xi] = P_x(D(\operatorname{grad} f)(x)[\xi]),
\]
and the exponential map on the sphere is
\[
\operatorname{Exp}_x(\xi)=x\cos(\|\xi\|)+\xi/\|\xi\| \sin(\|\xi\|).
\]
Because exact geodesic updates are often expensive, practical algorithms use retractions. Examples include sphere normalization,
\[
R_x(\xi)=\frac{x+\xi}{\|x+\xi\|},
\]
and Stiefel retractions based on QR factorization or polar decomposition [2602.08225].

The recurrent formulations of RMO draw on this same toolkit. The survey treatment of orthogonal RNNs makes the Stiefel geometry explicit, while the SPD-SRU relies on manifold-valued averages and orthogonal actions, and RBE-Flow uses local Gauss–Newton steps whose damping is adapted by posterior covariance [2302.08210]. This suggests that recurrence does not replace manifold optimization; rather, it composes manifold-respecting local updates over time or across refinement iterations.

## 3. Orthogonality-constrained recurrent networks on the Stiefel manifold

In the orthogonal-RNN formulation, the recurrent model is
\[
h_t=\sigma(Ux_t + Wh_{t-1}+b),
\]
with loss
\[
f(W,U,b)=\sum_{t=1}^{T}\ell(y_t,\;Vh_t+c),
\]
subject to
\[
W \in \St(m,m)=\{W: W^T W = I\}.
\]
The purpose of the constraint is explicit: keeping \(W\) orthogonal stabilizes long-range gradient flow and avoids vanishing or exploding norms [2302.08210].

The tangent space at \(W\) is
\[
T_W \St(m,m)=\{\Xi \in \mathbb R^{m\times m}: W^T\Xi + \Xi^T W = 0\},
\]
and the Riemannian gradient is obtained by projecting the Euclidean gradient \(\nabla_W f\) onto that tangent space:
\[
\operatorname{grad} f(W)=\nabla_W f - W\,\operatorname{Sym}(W^T \nabla_W f),
\]
where \(\operatorname{Sym}(A)=\tfrac12(A+A^T)\). An equivalent expression is
\[
\operatorname{grad} f(W)=(I-WW^T)\nabla_Wf + W\,\operatorname{Skew}(W^T\nabla_Wf).
\]

After computing a descent direction in \(T_W\St\), the update is returned to the manifold with the QR retraction
\[
\operatorname{Retr}_W(\Xi)=\mathrm{qf}(W+\Xi),
\]
so that
\[
W_{\text{new}}=\mathrm{qf}(W-\gamma\,\operatorname{grad} f(W))
\]
remains on \(\St(m,m)\). When momentum is used, a tangent vector \(\Xi \in T_W \St\) can be transported to \(T_{W'}\St\) by projection:
\[
\mathcal T_{W\to W'}(\Xi)=\Xi-W'\,\operatorname{Sym}((W')^T\Xi).
\]
The survey presents a one-epoch Riemannian SGD-with-momentum loop built from these ingredients: Euclidean backpropagation through time, tangent projection, momentum update in the tangent space, retraction for \(W\), and ordinary Euclidean updates for \(U\) and \(b\) [2302.08210].

The same source states that treating \(\St(m,m)\) as a compact Riemannian manifold with bounded curvature gives standard convergence results for Riemannian SGD-with-momentum. It also reports that expRNN uses this QR retraction and parallel transport and shows stable long-term memory on copying and addition tasks, while GORU augments the orthogonal RNN with forget gates and achieves \(98.9\%\) on permuted-MNIST, outperforming Euclidean LSTMs, uRNNs, and soft-orthogonal variants [2302.08210]. The built-in QR retraction has cost \(O(m^3)\), and the same source notes that GeoTorch or McTorch can be used to implement manifold-constrained training in PyTorch.

## 4. SPD-valued recurrence and statistical recurrent models

The SPD-SRU generalizes recurrence from Euclidean hidden states to the manifold
\[
SPD(n)=\{P\in \mathbb R^{n\times n}\mid P^T=P,\;P\succ 0\},
\]
whose tangent space at \(P\) is
\[
T_P SPD(n)=\{V\in \mathbb R^{n\times n}\mid V^T=V\}.
\]
Two metrics are emphasized: the affine-invariant metric
\[
\langle U,V\rangle_P = \operatorname{Tr}(P^{-1} U P^{-1} V),
\]
with geodesic
\[
\gamma(t)=P^{1/2}(P^{-1/2}QP^{-1/2})^tP^{1/2},
\]
distance
\[
\operatorname{dist}_A(P,Q)=\|\log(P^{-1/2}QP^{-1/2})\|_F,
\]
and corresponding exponential and logarithm; and the Stein metric
\[
d_S(P,Q)=\sqrt{\log\det\!\left(\frac{P+Q}{2}\right)-\frac12\log\det(PQ)},
\]
which is described as computationally cheaper and is used for the weighted Fréchet means in the cell [1805.11204].

Given an input sequence \(\{X_t\}_{t=1}^T \subset SPD(n)\), the SPD-SRU maintains moving means \(M_t^{(\alpha)} \in SPD(n)\) over a collection of scales \(\alpha \in J \subset (0,1)\). Its forward pass is defined by seven manifold operations: combining past scales into \(Y_t\) via weighted Fréchet mean, applying the orthogonal translation action \(T(P,g)=gPg^T\) to obtain \(R_t\), fusing \(R_t\) with the input \(X_t\) to get \(T_t\), translating to \(\Phi_t\), updating each moving average \(M_t^{(\alpha)}\), aggregating them into \(S_t\), and finally producing
\[
O_t=\operatorname{Chol}\!\bigl(\operatorname{ReLU}(\operatorname{Chol}(T(S_t,g^{(y)})))\bigr)\in SPD(n).
\]
The hidden state is \(H_t := O_t\) [1805.11204].

The optimization problem is explicit:
\[
\min_\theta L(\theta)
\]
with respect to
\[
\theta=\{G^{(r)},G^{(p)},G^{(y)}\in \operatorname{skew}(n),\;
u^{(y,\alpha)},u^{(t)},u^{(s,\alpha)}\in \mathbb R,\;
U\in\mathbb R^{K\times m}, b\in\mathbb R^K\},
\]
subject to \(g=\exp(G)\in O(n)\), \(\omega_i \ge 0\), and \(\sum \omega_i=1\). The paper reduces optimization to Euclidean updates by parametrizing \(g\) as \(\exp(G)\) with \(G^T=-G\), and by parametrizing the weights through unconstrained \(u_i\) followed by normalization \(\omega_i = u_i^2/\sum_j u_j^2\). Standard backpropagation computes \(\partial L/\partial G\), \(\partial L/\partial u\), \(\partial L/\partial U\), and \(\partial L/\partial b\), after which \(G\) is reprojected to enforce skew-symmetry and the normalized weights are recomputed [1805.11204].

The statistical analysis is based on an isometric mapping \(\Phi: SPD(n)\to S_+^\infty\) into a Hilbert sphere of normalized Gaussian densities, with
\[
d_S(\Phi(P),\Phi(Q)) = d_{\text{Stein}}(2P,2Q).
\]
A recursive weighted Fréchet mean estimator on the positive orthant of \(S^\infty\) is given in closed form, and Proposition 5 states that \(\operatorname{Var}(m_k)\to 0\) as \(k\to\infty\) and that the convergence rate is super-linear [1805.11204].

The empirical results are reported in three domains. On Moving MNIST, SPD-SRU with hidden SPD dimension \(c+1=11\) uses \(1{,}559\) parameters and \(6.2\) seconds per epoch, and achieves \(1.00\pm0.00\) for the \(2\)-class \((30\!-\!60^\circ)\) setting, \(0.96\pm0.02\) for the \(2\)-class \((10\!-\!15^\circ)\) setting, and \(0.94\pm0.02\) for the \(3\)-class \((10\!-\!15\!-\!20^\circ)\) setting. The same table states that SPD-SRU uses approximately \(10\times\) fewer parameters than SRU/LSTM while maintaining near-perfect accuracy as the angle gap shrinks. On UCF11 action recognition, SPD-SRU uses \(3{,}337\) parameters, about \(76\) seconds per epoch, and reaches test accuracy \(0.78\), equal to TT-RNN with approximately \(2\!-\!4\times\) fewer parameters. In a brain-imaging group-difference test on Parkinson’s versus control tracts, a permutation test with \(10{,}000\) permutations yields \(p=0.01\) for Left M1 and \(p=0.032\) for Right M1, whereas a baseline based on point-wise Fréchet mean plus Cramér’s test yields \(p>0.17\) [1805.11204].

## 5. Closed-loop Bayesian refinement on learned feature manifolds

RBE-Flow reformulates dense cross-modal flow estimation as a recurrent Bayesian estimation problem on learned feature manifolds. At each pixel \(u\), with feature maps \(\mathbf F_0,\mathbf F_1:\Omega\to\mathbb R^d\), the residual is
\[
r(\Flow(u))=\mathbf F_0(u)-\mathbf F_1(u+\Flow(u))\in \mathbb R^d.
\]
Over a local neighborhood, residuals are stacked into \(\mathbf r \in \mathbb R^m\) and Jacobians into \(\mathbf J \in \mathbb R^{m\times 2}\). The Recurrent Manifold Optimization block solves at iteration \(k\) the damped non-linear least-squares problem
\[
\Delta \Flow_k = \arg\min_{\Delta}\; \tfrac12\|\mathbf r_k+\mathbf J_k\Delta\|_2^2 + \tfrac12\lambda_k' \|\Delta\|_2^2,
\]
which under Gauss–Newton linearization gives
\[
(\mathbf J_k^T \mathbf J_k + \lambda_k' \mathbf I)\Delta\Flow_k = -\mathbf J_k^T \mathbf r_k. \tag{1}
\]
The resulting flow observation is
\[
\mathbf Z_k=\Flow_{k-1}+\Delta\Flow_k, \tag{2}
\]
and the block also emits a local measurement covariance \(\mathbf R_k\in\mathbb R^{2\times 2}\), with \(\mathbf R_k=\operatorname{CovNet}(\text{features})\) in the measurement model [2606.30492].

The probabilistic update stage is the Uncertainty-Adaptive Probabilistic Update (UAPU). From prior mean \(\hat\Flow_{k\mid k-1}\) and covariance \(\mathbf P_{k\mid k-1}\), sigma points are formed using the Cholesky factor of \(\mathbf P\), propagated through a learned observation network \(\operatorname{ObsNet}\), and used to compute predicted observation mean \(\hat{\mathbf Z}_k\), innovation covariance \(\mathbf P_{zz}\), and cross-covariance \(\mathbf P_{xz}\). The Kalman-style gain is
\[
\mathbf K_k=\mathbf P_{xz}\mathbf P_{zz}^{-1}, \tag{K}
\]
and the posterior updates are
\[
\hat\Flow_{k\mid k}
=
\hat\Flow_{k\mid k-1}
+
\mathbf K_k(\mathbf Z_k-\hat{\mathbf Z}_k), \tag{F}
\]
\[
\mathbf P_{k\mid k}
=
\mathbf P_{k\mid k-1}
-
\mathbf K_k \mathbf P_{zz}\mathbf K_k^T. \tag{P}
\]

A defining feature of this formulation is the feedback of posterior uncertainty into the next optimization step:
\[
\lambda_k'=\lambda_k+\beta\,\operatorname{tr}(\mathbf P_{k-1\mid k-1}). \tag{D}
\]
The paper states that in regions or iterations of high uncertainty, the solver automatically shifts toward smaller, more conservative update steps. To stabilize training, it introduces a geometry-aware rectified NLL loss, motivated by the observation that a naive Gaussian NLL of the form \(\tfrac12[\log \sigma^2 + (e^2/\sigma^2)]\) often collapses \(\sigma\to 0\). The full loop includes multi-scale features \(\mathbf F_{1/2}, \mathbf F_{1/4}, \mathbf F_{1/8}\), a Global Flow Init stage that builds a \(4\)D correlation at \(1/8\) scale and applies a softmax to obtain \(\Flow_{\mathrm{init}}\), recurrent refinement in which ConvGRU predicts \(\mathbf J_k,\mathbf g_k,\lambda_k\), and a total loss \(\mathcal L=\lambda_{\mathrm{init}}\mathcal L_{\mathrm{init}}+\lambda_{\mathrm{NLL}}\mathcal L_{\mathrm{fine}}\) [2606.30492].

The abstract reports extensive experiments on OSdataset, WHU-OPT-SAR, and RoadScene, stating that RBE-Flow consistently achieves state-of-the-art performance and outperforms existing methods by a significant margin, particularly under strict sub-pixel criteria [2606.30492]. The paper’s own summary interprets the method as converting each feature-metric refinement into a local Gauss–Newton measurement with uncertainty, then fusing these non-linear observations into a global Bayesian belief.

## 6. Broader Riemannian optimization context, applications, and practical interpretation

The broader Riemannian-optimization literature clarifies why manifold-aware recurrence is attractive in high-dimensional nonconvex problems. Li et al. describe next-generation wireless systems such as massive MIMO, reconfigurable intelligent surfaces, integrated sensing and communication, and fluid antenna systems as settings in which large-scale optimization with nonconvex constraints is central, and they argue that conventional Euclidean-space methods rely on approximations or relaxations that degrade performance and incur substantial computational costs [2602.08225]. Their formulation of RMO operates directly on the manifold defined by the constraints, satisfying the constraints at every optimization step.

The paper gives three baseline manifold algorithms. Riemannian Gradient Descent computes \(g_k=\operatorname{grad} f(x_k)\), chooses a step size, and updates by retraction \(x_{k+1}=R_{x_k}(-\alpha_k g_k)\). Its worst-case overall complexity is \(O(n^3)\) for Stiefel-type settings with \(p\approx n\), or \(O(n)\) for the sphere. Riemannian Conjugate Gradient augments the descent direction with a transported previous direction and a conjugacy coefficient, with per-iteration complexity of the same order as RGD plus vector-transport cost \(O(n^3)\). Riemannian Trust-Region solves a quadratic model over the tangent space and uses the ratio
\[
\rho_k = \frac{f(x_k)-f(R_{x_k}(\eta_k))}{m_k(0)-m_k(\eta_k)}
\]
to accept steps and adapt the trust-region radius [2602.08225].

The convergence guarantees are stated under standard assumptions: \(f\) is \(C^2\), \(\operatorname{grad} f\) is Lipschitz, and \(\mathcal M\) is compact. Under these conditions, RGD with exact line search converges to critical points with global sublinear rate \(O(1/\epsilon^2)\) in \(\|\operatorname{grad} f\|\); RCG often shows superlinear convergence locally under nondegeneracy; and RTR obtains quadratic convergence near a nondegenerate local minimum and global convergence to second-order critical points [2602.08225].

The wireless case studies provide concrete manifold assignments. In FAS-assisted secure beamforming in NOMA, the beamformer \(w\in \mathbb C^N\) satisfies \(\|w\|_2^2\le P\), so the problem is cast on the complex sphere with secrecy-rate objective
\[
\max_{w\in \mathbb S^{N-1}}
R_s(w)=
\Bigl[
\log_2\!\Bigl(1+\frac{|h_b(\theta)^H w|^2}{\sigma_b^2}\Bigr)
-
\log_2\!\Bigl(1+\frac{|h_e(\theta)^H w|^2}{\sigma_e^2}\Bigr)
\Bigr]^+.
\]
This is solved by RGD and RTR. The reported performance comparison states that RTR improves average secrecy rate by approximately \(66.6\%\) over MM and \(50.3\%\) over SDR, while average runtime to convergence is \(0.10\) s for RTR, \(1.23\) s for RGD, \(35.01\) s for SDR, and \(161.93\) s for MM. In massive MIMO precoding, the objective
\[
\max_{F\in \St(p,n)} \log_2 \left| I + HHF F^H H^H \right|
\]
is posed on the Stiefel manifold, and RCG is described as having linear per-iteration cost in \(n\) and \(p\), outperforming \(O(n^3)\) SDR. In RIS phase-shift design, optimization is carried out on the complex circle manifold with \(O(M)\) or \(O(M^2)\) cost versus \(O(M^3)\) for MM. In ISAC waveform and phase optimization, the covariance matrix \(R\) is optimized on the HPD manifold by RTR, avoiding costly PSD projections [2602.08225].

The implementation guidelines in the same source are also directly relevant to recurrent formulations. Manifold choice should match the physical or geometric constraint: unit modulus to the circle manifold, orthonormal columns to the Stiefel manifold, subspace invariance to the Grassmann manifold, and PSD covariance to the HPD manifold. Retraction choice depends on the manifold and computational regime: normalization on the sphere; QR on Stiefel when \(p\ll n\); polar decomposition when higher numerical accuracy is needed; and matrix-exponential or symmetric polar maps for PSD-type settings. Armijo backtracking with \(0<\beta<1\) and initial \(\alpha_0\sim 1\) is recommended for step-size tuning; trust-region radius can be adjusted by halving when \(\rho_k<0.25\) and doubling when \(\rho_k>0.75\); and the Polak–Ribière coefficient is reported to often outperform Fletcher–Reeves in practice [2602.08225].

Taken together, these results suggest a unifying interpretation of Recurrent Manifold Optimization. In one line of work, the manifold constraint is imposed on recurrent parameters, as with orthogonal RNNs on \(\St(m,m)\). In another, the hidden state itself evolves on a manifold, as in the SPD-SRU on \(SPD(n)\). In a third, the recurrent element is an uncertainty-aware refinement loop on a learned manifold, as in RBE-Flow. Across all of these settings, the recurring technical components are tangent-space computation, manifold-compatible retraction, transport or covariance propagation, and update rules that preserve the defining geometry rather than approximating it away.

Source: https://www.emergentmind.com/topics/recurrent-manifold-optimization-rmo