--- title: 'RiemannLoRA: Geometric Low-Rank Adaptation' url: https://www.emergentmind.com/topics/riemannlora type: topic --- # RiemannLoRA: Geometric Low-Rank Adaptation RiemannLoRA denotes a family of geometric reformulations of Low-Rank Adaptation (LoRA) in which low-rank adapters are optimized with explicit Riemannian structure rather than as unconstrained Euclidean factors. In the 2025 literature, the name is used for two closely related but distinct constructions. One imposes an orthonormality constraint on the LoRA factor \(B\) and optimizes it on the compact Stiefel manifold \(St(d,r)\), with the stated goal of eliminating basis redundancy and restoring full effective rank in \(\Delta W = BA\) [2508.17901]. The other treats the set of all rank-\(r\) updates itself as a smooth manifold \(\mathcal M_r\), thereby removing the factorization ambiguity \((A,B)\mapsto(AS,B\,S^{-\top})\), defining a manifold steepest-descent direction, and deriving a locally-optimal initialization [2507.12142]. Both formulations preserve the parameter-efficient fine-tuning setting of LoRA while replacing standard Euclidean optimization with manifold-aware updates. ## 1. LoRA setting and the motivation for a geometric reformulation In the Stiefel-manifold formulation, a frozen pre-trained weight matrix in an LLM layer is written as \(W_0\in\mathbb R^{d\times k}\), and LoRA replaces a full fine-tuning update \(\Delta W\) with a rank-\(r\) factorization \[ \Delta W = B\,A, \] where \(B\in\mathbb R^{d\times r}\), \(A\in\mathbb R^{r\times k}\), and \(r\ll \min(d,k)\). The fine-tuned weight is \[ W = W_0 + BA. \] Standard LoRA then solves the unconstrained minimization \[ \min_{A,B} L(W_0 + B\,A;D) \] in Euclidean space, typically with AdamW [2508.17901]. The fixed-rank-manifold formulation uses the equivalent low-rank viewpoint \[ \Delta W = A\,B^\top,\quad A\in\mathbb R^{d\times r},\;B\in\mathbb R^{k\times r}, \] and identifies all such rank-\(r\) matrices with \[ \mathcal M_r=\{X\in\mathbb R^{d\times k}\mid \mathrm{rank}(X)=r\}=\{A\,B^\top\}, \] a smooth embedded manifold of dimension \((d+k)r-r^2\) [2507.12142]. The shared motivation is that standard LoRA inherits optimizer pathologies from the matrix factorization itself. One work states that LoRA suffers from “basis redundancy in LoRA’s \(B\) matrix when using AdamW,” and that this fundamentally limits performance [2508.17901]. The other emphasizes “overparametrization in low-rank matrix factorization” and the resulting parametrization ambiguity [2507.12142]. Taken together, these formulations recast LoRA training as a problem in constrained or intrinsic optimization over low-rank geometry rather than over arbitrary factors. ## 2. Two formulations of RiemannLoRA The two papers use the same name but organize the geometry differently. | Formulation | Geometric object | Principal effect | |---|---|---| | Stiefel-constrained LoRA | \(B\in St(d,r)\) with \(B^\top B=I_r\) | Enforces orthogonality of LoRA basis vectors | | Fixed-rank-manifold LoRA | \(\Delta W\in\mathcal M_r\) with \(\mathrm{rank}(\Delta W)=r\) | Removes factorization ambiguity intrinsically | In the Stiefel-constrained variant, the objective is \[ \min_{A\in\mathbb R^{r\times k},\,B\in\mathbb R^{d\times r}} f(A,B) \quad\text{s.t.}\quad B^T B = I_r, \] where \(f(A,B)\equiv L(W_0+BA;D)\). The constraint forces the \(r\) columns of \(B\) to be mutually orthogonal [2508.17901]. In the fixed-rank-manifold variant, the optimization variable is the adapter \(\Delta W\) itself, viewed as a point on \(\mathcal M_r\). Because \(\mathcal M_r\subset\mathbb R^{d\times k}\) is a smooth submanifold, optimization on \(\mathcal M_r\) “automatically factors out” the ambiguity \[ (A,B)\mapsto(A\,S,B\,S^{-\top}) \] that leaves the low-rank product unchanged [2507.12142]. A common misconception is to treat these formulations as identical. They are not stated identically in the source material. One constrains a specific factor \(B\) to lie on a Stiefel manifold; the other treats the rank-\(r\) update matrix as the manifold point. This suggests that “RiemannLoRA” is best understood as a geometric family of LoRA optimizers rather than a single algorithmic template. ## 3. Differential-geometric structure For the Stiefel formulation, the manifold is \[ St(d,r)=\{B\in\mathbb R^{d\times r}: B^\top B=I_r\}, \] the compact Stiefel manifold of orthonormal \(d\times r\) frames. At any \(B\in St(d,r)\), the tangent space is \[ T_B St(d,r)=\{U\in\mathbb R^{d\times r}: B^\top U + U^\top B = 0\}, \] namely the first-order orthogonality-preserving directions [2508.17901]. For the fixed-rank-manifold formulation, if \[ X=A_LB^\top,\quad A_L^\top A_L=I_r,\;B^\top B=G\succ 0, \] then the tangent space at \(X\in\mathcal M_r\) is \[ T_X\mathcal M_r = \{\dot A\,B^\top + A_L\,\dot B^\top \mid A_L^\top\dot A=0\} \subset \mathbb R^{d\times k}. \] This describes admissible first-order perturbations that remain within the fixed-rank geometry [2507.12142]. The corresponding projections define the Riemannian gradients. On \(St(d,r)\), for any \(X\in\mathbb R^{d\times r}\) and \(\mathrm{Sym}(M)=(M+M^\top)/2\), \[ \mathrm{Proj}_{T_B}(X)=X-B\,\mathrm{Sym}(B^\top X), \] and hence \[ \mathrm{grad}_B f=(I-BB^\top)\,\nabla_B f. \] In the fixed-rank setting, for ambient \(Z\in\mathbb R^{d\times k}\), the paper gives \[ P_{T_X\mathcal M_r}[Z] =(I-A_LA_L^\top)\,Z\,B_RB_R^\top + A_LA_L^\top\,Z, \] as the simpler two-term form used in the paper, and the Riemannian gradient is \[ \mathrm{grad}\,\mathcal L = P_{T_X\mathcal M_r}\bigl[\nabla_{\Delta W}\mathcal L\bigr]. \] In both cases, the Euclidean gradient is not used directly; it is projected onto the relevant tangent space before the step is taken [2508.17901; 2507.12142]. ## 4. Optimization procedures and retractions The Stiefel-manifold method combines Adam-style preconditioning in ambient space with tangent projection and QR-based retraction. Given Euclidean gradients \(g_A=\nabla_A f\) and \(g_B=\nabla_B f\), \(A\) is updated with a standard Adam step, whereas \(B\) undergoes Adam-style moment estimation, projection to the tangent space, \[ \xi = M'_B - B\cdot \mathrm{Sym}(B^\top M'_B), \] and a QR retraction: \[ Y' = B-\alpha_B\,\xi,\qquad [Q,R]=qr(Y'),\qquad B\leftarrow Q. \] The retraction is stated as \(B^+=R_B(\xi)=Q\), with \(\mathrm{diag}(R)>0\). Vector transport is approximated by projecting a tangent vector onto the new tangent space [2508.17901]. The fixed-rank-manifold method uses a retraction defined by truncated SVD: \[ R_X(\xi)=\mathrm{truncSVD}_r(X+\xi)=U_r\,\Sigma_r\,V_r^\top. \] If \(\mathrm{rank}(X+\xi)\le 2r\), the truncated SVD can be computed in \[ \mathcal O((d+k)r^2+r^3) \] via low-rank techniques. The algorithmic outline includes BackPropRSVD on \(\nabla_W\mathcal L(W)\), a frozen complement \(W' = W-A_LB^\top\), QR re-orthonormalization of \(B\), gradient-factor computation via one backward pass, vector transport of previous momentum, formation of a Riemannian direction, optional simulated Adam normalization, and retraction through truncated SVD [2507.12142]. The source material emphasizes numerical stability in both variants. The Stiefel formulation uses project-to-tangent plus QR retraction as a lightweight Riemannian optimizer [2508.17901]. The fixed-rank formulation states that numerical stability is ensured by QR factorizations, robust truncated SVD retractions, and inversion-free formulas in transport and gradient steps [2507.12142]. A plausible implication is that both methods seek to preserve low-rank structure while avoiding the degeneracies that arise when factor matrices are updated independently in unconstrained Euclidean space. ## 5. Initialization and ambiguity-free optimization A central distinction between the fixed-rank-manifold formulation and the Stiefel-constrained formulation is the role of initialization. The fixed-rank-manifold paper explicitly derives a locally-optimal initialization: \[ \Delta W^{(0)} = \Argmax_{\Delta W\in\mathcal M_r} \bigl\|P_{T_{\Delta W}\mathcal M_r}\,\nabla_W\mathcal L(W)\bigr\|_F^2. \] By an Eckart–Young argument, if the full gradient has blockwise singular-vector decomposition \[ \nabla_W\mathcal L(W) = \underbrace{[\,U_{1:r}\;\;U_{r+1:2r}\;\;\cdots\,]}_{U} \begin{bmatrix}\Sigma_{1:r}\\\Sigma_{r+1:2r}\\\cdots\end{bmatrix} [\,V_{1:r}\;\;V_{r+1:2r}\;\;\cdots\,]^\top, \] then any optimal initializer has the form \[ \Delta W^{(0)} = \alpha\,U_{1:r}\,V_{r+1:2r}^\top,\quad \alpha\neq 0, \] and in practice \(\alpha=1\) is used [2507.12142]. This initialization is computed through a randomized SVD, accelerated by the “double-rank” backprop trick, QR orthonormalizations, and power iterations, yielding an \(\mathcal O((d+k)r^2)\) initialization cost [2507.12142]. The same paper frames this as resolving two issues simultaneously: the manifold viewpoint removes overparametrization, and the steepest-descent direction along the manifold prescribes initialization [2507.12142]. By contrast, the Stiefel-constrained work centers on the optimizer inefficiency arising from redundancy in \(B\) under AdamW and does not, in the supplied material, foreground a comparable initialization theorem [2508.17901]. A common misunderstanding is to assume that geometric optimization only changes the update rule after initialization. The fixed-rank-manifold formulation directly ties initialization to the manifold geometry. This suggests that, in that line of work, initialization is not a separate heuristic but part of the same intrinsic optimization framework. ## 6. Empirical findings The Stiefel-constrained paper reports a direct benchmark comparison between standard LoRA+AdamW and RiemannLoRA on seven commonsense reasoning datasets with rank \(r=16\) on LLaMA-3.2-1B [2508.17901]. | Model | Optimizer | Avg | |---|---|---| | LoRA (AdamW) | AdamW | 47.6 | | LoRA (Stiefel) | Stiefel | 59.7 | The same table gives dataset-wise scores: BoolQ \(63.2\rightarrow 75.2\), PIQA \(53.4\rightarrow 70.9\), SIQA \(50.1\rightarrow 65.3\), HellaSwag \(25.4\rightarrow 29.2\), ARC-e \(58.8\rightarrow 70.5\), ARC-c \(35.7\rightarrow 44.2\), and OBQA \(46.6\rightarrow 63.2\) [2508.17901]. The paper further states that on reading comprehension (SQuAD/QuAC) and math (GSM8K/MATH), RiemannLoRA consistently outperforms by 3–15 points. Figure 1 is described as showing that under AdamW the effective rank of \(\Delta W=BA\) is often \(