RiemannLoRA: Geometric Low-Rank Adaptation

• RiemannLoRA is a family of geometric methods that reformulate low-rank adaptation by replacing Euclidean updates with manifold-aware optimization on either the Stiefel or fixed-rank matrix manifolds.
• It addresses inherent issues like basis redundancy and factorization ambiguity in traditional LoRA, resulting in improved convergence, effective rank utilization, and performance efficiency.
• Empirical evaluations show that RiemannLoRA outperforms standard LoRA in benchmarks such as commonsense reasoning and fine-tuning, with faster convergence and enhanced numerical stability.

RiemannLoRA denotes a family of geometric reformulations of Low-Rank Adaptation (LoRA) in which low-rank adapters are optimized with explicit Riemannian structure rather than as unconstrained Euclidean factors. In the 2025 literature, the name is used for two closely related but distinct constructions. One imposes an orthonormality constraint on the LoRA factor $B$ and optimizes it on the compact Stiefel manifold $St(d,r)$, with the stated goal of eliminating basis redundancy and restoring full effective rank in $\Delta W = BA$ (Park et al., 25 Aug 2025). The other treats the set of all rank-$r$ updates itself as a smooth manifold $\mathcal M_r$, thereby removing the factorization ambiguity $(A,B)\mapsto(AS,B\,S^{-\top})$, defining a manifold steepest-descent direction, and deriving a locally-optimal initialization (Bogachev et al., 16 Jul 2025). Both formulations preserve the parameter-efficient fine-tuning setting of LoRA while replacing standard Euclidean optimization with manifold-aware updates.

1. LoRA setting and the motivation for a geometric reformulation

In the Stiefel-manifold formulation, a frozen pre-trained weight matrix in an LLM layer is written as $W_0\in\mathbb R^{d\times k}$, and LoRA replaces a full fine-tuning update $\Delta W$ with a rank-$r$ factorization

$\Delta W = B\,A,$

where $St(d,r)$0, $St(d,r)$1, and $St(d,r)$2. The fine-tuned weight is

$St(d,r)$3

Standard LoRA then solves the unconstrained minimization

$St(d,r)$4

in Euclidean space, typically with AdamW (Park et al., 25 Aug 2025).

The fixed-rank-manifold formulation uses the equivalent low-rank viewpoint

$St(d,r)$5

and identifies all such rank-$St(d,r)$6 matrices with

$St(d,r)$7

a smooth embedded manifold of dimension $St(d,r)$8 (Bogachev et al., 16 Jul 2025).

The shared motivation is that standard LoRA inherits optimizer pathologies from the matrix factorization itself. One work states that LoRA suffers from “basis redundancy in LoRA’s $St(d,r)$9 matrix when using AdamW,” and that this fundamentally limits performance (Park et al., 25 Aug 2025). The other emphasizes “overparametrization in low-rank matrix factorization” and the resulting parametrization ambiguity (Bogachev et al., 16 Jul 2025). Taken together, these formulations recast LoRA training as a problem in constrained or intrinsic optimization over low-rank geometry rather than over arbitrary factors.

2. Two formulations of RiemannLoRA

The two papers use the same name but organize the geometry differently.

Formulation	Geometric object	Principal effect
Stiefel-constrained LoRA	$\Delta W = BA$0 with $\Delta W = BA$1	Enforces orthogonality of LoRA basis vectors
Fixed-rank-manifold LoRA	$\Delta W = BA$2 with $\Delta W = BA$3	Removes factorization ambiguity intrinsically

In the Stiefel-constrained variant, the objective is

$\Delta W = BA$4

where $\Delta W = BA$5. The constraint forces the $\Delta W = BA$6 columns of $\Delta W = BA$7 to be mutually orthogonal (Park et al., 25 Aug 2025).

In the fixed-rank-manifold variant, the optimization variable is the adapter $\Delta W = BA$8 itself, viewed as a point on $\Delta W = BA$9. Because $r$0 is a smooth submanifold, optimization on $r$1 “automatically factors out” the ambiguity

$r$2

that leaves the low-rank product unchanged (Bogachev et al., 16 Jul 2025).

A common misconception is to treat these formulations as identical. They are not stated identically in the source material. One constrains a specific factor $r$3 to lie on a Stiefel manifold; the other treats the rank-$r$4 update matrix as the manifold point. This suggests that “RiemannLoRA” is best understood as a geometric family of LoRA optimizers rather than a single algorithmic template.

3. Differential-geometric structure

For the Stiefel formulation, the manifold is

$r$5

the compact Stiefel manifold of orthonormal $r$6 frames. At any $r$7, the tangent space is

$r$8

namely the first-order orthogonality-preserving directions (Park et al., 25 Aug 2025).

For the fixed-rank-manifold formulation, if

$r$9

then the tangent space at $\mathcal M_r$0 is

$\mathcal M_r$1

This describes admissible first-order perturbations that remain within the fixed-rank geometry (Bogachev et al., 16 Jul 2025).

The corresponding projections define the Riemannian gradients. On $\mathcal M_r$2, for any $\mathcal M_r$3 and $\mathcal M_r$4,

$\mathcal M_r$5

and hence

$\mathcal M_r$6

In the fixed-rank setting, for ambient $\mathcal M_r$7, the paper gives

$\mathcal M_r$8

as the simpler two-term form used in the paper, and the Riemannian gradient is

$\mathcal M_r$9

In both cases, the Euclidean gradient is not used directly; it is projected onto the relevant tangent space before the step is taken (Park et al., 25 Aug 2025, Bogachev et al., 16 Jul 2025).

4. Optimization procedures and retractions

The Stiefel-manifold method combines Adam-style preconditioning in ambient space with tangent projection and QR-based retraction. Given Euclidean gradients $(A,B)\mapsto(AS,B\,S^{-\top})$0 and $(A,B)\mapsto(AS,B\,S^{-\top})$1, $(A,B)\mapsto(AS,B\,S^{-\top})$2 is updated with a standard Adam step, whereas $(A,B)\mapsto(AS,B\,S^{-\top})$3 undergoes Adam-style moment estimation, projection to the tangent space,

$(A,B)\mapsto(AS,B\,S^{-\top})$4

and a QR retraction: $(A,B)\mapsto(AS,B\,S^{-\top})$5 The retraction is stated as $(A,B)\mapsto(AS,B\,S^{-\top})$6, with $(A,B)\mapsto(AS,B\,S^{-\top})$7. Vector transport is approximated by projecting a tangent vector onto the new tangent space (Park et al., 25 Aug 2025).

The fixed-rank-manifold method uses a retraction defined by truncated SVD: $(A,B)\mapsto(AS,B\,S^{-\top})$8 If $(A,B)\mapsto(AS,B\,S^{-\top})$9, the truncated SVD can be computed in

$W_0\in\mathbb R^{d\times k}$0

via low-rank techniques. The algorithmic outline includes BackPropRSVD on $W_0\in\mathbb R^{d\times k}$1, a frozen complement $W_0\in\mathbb R^{d\times k}$2, QR re-orthonormalization of $W_0\in\mathbb R^{d\times k}$3, gradient-factor computation via one backward pass, vector transport of previous momentum, formation of a Riemannian direction, optional simulated Adam normalization, and retraction through truncated SVD (Bogachev et al., 16 Jul 2025).

The source material emphasizes numerical stability in both variants. The Stiefel formulation uses project-to-tangent plus QR retraction as a lightweight Riemannian optimizer (Park et al., 25 Aug 2025). The fixed-rank formulation states that numerical stability is ensured by QR factorizations, robust truncated SVD retractions, and inversion-free formulas in transport and gradient steps (Bogachev et al., 16 Jul 2025). A plausible implication is that both methods seek to preserve low-rank structure while avoiding the degeneracies that arise when factor matrices are updated independently in unconstrained Euclidean space.

5. Initialization and ambiguity-free optimization

A central distinction between the fixed-rank-manifold formulation and the Stiefel-constrained formulation is the role of initialization. The fixed-rank-manifold paper explicitly derives a locally-optimal initialization: $W_0\in\mathbb R^{d\times k}$4 By an Eckart–Young argument, if the full gradient has blockwise singular-vector decomposition

$W_0\in\mathbb R^{d\times k}$5

then any optimal initializer has the form

$W_0\in\mathbb R^{d\times k}$6

and in practice $W_0\in\mathbb R^{d\times k}$7 is used (Bogachev et al., 16 Jul 2025).

This initialization is computed through a randomized SVD, accelerated by the “double-rank” backprop trick, QR orthonormalizations, and power iterations, yielding an $W_0\in\mathbb R^{d\times k}$8 initialization cost (Bogachev et al., 16 Jul 2025).

The same paper frames this as resolving two issues simultaneously: the manifold viewpoint removes overparametrization, and the steepest-descent direction along the manifold prescribes initialization (Bogachev et al., 16 Jul 2025). By contrast, the Stiefel-constrained work centers on the optimizer inefficiency arising from redundancy in $W_0\in\mathbb R^{d\times k}$9 under AdamW and does not, in the supplied material, foreground a comparable initialization theorem (Park et al., 25 Aug 2025).

A common misunderstanding is to assume that geometric optimization only changes the update rule after initialization. The fixed-rank-manifold formulation directly ties initialization to the manifold geometry. This suggests that, in that line of work, initialization is not a separate heuristic but part of the same intrinsic optimization framework.

6. Empirical findings

The Stiefel-constrained paper reports a direct benchmark comparison between standard LoRA+AdamW and RiemannLoRA on seven commonsense reasoning datasets with rank $\Delta W$0 on LLaMA-3.2-1B (Park et al., 25 Aug 2025).

Model	Optimizer	Avg
LoRA (AdamW)	AdamW	47.6
LoRA (Stiefel)	Stiefel	59.7

The same table gives dataset-wise scores: BoolQ $\Delta W$1, PIQA $\Delta W$2, SIQA $\Delta W$3, HellaSwag $\Delta W$4, ARC-e $\Delta W$5, ARC-c $\Delta W$6, and OBQA $\Delta W$7 (Park et al., 25 Aug 2025). The paper further states that on reading comprehension (SQuAD/QuAC) and math (GSM8K/MATH), RiemannLoRA consistently outperforms by 3–15 points. Figure 1 is described as showing that under AdamW the effective rank of $\Delta W$8 is often $\Delta W$9, whereas Stiefel-LoRA achieves full effective rank $r$0. Figure 2 reports large variance in layer-wise mean cosine similarity of $r$1's columns under AdamW, with $r$2, while Stiefel-LoRA maintains exact orthogonality with $r$3. Figure 3 states that only RiemannLoRA fully utilizes all $r$4 dimensions in the Shannon-entropy-based layer-wise effective rank (Park et al., 25 Aug 2025).

The fixed-rank-manifold paper reports LLM fine-tuning results on Llama 3.2 1B, rank 16, over eight commonsense tasks: BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Chall, and OBQA. It reports $r$5 average accuracy for RiemannLoRA-LOI + SGD versus $r$6 for LoRA, and $r$7 average for RiemannLoRA-LOI + Adam versus $r$8 for best LoRA-LOI, but with larger variance for the latter (Bogachev et al., 16 Jul 2025). It also states that train loss halves in approximately $r$9 fewer steps, that variance across random seeds is lower, and that memory is the same as standard LoRA, namely $\Delta W = B\,A,$0 parameters (Bogachev et al., 16 Jul 2025).

Beyond LLM fine-tuning, the fixed-rank-manifold work reports subject-driven diffusion generation results with Stable Diffusion 2 on a DreamBooth subset of 15 concepts. Using CLIP-based image-to-reference (BaseIS) and text-condition (TS) cosine similarities, it states that RiemannLoRA reaches target similarities in about 600 steps versus about 1400 for LoRA, with higher final similarity and better visual fidelity (Bogachev et al., 16 Jul 2025).

7. Interpretation, significance, and points of clarification

The Stiefel-constrained paper attributes its gains to four mechanisms: elimination of basis redundancy, maximization of effective rank, stabilization of training dynamics, and enhanced parameter efficiency, with the specific claim that fewer parameters achieve the same or better performance, reducing the need to increase $\Delta W = B\,A,$1 (Park et al., 25 Aug 2025). Its summary states that the geometric constraint removes representational redundancy in LoRA, yielding faster convergence, higher effective rank, and consistent performance gains over Euclidean-trained LoRA (Park et al., 25 Aug 2025).

The fixed-rank-manifold paper frames the significance somewhat differently. It argues that standard LoRA updates $\Delta W = B\,A,$2 independently, “effectively moving in only a subpart of the tangent space,” whereas RiemannLoRA finds the true steepest-descent direction in the full tangent space. It also emphasizes that the manifold viewpoint removes ambiguity and that initialization from the manifold geometry aligns the first step with the full-model gradient (Bogachev et al., 16 Jul 2025).

These accounts are compatible at the level of geometric intent but differ in emphasis. One focuses on orthogonality in a factor matrix $\Delta W = B\,A,$3; the other focuses on intrinsic optimization over the manifold of fixed-rank matrices. This suggests that the most precise encyclopedic characterization of RiemannLoRA is not a single canonical update rule, but a 2025 line of work that replaces Euclidean LoRA optimization with Riemannian optimization to address redundancy, ambiguity, and inefficient use of rank.

A further point of clarification concerns cost. The fixed-rank-manifold paper states that the per-step cost is of the same order as standard LoRA and gives $\Delta W = B\,A,$4 plus one or two back-prop calls for each step (Bogachev et al., 16 Jul 2025). The Stiefel-constrained paper characterizes its optimizer as lightweight, based on tangent projection, QR retraction, and Adam-style preconditioning (Park et al., 25 Aug 2025). Neither source describes geometric optimization as free; rather, both present additional QR- or SVD-based structure as the computational price for enforcing the low-rank geometry more faithfully.

As of the cited 2025 works, RiemannLoRA therefore refers to a geometric turn in parameter-efficient fine-tuning: either by constraining LoRA factors to a Stiefel manifold or by treating the adapter itself as a point on the smooth manifold of rank-$\Delta W = B\,A,$5 matrices. In both versions, the central thesis is that respecting the intrinsic geometry of low-rank updates materially changes the optimization behavior of LoRA (Park et al., 25 Aug 2025, Bogachev et al., 16 Jul 2025).