Stiefel-LoRA: Geometric Low-Rank Adaptation

• Stiefel-LoRA is a parameter-efficient fine-tuning method that enforces orthonormality on low-rank adapters via the Stiefel manifold.
• It leverages Riemannian optimization with tangent projection and QR/SVD retraction to maintain orthonormal constraints throughout training.
• Empirical results demonstrate enhanced accuracy, improved uncertainty calibration, and domain robustness with minimal computational overhead.

Stiefel-LoRA refers to a family of parameter-efficient fine-tuning (PEFT) methods for neural networks in which the low-rank adaptation matrices are structured via explicit orthonormal constraints on the Stiefel manifold. This geometric approach augments or replaces standard LoRA update schemes by enforcing that the learned subspaces for adaptation—typically the column spaces of the low-rank adapter factors—remain orthonormal throughout optimization, either via deterministic Riemannian optimization or within a Bayesian framework. The intent is to unlock the full representational and inferential power of low-rank adapters, enabling improved fine-tuning efficiency, reliability under domain shift, and, in the Bayesian setting, well-calibrated predictive uncertainty.

1. Mathematical Foundations and Formulation

The core idea in Stiefel-LoRA is to decompose adaptation updates using factors parameterized on the Stiefel manifold—the set of all $d\times r$ real matrices with orthonormal columns—thereby encoding orthonormal subspaces directly into adapter design.

In classical LoRA, a frozen pretrained weight $W_0\in\mathbb{R}^{d_\text{out}\times d_\text{in}}$ is updated by a low-rank addition,

$W = W_0 + A B^\top,$

where $A\in\mathbb{R}^{d_\text{out}\times k}$ and $B\in\mathbb{R}^{d_\text{in}\times k}$, typically for $k\ll\min(d_\text{out},d_\text{in})$.

Stiefel-LoRA instantiates geometry-aware variants as follows:

Variant	Adapter Parameterization	Orthonormal Constraints
Stiefel-LoRA	$B\in\mathrm{St}(d,r)$; $A$ unconstrained	$B^\top B=I_r$
StelLA (Li et al., 2 Oct 2025)	$USV^\top$, $W_0\in\mathbb{R}^{d_\text{out}\times d_\text{in}}$0, $W_0\in\mathbb{R}^{d_\text{out}\times d_\text{in}}$1, $W_0\in\mathbb{R}^{d_\text{out}\times d_\text{in}}$2	$W_0\in\mathbb{R}^{d_\text{out}\times d_\text{in}}$3, $W_0\in\mathbb{R}^{d_\text{out}\times d_\text{in}}$4
SBA (Shihab et al., 19 Feb 2026)	$W_0\in\mathbb{R}^{d_\text{out}\times d_\text{in}}$5; $W_0\in\mathbb{R}^{d_\text{out}\times d_\text{in}}$6 on Stiefel, $W_0\in\mathbb{R}^{d_\text{out}\times d_\text{in}}$7 diagonal	$W_0\in\mathbb{R}^{d_\text{out}\times d_\text{in}}$8, $W_0\in\mathbb{R}^{d_\text{out}\times d_\text{in}}$9

This structure is operationally equivalent to a dynamic SVD during optimization, separating adaptation directions (subspaces) from their scaling.

2. Riemannian Optimization on the Stiefel Manifold

Optimization over the Stiefel manifold necessitates respecting the orthonormality constraints throughout training. Riemannian gradient-based methods are employed:

• Tangent Space: At $W = W_0 + A B^\top,$0, the tangent space $W = W_0 + A B^\top,$1 consists of matrices $W = W_0 + A B^\top,$2 such that $W = W_0 + A B^\top,$3.
• Riemannian Gradient: For a loss $W = W_0 + A B^\top,$4, the ambient (Euclidean) gradient $W = W_0 + A B^\top,$5 is projected onto the tangent space:

$W = W_0 + A B^\top,$6

with $W = W_0 + A B^\top,$7.

• Retraction: Update steps are mapped back onto the manifold, commonly using the QR decomposition: $W = W_0 + A B^\top,$8 is decomposed as $W = W_0 + A B^\top,$9 and the $A\in\mathbb{R}^{d_\text{out}\times k}$0-factor provides the next iterate.

StelLA generalizes this paradigm by optimizing both $A\in\mathbb{R}^{d_\text{out}\times k}$1 and $A\in\mathbb{R}^{d_\text{out}\times k}$2 via projection and retraction, typically using SVD-based polar decomposition for retraction (Li et al., 2 Oct 2025).

3. Bayesian Stiefel-LoRA and Uncertainty Quantification

Stiefel-Bayes Adapters (SBA) (Shihab et al., 19 Feb 2026) introduce a Bayesian approach by defining matrix Langevin (von Mises–Fisher) priors over the Stiefel manifolds for $A\in\mathbb{R}^{d_\text{out}\times k}$3 and $A\in\mathbb{R}^{d_\text{out}\times k}$4. This prior has the form: $A\in\mathbb{R}^{d_\text{out}\times k}$5 with $A\in\mathbb{R}^{d_\text{out}\times k}$6 controlling prior concentration.

Posterior inference is approximated via tangent-space Laplace approximation:

• The posterior is locally approximated as Gaussian in the tangent space at the MAP estimate, with covariance set by the local Hessian.
• Geodesic retraction (QR or polar) maps samples from the Gaussian approximate posterior back to the manifold, generating distributions over $A\in\mathbb{R}^{d_\text{out}\times k}$7 and $A\in\mathbb{R}^{d_\text{out}\times k}$8.
• Predictive inference marginalizes over sampled adapters from this geometry-aware posterior.

A key theorem establishes that this intrinsic manifold-based approach avoids the variance inflation that plagues flat-space Gaussian projections, yielding posteriors that reflect only meaningful epistemic uncertainty tied to the geometry of the adapter subspaces.

4. Empirical Results and Comparative Analysis

Across standard NLP, vision, and generative benchmarks, Stiefel-LoRA variants display consistent advantages:

• Accuracy Improvements: Enforcing orthogonality of LoRA factors yields significant accuracy boosts. For instance, on commonsense reasoning with LLaMA-3B, Stiefel-LoRA improves accuracy from 78.9% (AdamW) to 82.4% (+4.5%). Mathematical reasoning shows even larger gains (e.g., 29.1% to 43.4% on GSM8K) (Park et al., 25 Aug 2025).
• Effective Rank and Orthogonality: Stiefel-LoRA achieves full utilization of adaptation rank: the effective rank of $A\in\mathbb{R}^{d_\text{out}\times k}$9 matches the nominal $B\in\mathbb{R}^{d_\text{in}\times k}$0, and all adapter subspace vectors remain almost perfectly orthogonal.
• Calibration and Uncertainty: SBA reduces Expected Calibration Error (ECE) by 18–34% relative to deterministic LoRA and outperforms both deep ensembles and Laplace/flat-space Bayesian baselines for OOD detection AUROC. Under domain shift, selective-prediction AUROC increases by 12–25% over deterministic baselines (Shihab et al., 19 Feb 2026).
• Compatibility and Overhead: The geometric constraints incur only modest additional computational cost (~5–15% overhead, dominated by QR or SVD per step) and are fully compatible with standard fine-tuning pipelines. SBA's Bayesian sampling costs are comparable to deep ensembles, but posterior distillation can recover most calibration benefits at single-model cost.

5. Algorithmic Implementation and Integration

Stiefel-LoRA variants are implemented via optimizer hooks that convert Euclidean optimizers (AdamW or others) into Riemannian counterparts:

• Gradient step: Euclidean gradients are projected to tangent space before update.
• Retraction: Adapter variables are projected back onto the Stiefel manifold with either QR or polar retraction (SVD-based). Batched SVD can process all adapted layers efficiently.
• Parameter Counting: Stiefel-LoRA maintains LoRA’s adapter parameter count, as only the factors and associated scales are optimized.
• Inference and Merging: At inference, the constrained adapter update $B\in\mathbb{R}^{d_\text{in}\times k}$1 or $B\in\mathbb{R}^{d_\text{in}\times k}$2 can be merged into $B\in\mathbb{R}^{d_\text{in}\times k}$3 with no run-time overhead. Bayesian variants draw samples per layer and aggregate predictions.

Stiffel-LoRA and StelLA both slot directly into HuggingFace PEFT libraries and PyTorch-based pipelines with minimal interface changes (Li et al., 2 Oct 2025).

6. Theoretical Insights and Limitations

By structuring adaptation on the Stiefel manifold:

• Columns of $B\in\mathbb{R}^{d_\text{in}\times k}$4 (or $B\in\mathbb{R}^{d_\text{in}\times k}$5, $B\in\mathbb{R}^{d_\text{in}\times k}$6) are guaranteed to form independent adaptation directions, avoiding basis collapse or redundancy prevalent in unconstrained LoRA. This fully utilizes the low-rank correction capacity.
• Momentum-based Riemannian optimization (AdamW followed by tangent projection) balances adaptive steps with geometric validity.
• Intrinsic manifold-based Bayesian inference avoids structural variance inflation—Gaussian+projection inflates tangent variance due to ambient noise mixing, strictly increasing KL divergence from the true posterior (Shihab et al., 19 Feb 2026).
• Practical retraction choices (QR, Cayley, exponential map) make negligible difference in calibration or accuracy.

Current limitations include additional SVD cost per step, minor relative overhead, and lack of comprehensive scaling evaluations for ultra-large models ($B\in\mathbb{R}^{d_\text{in}\times k}$7B parameters). Future work may focus on adaptive rank scheduling, quotient-manifold metrics, and extension to other PEFT modalities (Park et al., 25 Aug 2025, Li et al., 2 Oct 2025).

7. Relationship to Other PEFT Methods and Future Directions

Stiefel-LoRA generalizes and subsumes various recent trends:

• StelLA explicitly structures adapters as three-factor SVD-style decompositions with orthonormal subspaces for both input and output, achieving superior performance across language, vision, and generative tasks (Li et al., 2 Oct 2025).
• DoRA and related variants match LoRA in accuracy but do not encode calibrated uncertainty or leverage explicit geometric constraints.
• Bayesian Stiefel methods demonstrate that principled geometrization of the uncertainty structure yields epistemically meaningful and robust calibration, outperforming both flat Bayesian and ensemble/post-hoc calibration competitors under distribution shift and OOD scenarios (Shihab et al., 19 Feb 2026).

Ongoing research targets more efficient retraction methods, integration with rank-adaptive PEFT (e.g., AdaLoRA), and exploration of quotient-manifold metrics tailored for various adaptation regimes.

---

Key References:

• "Calibrated Adaptation: Bayesian Stiefel Manifold Priors for Reliable Parameter-Efficient Fine-Tuning" (Shihab et al., 19 Feb 2026)
• "Riemannian Optimization for LoRA on the Stiefel Manifold" (Park et al., 25 Aug 2025)
• "StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold" (Li et al., 2 Oct 2025)