Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR

Abstract: Low-rank adaptation (LoRA) and its variants enable parameter-efficient fine-tuning of LLMs under the supervised fine-tuning (SFT) paradigm. However, their efficacy and behavior under Reinforcement learning with verifiable rewards (RLVR) are less well understood. In particular, two structurally initialized LoRA variants, PiSSA and MiLoRA, which outperform standard LoRA under SFT, can underperform standard LoRA under RLVR and may even exhibit training instability. These observations suggest that how to initialize the low-rank matrices in RLVR remains unclear. In this work, we develop a theoretical analysis of LoRA in RLVR, showing that orthonormal initialization achieves the minimal gap between LoRA outcome and that of full fine-tuning. Guided by this insight, we propose geometry-preserving orthonormal initialization for low-rank adaptation in RLVR, leading to two new variants, RLPO and RLMO. Experiments on mathematical reasoning benchmarks show that the proposed orthonormal initialization stabilizes RLVR training and outperforms standard LoRA, contrasting with PiSSA and MiLoRA. Finally, our unified analysis for LoRA initialization also explains why PiSSA and MiLoRA can underperform in RLVR, which may be of independent interest. Code and checkpoints are publicly available at https://github.com/Richard-ZZZ/geometry-preserving-orthonormal-init-rlvr.

• The paper reveals that SVD-based LoRA initializations destabilize RLVR training by concentrating updates in limited subspaces and applying aggressive singular value scaling.
• It proposes geometry-preserving orthonormal initialization schemes that align with the spectral structure of pretrained weights while eliminating destabilizing factors.
• Empirical evaluations on mathematical reasoning benchmarks show these methods yield faster convergence, steadier KL dynamics, and enhanced test accuracy compared to standard LoRA.

Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR: A Technical Overview

Introduction

Parameter-efficient fine-tuning (PEFT) has become indispensable for adapting LLMs to diverse tasks when computational resources preclude full fine-tuning. Among PEFT approaches, Low-Rank Adaptation (LoRA) is widely adopted due to its effectiveness and simplicity. The LoRA paradigm constrains weight updates to a low-rank subspace via matrix factorization, greatly reducing trainable parameters but preserving expressive adaptation. However, while LoRA and its SVD-inspired variants (e.g., PiSSA, MiLoRA) have shown success in supervised fine-tuning (SFT), their behavior under reinforcement learning with verifiable rewards (RLVR) is less understood and, empirically, often unstable.

This paper ("Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR" (2606.31813)) provides a rigorous theoretical and empirical analysis of LoRA initialization in RLVR. It identifies the core issues behind SVD-based initialization instabilities, establishes fundamental theory for orthonormal initialization, and introduces new geometry-preserving LoRA variants that demonstrate improved performance and stability in RLVR-based mathematical reasoning tasks.

Dissection of SVD-Based LoRA Initialization Instability in RLVR

LoRA parameterizes weight updates $W = W_0 + BA$ with $B \in \mathbb{R}^{m \times r}$, $A \in \mathbb{R}^{r \times n}$, and $r \ll \min(m, n)$. Standard LoRA initializes $B_0 = 0$, $A_0$ as Gaussian random. SVD-based variants—most notably PiSSA and MiLoRA—utilize singular vectors from the pretrained model weights $W_0$: PiSSA selects top-$r$ (principal) singular vectors, MiLoRA targets the bottom-$r$ (minor) vectors, both with singular value scaling and $B_0 \neq 0$.

Under RLVR, empirical findings reveal that while PiSSA and MiLoRA accelerate SFT learning, they are frequently unstable in RLVR, with severe training reward collapse, high gradient norms, and excessive KL-divergence growth.

Figure 1: RLVR training with DAPO shows that PiSSA and MiLoRA incur training collapse, larger gradients, and KL divergence versus standard LoRA.

Subsequent spectral analyses demonstrate that SVD-based initializations do not merely affect the starting point but decisively anchor the entire optimization trajectory and the energy distribution of updates in the post-training solution.

Figure 2: LoRA update distributions after training are dominated by their initialized subspace, confirming that initialization shapes optimization geometry and spectral energy.

Theoretical Analysis: Orthonormal Initialization and Optimization Dynamics

The investigation establishes, both empirically and theoretically, that the instability of SVD-based LoRA variants under RLVR is attributed to two intertwined factors:

1. Subspace Geometry: Initialization in select singular subspaces (principal or minor) concentrates updates and restricts adaptation directionality, increasing risk of violating RLVR’s trust-region (KL-divergence) constraints.
2. Singular Value Scaling: Amplifies effective weight updates along the chosen directions, exacerbating overshooting and instability.

Through stepwise analytical comparison between PiSSA and an orthonormal LoRA variant (OLoRA), it is shown that singular value scaling alone can inflate gradient norms by an order determined by the largest involved singular value, thus rapidly pushing the policy outside the permissible KL neighborhood.

Figure 3: Removing singular value scaling by moving from PiSSA/MiLoRA to OLoRA/OLoRA-tail mitigates reward collapse and KL divergence, showing scaling is the primary source of RLVR instability.

A general theoretical bound is provided (Theorem 1): initializing $B \in \mathbb{R}^{m \times r}$0 as an orthonormal basis, with $B \in \mathbb{R}^{m \times r}$1, yields the minimal approximation gap between LoRA and the trajectory of full fine-tuning under smooth loss landscapes. This directly bounds policy update magnitudes and enforces a smoother optimization path consistent with RLVR’s conservative update requirements.

Geometry-Preserving Orthonormal LoRA: Method and Analysis

Building on this analysis, the paper introduces two new LoRA initialization schemes—geometry-preserving orthonormal principal and minor variants—which:

• Set $B \in \mathbb{R}^{m \times r}$2
• Initialize $B \in \mathbb{R}^{m \times r}$3 to the top-$B \in \mathbb{R}^{m \times r}$4 (principal) or bottom-$B \in \mathbb{R}^{m \times r}$5 (minor) right singular vectors of $B \in \mathbb{R}^{m \times r}$6, with row-orthonormality.

This approach leverages the geometric structure of the pretrained weights—retaining spectral priors without the destabilizing effect of singular value scaling. The two orthonormal methods deliver:

• Minimally increased KL-divergence during training
• Greater alignment of learned adapters with the intended spectral subspaces
• Substantially improved final task accuracy versus standard LoRA and SVD-based counterparts

  Figure 4: Geometry-preserving orthonormal methods ( , ) yield the lowest and most stable KL divergence among all initializations.

  

  Figure 5: Subspace similarity analysis confirms that geometry-preserving variants maintain spectral alignment throughout training, in contrast to model-agnostic orthonormal initializations.

Empirical Results: RLVR and Generalization

Evaluation across rigorous RLVR mathematical reasoning benchmarks (GSM8K, MATH500, AIME) with DAPO and GRPO algorithms demonstrates:

• Superior average test accuracy for geometry-preserving orthonormal methods, outperforming not only standard LoRA but also SVD-based PiSSA/MiLoRA.
• Training stability: Elimination of reward collapse or KL-divergence excursions.
• Rapid convergence: Orthogonal initialization also improves SFT efficiency, and achieves faster/lower training loss versus standard LoRA in supervised settings.

Performance gains are robust to model architecture, scale, and learning-rate schedule, and are validated for both principal and minor spectral variants.

Figure 6: LoRA-RLPO and LoRA-RLMO achieve both faster convergence and lower final loss than standard LoRA, confirming generalization to SFT.

Moreover, ablation with model-agnostic row-orthonormal initializations (DCT-LoRA, Wavelet-LoRA) shows that orthonormality alone confers improvement over standard LoRA, but leveraging SVD geometry in $B \in \mathbb{R}^{m \times r}$7 yields further benefit, particularly in the principal subspace.

Implications and Future Directions

Theoretical Implications: This work establishes a framework connecting initialization geometry, singular scaling, and constrained-update optimization in RLVR. The findings clarify that geometry-aware orthonormality, not merely random orthonormality, is critical for closing the gap to full fine-tuning while maintaining trust-region constraints. The counterproductive effect of singular value scaling for RLVR (despite its SFT benefits) highlights that PEFT design principles are task-regime specific.

Practical Implications: Geometry-preserving orthonormal initialization provides a robust, principled drop-in replacement for SVD-based LoRA under RLVR, requiring only SVD preprocessing (one-time, inexpensive relative to training) and delivering strong accuracy, efficiency, and stability gains.

Future Prospects: Ongoing work will likely extend these orthonormal geometric priors to higher-order optimization, mixture-of-experts PEFT architectures, and dynamic rank/allocation methods. Beyond RLVR, application to continual learning, task adaptation, and more aggressive distribution shifts may prompt further theoretical exploration of orthonormality and spectral geometry in PEFT for foundation models.

Conclusion

This paper delivers the first principled, unified theoretical and empirical explanation for the instability of SVD-based LoRA variants in RLVR. It establishes that geometry-preserving orthonormal initialization, aligned to the spectral structure of pretrained weights but eschewing singular value scaling and non-zero $B \in \mathbb{R}^{m \times r}$8, stabilizes RLVR optimization and improves accuracy. These insights should guide the next generation of parameter-efficient adaptation methods for reinforcement learning in LLMs, bridging the gap between efficient PEFT and the dynamic, constraint-driven optimization regimes of RLVR.