Rank-Stabilized LoRA (rsLoRA)

• Rank-Stabilized LoRA (rsLoRA) is a parameter-efficient fine-tuning method that uses a 1/sqrt(r) scaling to maintain stable activations and gradients across diverse adapter ranks.
• It addresses the gradient collapse issue of standard LoRA by ensuring stable learning dynamics, which improves performance as adapter rank increases, as validated on models like Llama 2.
• The method achieves efficient adaptation without additional inference cost and extends to federated and privacy-preserving settings via adaptations such as FedSVD.

Rank-Stabilized LoRA (rsLoRA) is an improved parameter-efficient fine-tuning (PEFT) methodology for LLMs and other deep neural architectures. It addresses a critical limitation in the canonical Low-Rank Adapter (LoRA) approach, specifically the rank-dependent scaling factor that hinders effective adaptation for higher-rank adapters. By replacing the previously used scaling factor proportional to $1/r$ with a theoretically derived $1/\sqrt{r}$ scaling, rsLoRA enables stable and efficient learning dynamics across a much wider range of adapter ranks, thus facilitating better compute/performance trade-offs without increasing inference costs (Kalajdzievski, 2023).

1. Formulation and Motivation

The standard LoRA method augments a frozen pretrained weight matrix $W \in \mathbb{R}^{d_2 \times d_1}$ with a low-rank correction $\Delta W$, parameterized as

$\Delta W = \gamma \cdot B A,$

where $B \in \mathbb{R}^{d_2 \times r}$, $A \in \mathbb{R}^{r \times d_1}$, and $r \ll \min(d_1, d_2)$. The scaling factor $\gamma$ is typically set as $\alpha / r$, with $1/\sqrt{r}$0 a constant hyperparameter.

Empirical and theoretical analysis reveal that as $1/\sqrt{r}$1 increases, the $1/\sqrt{r}$2 scaling causes gradients with respect to $1/\sqrt{r}$3 and $1/\sqrt{r}$4 to collapse as $1/\sqrt{r}$5, dramatically slowing adaptation—effectively nullifying the potential benefits of using higher adapter ranks. Empirically, increasing $1/\sqrt{r}$6 in standard LoRA beyond small values (e.g., $1/\sqrt{r}$7) does not improve learning, with loss curves saturating and matching the low-rank case. The rsLoRA framework is motivated by the need to stabilize both the magnitude of forward-pass activations and backward-pass gradients as $1/\sqrt{r}$8 grows (Kalajdzievski, 2023).

2. Theoretical Foundation for $1/\sqrt{r}$9 Scaling

To ensure activations and gradients remain $W \in \mathbb{R}^{d_2 \times d_1}$0 as $W \in \mathbb{R}^{d_2 \times d_1}$1, the scaling $W \in \mathbb{R}^{d_2 \times d_1}$2 is analytically established:

• Forward-pass: Under standard initializations ($W \in \mathbb{R}^{d_2 \times d_1}$3 zeros, $W \in \mathbb{R}^{d_2 \times d_1}$4), the variance of output activations due to $W \in \mathbb{R}^{d_2 \times d_1}$5 is proportional to $W \in \mathbb{R}^{d_2 \times d_1}$6. Ensuring $W \in \mathbb{R}^{d_2 \times d_1}$7 requires $W \in \mathbb{R}^{d_2 \times d_1}$8, so $W \in \mathbb{R}^{d_2 \times d_1}$9.
• Backward-pass: Gradient magnitudes for $\Delta W$0 and $\Delta W$1 similarly scale with $\Delta W$2, with norms in $\Delta W$3 and $\Delta W$4 growing as $\Delta W$5. Stability again requires $\Delta W$6, enforcing the same scaling.

The main theoretical result (see Appendix, (Kalajdzievski, 2023)) is that only $\Delta W$7 simultaneously bounds the moments of both activations and gradients for arbitrary rank $\Delta W$8. Faster decay (such as $\Delta W$9) collapses gradients; slower ($\Delta W = \gamma \cdot B A,$0) causes exploding activations or gradients.

Definition (Rank-Stabilized Adapter): An adapter $\Delta W = \gamma \cdot B A,$1 is rank-stabilized if for all orders $\Delta W = \gamma \cdot B A,$2:

1. $\Delta W = \gamma \cdot B A,$3,
2. $\Delta W = \gamma \cdot B A,$4

This is provably only satisfied by $\Delta W = \gamma \cdot B A,$5.

3. Implementation Details and Pseudocode

The rsLoRA workflow modifies only the scaling of the adapter relative to the canonical LoRA algorithm. Concretely:

$\alpha / r$7

The critical difference: set $\Delta W = \gamma \cdot B A,$6 rather than $\Delta W = \gamma \cdot B A,$7.

Hyperparameters:

• Rank $\Delta W = \gamma \cdot B A,$8: Select to match GPU budget. Effective range: 4–1024; higher ranks (256–2048) unlock better fine-tuning when rsLoRA is used.
• Scaling $\Delta W = \gamma \cdot B A,$9: Default $B \in \mathbb{R}^{d_2 \times r}$0. For $B \in \mathbb{R}^{d_2 \times r}$1, tuning in $B \in \mathbb{R}^{d_2 \times r}$2 is suggested.
• Learning rate $B \in \mathbb{R}^{d_2 \times r}$3: As in standard LoRA (e.g., AdamW $B \in \mathbb{R}^{d_2 \times r}$4).
• Initialization $B \in \mathbb{R}^{d_2 \times r}$5: Use $B \in \mathbb{R}^{d_2 \times r}$6 as in standard LoRA, or $B \in \mathbb{R}^{d_2 \times r}$7.

4. Empirical Results and Performance

Experiments with Llama 2 (7B), using the OpenOrca dataset (20k examples, perplexity metric):

• Standard LoRA ($B \in \mathbb{R}^{d_2 \times r}$8): Perplexity saturates at $B \in \mathbb{R}^{d_2 \times r}$9 for all $A \in \mathbb{R}^{r \times d_1}$0; no improvement after $A \in \mathbb{R}^{r \times d_1}$1.
• rsLoRA ($A \in \mathbb{R}^{r \times d_1}$2): Higher ranks progressively improve perplexity: $A \in \mathbb{R}^{r \times d_1}$3 (1.88), $A \in \mathbb{R}^{r \times d_1}$4 (1.87), $A \in \mathbb{R}^{r \times d_1}$5 (1.84), $A \in \mathbb{R}^{r \times d_1}$6 (1.82).

Gradient-norm diagnostics:

• Standard LoRA: $A \in \mathbb{R}^{r \times d_1}$7 collapses as $A \in \mathbb{R}^{r \times d_1}$8, leading to extremely slow adaptation at larger $A \in \mathbb{R}^{r \times d_1}$9.
• rsLoRA: Gradient norms are $r \ll \min(d_1, d_2)$0 and stable for all $r \ll \min(d_1, d_2)$1.

Additional ablations confirm:

• Scaling only initialization by $r \ll \min(d_1, d_2)$2, but not the adapter, does not resolve the collapse.
• Alternative scaling laws (e.g., $r \ll \min(d_1, d_2)$3, $r \ll \min(d_1, d_2)$4) either explode or collapse activations/gradients more severely.
• Restricting LoRA adapters only to attention sublayers preserves the rsLoRA qualitative improvement.

This suggests the benefits of rsLoRA generalize across architectures, datasets, and optimizer choices (Kalajdzievski, 2023).

5. Practical Guidelines and Limitations

Adoption and settings:

• Use rsLoRA whenever high adapter rank ($r \ll \min(d_1, d_2)$5) is desired to exploit available training compute for improved adaptation, incurring no extra inference cost.
• Recommended rank: $r \ll \min(d_1, d_2)$6–$r \ll \min(d_1, d_2)$7 for most scenarios; increase to $r \ll \min(d_1, d_2)$8–$r \ll \min(d_1, d_2)$9 if memory budget allows.
• Maintain conventional learning rates and optimization schedules used for transformer fine-tuning.
• No further changes to training paradigms, optimizers, or initialization necessary.

Observed benefits:

• Fine-tuning loss/perplexity reductions up to several percentage points as $\gamma$0 increases from $\gamma$1 to $\gamma$2.
• rsLoRA achieves performance comparable to or better than full fine-tuning for many NLP tasks, with less than $\gamma$3–$\gamma$4 of model parameters trainable.

Limitations:

• For downstream tasks with very low intrinsic dimension ($\gamma$5), increased $\gamma$6 gives diminishing returns.
• rsLoRA addresses only the rank-based scaling issue; it does not mitigate challenges such as domain shift or catastrophic forgetting.

6. Relationship to Federated and Private Settings

While rsLoRA resolves rank-scaling issues in local and centralized applications, when deployed in federated learning with differential privacy mechanisms such as DP-SGD, further adaptation is necessary due to noise amplification through matrix multiplications in LoRA updates:

• Quadratic noise terms ($\gamma$7) arise when both $\gamma$8 and $\gamma$9 are locally adapted and independently perturbed on each client.
• Freezing one matrix (typically $\alpha / r$0) restricts expressiveness and degrades adaptation.

The FedSVD method, introduced in "FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA" (Lee et al., 19 May 2025), orthogonalizes one adapter ($\alpha / r$1) via truncated SVD of the aggregated product $\alpha / r$2 server-side after each communication round. This ensures:

• Only linear noise amplification occurs; the problematic $\alpha / r$3 cross term is eliminated.
• Orthonormal $\alpha / r$4 ensures gradient norm preservation under DP-SGD clipping and improves the conditioning of client optimization.
• Global SVD-based adaptation of $\alpha / r$5 recovers expressiveness lost in fixed-matrix strategies, delivering improved accuracy and stability under DP constraints.

Empirically, FedSVD achieves 86.27% average accuracy on non-private settings and 76.79% under ($\alpha / r$6) DP-SGD, outperforming other PEFT methods by substantial margins on GLUE benchmarks (Lee et al., 19 May 2025).

7. Impact and Significance

rsLoRA establishes a robust scaling prescription for low-rank adapters, correcting the core deficiency limiting the practical use of higher rank in LoRA-based PEFT. This provides researchers and practitioners with a tunable compute/performance trade-off, enabling efficient model adaptation in scenarios ranging from few-shot supervised tasks to large-sample fine-tuning. Its theoretical foundation ensures stable signal propagation and adaptable learning rates for modern deep models. In federated and privacy-preserving contexts, extensions such as FedSVD provide algorithmic solutions to new sources of instability induced by private noise injection, further broadening the impact and applicability of the rsLoRA scaling regime (Kalajdzievski, 2023, Lee et al., 19 May 2025).