Stable-LoRA: Low-Rank Adaptation Stability

• Stable-LoRA is a stabilization technique for LoRA fine-tuning that progressively shrinks the low-rank matrix during early training steps.
• It counteracts training instability by balancing non-zero initialization effects with controlled gradient accumulation to avoid exploding or vanishing updates.
• The approach is versatile, being applied in LLM optimization, diffusion acceleration, and tensor-parallel serving, with minimal overhead and improved performance.

Searching arXiv for papers on “Stable-LoRA” and closely related usages. Stable-LoRA most commonly denotes a modification of Low-Rank Adaptation (LoRA) for LLM fine-tuning in which the trainable low-rank matrix $A$ is progressively shrunk during the earliest optimization steps in order to stabilize feature learning while retaining the optimization benefits of the standard non-zero initialization (Wu et al., 5 Mar 2026). In adjacent literatures, the same expression or closely related variants also refer to LoRA-enabled Stable Diffusion personalization, LoRA-based diffusion acceleration, and the S-LoRA multi-adapter serving framework for tensor-parallel inference (Peter et al., 28 Feb 2025, Luo et al., 2023, Wang et al., 27 Oct 2025). Among these usages, the 2026 paper is the one that gives “Stable-LoRA” an explicit theoretical and algorithmic definition as a PEFT stabilization method.

1. Terminological scope

The recent literature uses “Stable-LoRA” in several non-identical ways. This has produced a mild naming ambiguity: in some papers the term refers to a specific optimization strategy for LoRA training, while in others it is a shorthand for Stable Diffusion combined with LoRA or for the S-LoRA serving system.

Usage in the literature	Referent	Core idea
“Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptation” (Wu et al., 5 Mar 2026)	Specific LoRA optimization method	Early-step shrinkage of $A$ to stabilize feature learning
“Advancing AI-Powered Medical Image Synthesis...” (Peter et al., 28 Feb 2025)	Stable Diffusion + LoRA / DreamBooth + LoRA	Parameter-efficient medical image synthesis
“LCM-LoRA: A Universal Stable-Diffusion Acceleration Module” (Luo et al., 2023)	LoRA-based LCM distillation for SD	Few-step diffusion acceleration
“Block-Diagonal LoRA for Eliminating Communication Overhead in Tensor Parallel LoRA Serving” (Wang et al., 27 Oct 2025)	S-LoRA serving context	Multi-adapter serving under tensor parallelism

A common misconception is that Stable-LoRA necessarily refers to Stable Diffusion. The literature does not support that as a universal reading. In the PEFT theory setting, Stable-LoRA is a training strategy for the canonical LoRA parameterization $W = W_0 + sBA$; in diffusion, “Stable-LoRA” is often an informal label for Stable Diffusion adapted with LoRA modules; in systems work, S-LoRA denotes a serving architecture rather than a training algorithm (Wu et al., 5 Mar 2026, Peter et al., 28 Feb 2025, Wang et al., 27 Oct 2025).

2. LoRA formulation and the stability problem

In the formal Stable-LoRA method, the starting point is standard LoRA. A frozen pretrained weight matrix $W_0$ is augmented by a low-rank update,

$W = W_0 + sBA,$

where $A \in \mathbb{R}^{r \times n}$, $B \in \mathbb{R}^{m \times r}$, $r \ll \min(m,n)$, and $s$ is a scaling factor (Wu et al., 5 Mar 2026). With learning rate $\eta$, the LoRA parameters are updated as

$A$0

The paper defines the learned feature at step $A$1 through the LoRA-induced change in output,

$A$2

and derives

$A$3

This decomposition is central because the notion of stability is output-centric rather than parameter-centric: LoRA feature learning is stable when $A$4 as width $A$5 grows. The objective is therefore to prevent the low-rank update from either vanishing or exploding with width.

The instability diagnosed by Stable-LoRA arises from the tension between theory and practice. The analysis identifies $A$6 and $A$7 as the initialization that best guarantees self-stability, but the same paper states that this practical regime is undesirable because it creates saddle-point halting and information loss: with $A$8, the $A$9 branch initially receives zero input (Wu et al., 5 Mar 2026). Standard LoRA therefore uses $W = W_0 + sBA$0 and non-zero $W = W_0 + sBA$1, typically He-style initialization. Stable-LoRA is designed to preserve that empirical choice while removing its long-term destabilizing effect.

3. Self-stability theory and the role of initialization

The theoretical analysis introduces a $W = W_0 + sBA$2-function notation in which a quantity $W = W_0 + sBA$3 scaling as $W = W_0 + sBA$4 is tracked through training. Stability is then the condition

$W = W_0 + sBA$5

Under two assumptions—optimized gradients are scale-stable, $W = W_0 + sBA$6, and gradient-input interaction satisfies $W = W_0 + sBA$7—the three terms in $W = W_0 + sBA$8 can be analyzed by width scaling (Wu et al., 5 Mar 2026).

The resulting self-stability theorem states that LoRA can naturally achieve and sustain stable feature learning if the hyperparameters satisfy

$W = W_0 + sBA$9

and the initialization obeys

$W_0$0

This is the “good case” in which the long-term dynamics are governed by gradient updates rather than by oversized initialization terms.

The paper’s principal negative result is that the practical non-zero initialization of $W_0$1 breaks these conditions for realistic small learning rates. Because $W_0$2 is non-zero, the inequality

$W_0$3

can fail, forcing training into a regime where the initialization of $W_0$4 continues to dominate. Empirically, this appears as $W_0$5 remaining relatively large while $W_0$6 grows rapidly from zero, causing the term

$W_0$7

to dominate

$W_0$8

A common misconception is that the scaling factor $W_0$9 alone can repair this problem. The analysis explicitly shows that the critical condition on $W = W_0 + sBA,$0 does not contain $W = W_0 + sBA,$1; the instability is tied to the interaction of initialization and learning-rate scale, not merely to the LoRA scaling hyperparameter (Wu et al., 5 Mar 2026).

4. Stable-LoRA algorithm

Stable-LoRA modifies ordinary LoRA with a weight-shrinkage optimization applied only to $W = W_0 + sBA,$2 and only during the earliest training phase. Before the gradient step, if training is not yet considered stable, the method shrinks $W = W_0 + sBA,$3 as

$W = W_0 + sBA,$4

where $W = W_0 + sBA,$5 is a small shrinkage ratio (Wu et al., 5 Mar 2026). Usual gradient updates then proceed, optionally with weight decay: $W = W_0 + sBA,$6

The stopping rule is norm-based. Shrinkage continues while

$W = W_0 + sBA,$7

and stops once

$W = W_0 + sBA,$8

The paper interprets this criterion as a practical proxy for balancing the two dominant contributions $W = W_0 + sBA,$9 and $A \in \mathbb{R}^{r \times n}$0. Once the average scales of $A \in \mathbb{R}^{r \times n}$1 and $A \in \mathbb{R}^{r \times n}$2 are comparable, the destabilizing influence of the initial $A \in \mathbb{R}^{r \times n}$3 has been sufficiently neutralized.

Unrolling the shrinkage recurrence yields

$A \in \mathbb{R}^{r \times n}$4

The significance of this expression is that the effect of the initial $A \in \mathbb{R}^{r \times n}$5 decays as $A \in \mathbb{R}^{r \times n}$6, whereas later gradients accumulate. This provides the mechanism by which a non-zero $A \in \mathbb{R}^{r \times n}$7 can be useful at the beginning of training but cease to dominate after sufficient early shrinkage.

The method changes neither the LoRA parameterization nor the memory footprint. It is therefore distinct from parameterization variants such as DoRA, VeRA, HiRA, QLoRA, and Flora; Stable-LoRA leaves $A \in \mathbb{R}^{r \times n}$8 intact and intervenes only through an early optimization schedule on $A \in \mathbb{R}^{r \times n}$9 (Wu et al., 5 Mar 2026).

5. Empirical performance, diagnostics, and computational cost

The empirical study evaluates Qwen2 $B \in \mathbb{R}^{m \times r}$0B and $B \in \mathbb{R}^{m \times r}$1B, LLaMA-3.2 $B \in \mathbb{R}^{m \times r}$2B and $B \in \mathbb{R}^{m \times r}$3B, and, in the appendix, LLaMA-3.1-$B \in \mathbb{R}^{m \times r}$4B. Tasks include HellaSwag, Social IQa, OpenBookQA, ARC-Easy, ARC-Challenge, and chain-of-thought mathematical reasoning with MetaMathQA training and GSM8K evaluation (Wu et al., 5 Mar 2026). The default LoRA setup uses rank $B \in \mathbb{R}^{m \times r}$5, AdamW, weight decay $B \in \mathbb{R}^{m \times r}$6, no dropout, $B \in \mathbb{R}^{m \times r}$7 sampled from $B \in \mathbb{R}^{m \times r}$8, and $B \in \mathbb{R}^{m \times r}$9. Stable-LoRA evaluates shrink rates $r \ll \min(m,n)$0.

Across QA benchmarks, Stable-LoRA consistently outperforms AdamW, LoRA+, Riemann-preconditioned LoRA, and LoRA-RITE. On average over five QA datasets, Qwen2 $r \ll \min(m,n)$1B improves from $r \ll \min(m,n)$2 with AdamW and $r \ll \min(m,n)$3 with LoRA+ to $r \ll \min(m,n)$4 with Stable-LoRA; LLaMA-3.2 $r \ll \min(m,n)$5B improves from $r \ll \min(m,n)$6 with AdamW and $r \ll \min(m,n)$7 with LoRA+ to $r \ll \min(m,n)$8 with Stable-LoRA (Wu et al., 5 Mar 2026). On GSM8K after MetaMathQA training, the $r \ll \min(m,n)$9B model reaches $s$0 at $s$1k steps with Stable-LoRA versus $s$2 with AdamW. On the combined QA dataset, Stable-LoRA outperforms AdamW for every tested model size and target-module choice; for example, on the $s$3B model with qkvogud, accuracy rises from $s$4 to $s$5.

The diagnostics align closely with the theory. In vanilla LoRA, $s$6 starts large and remains almost constant or slowly increasing, while $s$7 grows rapidly from near zero. With Stable-LoRA, $s$8 is gradually reduced until it crosses $s$9 at the stable condition, after which $\eta$0’s growth pattern remains similar. This is presented as empirical evidence that the method removes initialization-induced instability without harming the useful learning dynamics of $\eta$1.

The computational overhead is negligible. On Qwen2-$\eta$2B with HellaSwag, training time rises from $\eta$3 s with AdamW to $\eta$4 s with Stable-LoRA, about $\eta$5, while memory usage remains identical to vanilla LoRA because no new parameters or buffers are introduced (Wu et al., 5 Mar 2026). This operational profile differentiates Stable-LoRA from heavier optimizer-level alternatives such as Riemann preconditioning and LoRA-RITE, which incur materially larger overheads in the reported comparison.

6. Diffusion, acceleration, and serving-system meanings

Outside LLM optimization theory, the term “Stable-LoRA” or closely adjacent formulations describe different technical objects. In medical image synthesis for the MEDVQA-GI challenge, “Stable-LoRA” refers to a base Stable Diffusion model plus LoRA adapters trained on colonoscopy images and prompts, optionally within DreamBooth personalization using a dedicated concept token such as “V” (Peter et al., 28 Feb 2025). The paper reports that full fine-tuning of Stable Diffusion attains lower FID than DreamBooth + LoRA across all splits—$\eta$6 versus $\eta$7 for single-center, multi-center, and combined data—while DreamBooth + LoRA attains a slightly higher Inception Score, $\eta$8 versus $\eta$9. A plausible implication is that, in this medical setting, LoRA is primarily an efficiency mechanism rather than a fidelity-maximizing one.

In diffusion acceleration, LCM-LoRA distills Latent Consistency Models into Stable Diffusion through LoRA-only training and treats the resulting LoRA weights as a universal acceleration module (Luo et al., 2023). The module can be directly plugged into compatible Stable Diffusion checkpoints or style LoRAs without further training, enabling typical sampling in $A$00–$A$01 steps. The work explicitly frames LCM-LoRA as an “acceleration vector” rather than a content or style vector, which marks a conceptual shift in how LoRA can be used inside diffusion systems.

In serving systems, S-LoRA denotes a multi-adapter tensor-parallel serving framework for LLMs rather than a training rule (Wang et al., 27 Oct 2025). Its practical limitation is extra all-gather and all-reduce communication for LoRA computation under Megatron-style tensor parallelism. Block-Diagonal LoRA addresses this by constraining certain LoRA factors to be block-diagonal, eliminating additional LoRA communication and yielding up to $A$02 end-to-end speed-up for Llama-3.1-70B and up to $A$03 for Llama-3.1-8B on eight A100 GPUs, while remaining similarly parameter efficient as standard LoRA.

These parallel usages show that “Stable-LoRA” is not a single family of methods unified by one objective. In one line of work it means stabilizing feature learning during PEFT; in another it denotes Stable Diffusion adaptation; in another it refers to fast diffusion sampling modules; and in systems research it appears as S-LoRA, a serving architecture. The shared substrate is always low-rank adaptation, but the stabilized quantity differs: feature scale in LLM optimization, domain-specific image synthesis in medical diffusion, PF-ODE solver behavior in LCM-LoRA, or communication overhead in tensor-parallel serving (Wu et al., 5 Mar 2026, Peter et al., 28 Feb 2025, Luo et al., 2023, Wang et al., 27 Oct 2025).