Normalized Low-Rank Adaptation

Abstract: While low-rank adaptation (LoRA) is widely used for parameter-efficient model adaptation, how to regularize its training dynamics for stable and effective optimization remains underexplored. Because LoRA initializes the up-projection to zero, its early optimization dynamics are largely governed by the down-projection. Building on this observation, we introduce Normalized Low-Rank Adaptation (NoRA), a simple yet effective method that normalizes the down-projection matrices during training. We further show that the same normalization can be applied only at initialization, improving standard LoRA without requiring repeated normalization throughout training. Across pretraining, supervised finetuning, and reinforcement learning, NoRA consistently accelerates convergence, improves performance and training stability, and mitigates catastrophic forgetting. These benefits require neither additional trainable parameters nor inference-time computation, making NoRA a simple and broadly applicable enhancement to LoRA.

• The paper presents NoRA, which remedies an input-side preconditioning issue in standard LoRA by normalizing the columns of the low-rank factor $A$.
• NoRA improves supervised finetuning performance by 5.44 points over traditional LoRA, showing average improvements for tasks such as GSM8K, Math, HumanEval, and MBPP, illustrating its effectiveness in various scenarios.
• The proposed normalization technique contributes to significant enhancements in knowledge retention, preventing the degradation of retained benchmark performance observed in other low-rank adapters.
• Grammar and tone are refined for clarity and jargon-free comprehensibility.

Problem formulation and central claim

LoRA parameterizes an adaptation update as $\Delta W = \alpha BA$, with $A \in \mathbb{R}^{r \times k}$, $B \in \mathbb{R}^{d \times r}$, and $r \ll \min(d,k)$. Its standard initialization sets $B$ to zero and samples $A$ randomly. This preserves the pretrained function at initialization, but it also creates a pronounced asymmetry in early optimization: the initial gradient with respect to $A$ is zero, whereas the gradient with respect to $B$ is proportional to the full-model gradient multiplied by $A^\top$. Consequently, the initial scale and geometry of the down-projection determine the first effective update of the adapter.

"Normalized Low-Rank Adaptation" (2608.31036) argues that this down-projection has been underexamined as a source of optimization distortion. The paper’s main claim is that standard random LoRA induces an undesirable input-side preconditioner whose diagonal gains are small, random, and unrelated to the data geometry. NoRA addresses this problem by normalizing each column of $A$ along the rank dimension. For every input coordinate $A \in \mathbb{R}^{r \times k}$0, the corresponding vector $A \in \mathbb{R}^{r \times k}$1 is rescaled to unit Euclidean norm:

$A \in \mathbb{R}^{r \times k}$2

The resulting update is

$A \in \mathbb{R}^{r \times k}$3

Because the normalization depends only on the adapter parameters and not on $A \in \mathbb{R}^{r \times k}$4, the transformation remains linear in the input. The normalized matrix can therefore be folded into the merged weight update after training, preserving LoRA’s exact mergeability and introducing no inference-time computation.

The paper studies two variants. NoRA applies normalization throughout training, whereas NoRA-init normalizes $A \in \mathbb{R}^{r \times k}$5 only once at initialization and subsequently optimizes the factors using the standard LoRA parameterization. NoRA-init is evaluated with both random normalized initialization and Block Identity Matrix Initialization (BIMI), in which $A \in \mathbb{R}^{r \times k}$6 consists of repeated identity blocks.

A preconditioning interpretation of LoRA

The paper’s most substantive theoretical contribution is its interpretation of early LoRA optimization as gradient descent under an implicit low-rank input-side preconditioner. Let $A \in \mathbb{R}^{r \times k}$7 denote the full-finetuning gradient. At initialization, where $A \in \mathbb{R}^{r \times k}$8, an update to $A \in \mathbb{R}^{r \times k}$9 induces the following first-order change in the merged weight:

$B \in \mathbb{R}^{d \times r}$0

Full finetuning corresponds to $B \in \mathbb{R}^{d \times r}$1. LoRA therefore does not merely restrict the update to a low-rank factorization; during its initial optimization phase, it right-multiplies the full gradient by a positive-semidefinite matrix of rank at most $B \in \mathbb{R}^{d \times r}$2. This matrix acts on input coordinates and is analogous to an implicit input-side preconditioner.

The diagonal entries of $B \in \mathbb{R}^{d \times r}$3 are $B \in \mathbb{R}^{d \times r}$4. They determine the direct gain applied to each input-coordinate column of the full gradient. Off-diagonal entries introduce crosstalk through the inner products between latent projection vectors. Standard random initialization produces both effects without relating them to the task or to the curvature of the pretrained model. If the entries of $B \in \mathbb{R}^{d \times r}$5 have variance proportional to $B \in \mathbb{R}^{d \times r}$6, then the expected squared column norm is proportional to $B \in \mathbb{R}^{d \times r}$7, which is substantially below one at typical low ranks. The resulting adapter update can therefore operate at a small fraction of the full-finetuning scale.

This analysis gives a concrete interpretation to the observed gradient-norm gap between LoRA and full finetuning: the issue is not only the rank constraint, but also the scale of the induced preconditioner. Rank-dimension normalization sets the diagonal gains to a controlled value, while leaving the directional crosstalk determined by the latent vectors. Under suitable scaling, the expected preconditioner diagonal becomes one, and the expected gradient magnitude is brought closer to that of full finetuning independently of $B \in \mathbb{R}^{d \times r}$8.

The distinction between rank-wise and input-wise normalization is central. Normalizing the rows of $B \in \mathbb{R}^{d \times r}$9 does not control the norms of the input-coordinate projection vectors and leaves the diagonal of $r \ll \min(d,k)$0 random. In contrast, column normalization along the rank dimension directly fixes the coordinate-wise gains that govern the initial update.

Figure 1: NoRA normalizes each input-to-latent projection vector along the rank dimension and produces substantially stronger early gradient norms than standard LoRA across ranks.

The empirical gradient trajectories support this interpretation. NoRA exhibits substantially stronger early gradients than standard LoRA, with magnitudes approaching those of full finetuning. The authors emphasize that this is not simply a consequence of increasing the nominal rank: at the same rank, changing the initialization and normalization substantially alters optimization behavior. This supports the claim that low-rank capacity alone does not determine the effective learnability of a LoRA adapter.

Relationship to normalized latent representations and structured initialization

The construction is motivated by normalized latent representations in Multi-head Latent Attention. Directly applying input-dependent normalization to the latent feature, $r \ll \min(d,k)$1, would make the update nonlinear in $r \ll \min(d,k)$2 and prevent exact absorption into the pretrained linear layer. NoRA transfers the scale-control effect from the latent activation to the parameter matrix, yielding $r \ll \min(d,k)$3. This substitution is important: it retains a linear forward map and deployment equivalence with conventional LoRA, but it does not reproduce the exact functional behavior of input-dependent latent normalization.

The connection to MiSS provides an additional structural interpretation. When the input dimension is divisible by the rank, a fixed matrix formed by concatenated identity blocks produces a projection in which every column has unit norm. MiSS can therefore be represented as a special case of a LoRA-style update with a normalized, fixed down-projection. BIMI adopts the same block-identity matrix only for initialization and then allows the down-projection to remain trainable. This separates the potential benefit of the normalized initial geometry from the representational restriction imposed by permanently fixing the projection.

The experiments indicate that BIMI performs comparably to normalized random initialization. The result is consistent with the paper’s preconditioning account: the diagonal of $r \ll \min(d,k)$4 appears to be more consequential than the precise distribution of the off-diagonal inner products, at least in the evaluated settings. However, this conclusion is empirical rather than a complete characterization of the optimization dynamics.

Ablation results and normalization geometry

The supervised-finetuning ablation compares row-wise normalization, rank-wise normalization, several random initialization distributions, and BIMI. Rank-wise normalization consistently improves performance, whereas row-wise normalization produces little or no benefit. For the uniform initialization $r \ll \min(d,k)$5, rank-wise normalization increases GSM8K accuracy from 47.68 to 60.12 and the average of GSM8K and the Math benchmark from 29.27 to 37.16. Comparable improvements occur under Gaussian and Kaiming-uniform-style initializations.

Initialization	Method	GSM8K	Math	Average
$r \ll \min(d,k)$6	None	47.68	10.86	29.27
$r \ll \min(d,k)$7	Rank-wise normalization	60.12	14.20	37.16
$r \ll \min(d,k)$8	None	50.41	11.68	31.05
$r \ll \min(d,k)$9	Rank-wise normalization	59.96	13.93	36.95
$B$0	None	48.19	11.02	29.61
$B$1	Rank-wise normalization	58.98	14.34	36.66
BIMI	Block identity initialization	59.89	14.24	37.07

The fact that rank-wise normalization converges to similar performance across distinct initialization distributions is consistent with the claim that column-scale imbalance, rather than the unnormalized distribution itself, is the principal defect. It also explains why row normalization is ineffective: it regularizes a dimension that does not directly determine the per-input-coordinate gains in the first-order preconditioner.

The paper recommends $B$2 for NoRA under the common $B$3 implementation convention. This scaling is intended to align the initial gradient norm with full finetuning. The authors report that larger scaling, such as $B$4, can improve loss fitting in some cases but increases forgetting, motivating a trade-off between adaptation strength and retention.

Pretraining behavior

The pretraining experiments evaluate both normalized latent representations and NoRA-style parameter normalization. In the MLA setting, directly normalizing $B$5 produces the strongest results, while NoRA-init provides much of the gain without sacrificing linearity or mergeability. On the reported zero-shot aggregate, direct latent normalization obtains 41.17, standard low-rank adaptation obtains 40.36, and NoRA-init obtains 40.64. The smaller gap between NoRA-init and direct latent normalization is the expected cost of replacing input-dependent normalization with parameter-only normalization.

The MHA experiment exposes a more severe failure mode. Under the same 10-billion-token training budget, standard low-rank adaptation collapses on LAMBADA, with a reported accuracy of 0.00 and unavailable perplexity values. Its aggregate benchmark score is 27.81, compared with 39.69 for NoRA-init and 41.06 for full finetuning. NoRA-init also yields LAMBADA perplexity of 63.45 and WikiText perplexity of 34.39, whereas the standard low-rank run is marked as collapsed on both perplexity measurements.

These results support a strong claim in the paper: under a fixed low-rank parameterization, optimization can fail even when the architecture and training budget are unchanged, and initialization normalization can prevent that failure without increasing the number of trainable parameters. The result also qualifies capacity-based explanations of low-rank degradation. Insufficient rank may constrain the attainable update subspace, but the experiments show that the same rank can exhibit either catastrophic optimization failure or substantially better behavior depending on the geometry of $B$6.

Figure 2: NoRA exhibits more favorable supervised-finetuning loss and training dynamics than standard LoRA, with behavior closer to full finetuning than to the less stable low-rank baselines.

Supervised finetuning and knowledge retention

The SFT experiments use Llama-3.2-3B with rank 32 adapters applied to attention and MLP projections. NoRA obtains the highest reported average across GSM8K, the Math benchmark, HumanEval, and MBPP. Standard LoRA reaches an average of 37.93, NoRA-init reaches 42.38, and NoRA reaches 43.37. Thus, NoRA improves over standard LoRA by 5.44 points and adds 0.99 points over its initialization-only variant.

Its gains are particularly pronounced on GSM8K and HumanEval:

• GSM8K: 50.94 for LoRA versus 61.63 for NoRA.
• Math: 10.38 versus 14.46.
• HumanEval: 37.20 versus 42.10.
• MBPP: 53.20 versus 55.30.

NoRA also outperforms PiSSA, RSLoRA, OFT, and MiSS on the aggregate SFT score. The comparison with MiSS is informative because MiSS uses a fixed normalized block-identity projection, whereas NoRA keeps both factors trainable. NoRA’s higher average, 43.37 versus 42.70, indicates that persistent normalization combined with a trainable down-projection can improve on a fixed normalized projection.

The comparison between NoRA and NoRA-init separates initialization effects from persistent constraint effects. NoRA-init captures most of the gain over LoRA, confirming that the first optimization phase is decisive. NoRA nevertheless performs better, suggesting that continued normalization prevents later scale collapse or explosion and maintains the effective diagonal gains of the implicit preconditioner.

The paper also evaluates normalization within DoRA. DoRA’s average score is 38.30, compared with 41.40 for NoRA-init-DoRA and 43.34 for NoRA-DoRA. This transfer result supports the broader claim that rank-dimension normalization is not specific to the conventional LoRA factorization. It may instead be a general intervention on low-rank down-projections, although the experiments do not establish that it will transfer uniformly to every PEFT parameterization.

Knowledge retention is another prominent result. On MMLU, AGIEval, and ARC-C, the average change relative to the base model is $B$7 for LoRA, $B$8 for MiSS, and $B$9 for NoRA. Full finetuning also reports $A$0 in this evaluation. The paper therefore presents NoRA as improving task adaptation while largely avoiding the degradation of retained benchmark performance observed for several PEFT alternatives. This result should be interpreted within the specific benchmark protocol and one-epoch SFT regime; it does not establish universal immunity to catastrophic forgetting.

RLVR robustness

The RLVR evaluation uses DeepSeek-R1-Distill-Qwen-1.5B, the DAPO objective, and DAPO-Math-17K. Standard LoRA improves the overall benchmark average from 41.0 for the base model to 42.8. NoRA raises it further to 44.4, a gain of 1.6 points over LoRA and 3.4 points over the base model.

The improvements are distributed across the evaluation suite rather than concentrated on a single benchmark. NoRA reaches 72.7 on AMC compared with 68.0 for LoRA, 84.5 on MATH500 compared with 81.3, and 32.0 on Minerva compared with 30.1. Its AIME24 and AIME25 averages are slightly lower or comparable to LoRA, while the aggregate score improves because of broader gains elsewhere.

Method	AIME24	AIME25	AMC	HMMT	MATH500	Minerva	Average
Base	24.3	20.4	64.8	9.9	80.9	27.8	41.0
LoRA	28.0	23.1	68.0	10.3	81.3	30.1	42.8
NoRA	26.7	23.5	72.7	10.3	84.5	32.0	44.4

The contrast with spectral initialization is unusually large. PiSSA obtains an aggregate score of 0.2 and MiLoRA 18.0 in this RLVR setting, despite their utility in supervised adaptation. The paper attributes this fragility to the interaction between pretrained-weight spectral decompositions and RL optimization. NoRA does not require SVD, gradient estimation, or activation statistics, and its scale normalization remains stable under the evaluated on-policy training regime.

The result is important but bounded: the RL experiments use one model, one mathematical dataset, one RL objective, and 1,024 optimization steps. They demonstrate robustness in the reported configuration, not a general theorem about spectral initialization under reinforcement learning.

Limitations and open questions

The theoretical analysis is local to the initialization or early-training regime. The preconditioner derivation neglects terms involving the evolving $A$1 and $A$2, and the argument that initialization accounts for most of the benefit does not fully characterize the long-run dynamics of NoRA. In particular, the analysis does not derive convergence guarantees for AdamW, nor does it quantify how normalization interacts with adaptive optimizer state, momentum, weight decay, or gradient clipping.

The empirical study is broad in optimization regime but narrower in model and task diversity than the paper’s general claims suggest. Pretraining uses small controlled models, SFT centers on Llama-3.2-3B, and RLVR uses DeepSeek-R1-Distill-Qwen1.5B. The paper does not report large-scale comparisons across multiple contemporary model families, quantized LoRA settings, multilingual tasks, vision-LLMs, or very high-rank adapters. It also does not provide a systematic compute-normalized analysis of the overhead from repeatedly normalizing $A$3 during training.

A further open question concerns the role of the off-diagonal structure of $A$4. The paper identifies the diagonal as the decisive quantity based on comparisons between normalized random projections and BIMI, but these methods also differ substantially in crosstalk geometry. A controlled study that independently varies the diagonal gains and off-diagonal correlations would be needed to determine whether crosstalk is benign, harmful, or task-dependent.

Finally, the reported forgetting metric is based on a small set of retained benchmarks and a particular SFT schedule. NoRA’s favorable retention behavior is therefore an important empirical observation, but its dependence on scaling, dataset size, adapter placement, and training duration remains unresolved.

Conclusion

The paper identifies rank-dimension normalization of LoRA’s down-projection as a simple intervention on the optimization geometry of low-rank adaptation. Its preconditioning analysis explains why standard random LoRA can produce weak and uneven early updates: the matrix $A$5 implicitly rescales input coordinates through small, random diagonal gains. NoRA removes this scale imbalance while preserving linearity, exact weight merging, parameter count, and inference cost.

Across the reported experiments, NoRA improves SFT performance from 37.93 to 43.37 on the aggregate benchmark, prevents a severe MHA pretraining collapse observed with standard LoRA, improves RLVR from 42.8 to 44.4, and substantially reduces measured forgetting. NoRA-init demonstrates that much of the benefit is obtained by normalizing only at initialization, while persistent normalization provides additional stability. The paper’s principal unresolved issue is the extent to which these gains arise from diagonal preconditioner calibration versus the broader geometry of the normalized low-rank subspace.