Normalized Low-Rank Adaptation (NoRA) Systems

• NoRA is a method for parameter-efficient fine-tuning of LoRA-enabled ML models that normalizes down-projection to manage effective learning rates across dimensions.
• It uses a downstream strategy of mapping input into an r-dimensional bottleneck before reversing projections to manage the scale defects of traditional LoRA, optimizing for effective learning rates.
• In contrast to standard LoRA, which relies on manual intialization, NoRA handles re-balancing before the training starts, addressing parameter size and learning efficiency.

Normalized Low-Rank Adaptation (NoRA) is a parameter-efficient fine-tuning method that normalizes the down-projection of a LoRA adapter along its low-rank dimension during training. The method addresses the asymmetric initialization of standard LoRA, in which the up-projection is initialized to zero and early updates are governed by the random geometry and scale of the down-projection. By normalizing each column of the down-projection, NoRA controls coordinate-wise effective learning rates, improves early optimization dynamics, and preserves LoRA’s parameter efficiency, linearity, mergeability, and zero inference-time overhead after weight merging. The method is introduced in “Normalized Low-Rank Adaptation” (Kang et al., 31 Aug 2026).

1. Terminology and conceptual scope

The acronym NoRA is not unambiguous in the low-rank adaptation literature. In “Normalized Low-Rank Adaptation” (Kang et al., 31 Aug 2026), it denotes Normalized Low-Rank Adaptation. Other papers use the same acronym for different methods: “NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models” defines NoRA as Nested Low-Rank Adaptation (Lin et al., 2024), while “NoRA: Breaking the Linear Ceiling of Low-Rank Adaptation via Manifold Expansion” defines it as Non-linear Rank Adaptation (Chen, 26 Feb 2026). Consequently, the term must be disambiguated by its expansion or citation.

The normalized method is distinct from several related approaches. AutoLoRA performs layer-wise rank allocation through selection variables but does not normalize low-rank factors or updates (Zhang et al., 2024). OLoRA uses QR-derived orthonormal initialization but does not maintain orthonormality during training (Büyükakyüz, 2024). PoLAR constrains direction factors to Stiefel manifolds and separates them from a trainable core matrix, rather than normalizing the down-projection columns in the NoRA manner (Lion et al., 3 Jun 2025). Norm-Bounded Low-Rank Adaptation, or NB-LoRA, constrains the singular values of the complete update matrix and thereby imposes explicit Schatten-norm bounds (Wang et al., 31 Jan 2025).

Normalized NoRA instead targets the initialization and training geometry of the conventional LoRA factorization. Its normalization is applied to one factor, not to the complete update matrix, singular values, activations, or pretrained weights.

2. Standard LoRA and its early optimization dynamics

For a frozen pretrained weight matrix $\bm W_0\in\mathbb R^{d\times k}$, standard LoRA introduces a rank-$r$ update

$\Delta\bm W=\alpha\bm B\bm A,$

where

$$\bm A\in\mathbb R^{r\times k},\qquad \bm B\in\mathbb R^{d\times r},\qquad r\ll \min(d,k).$$

For an input $\bm x\in\mathbb R^k$, the adapted output is

$\bm y=\bm W_0\bm x+\alpha\bm B\bm A\bm x.$

The factor $\bm A$ is the down-projection, mapping the input into an $r$-dimensional bottleneck, whereas $\bm B$ is the up-projection, mapping the bottleneck representation back into the output dimension. The trainable parameter count is approximately $r(k+d)$ rather than $r$0. Implementations commonly use $r$1 in place of $r$2; under that convention, the recommended NoRA choice $r$3 gives unit effective scaling.

The usual LoRA initialization is

$r$4

which ensures

$r$5

The adapted model therefore initially equals the pretrained model. However, the two factors are not dynamically symmetric. If

$r$6

then

$r$7

At initialization,

$r$8

Thus, $r$9 moves first, while its gradient is determined by the random down-projection $\Delta\bm W=\alpha\bm B\bm A,$0.

A first gradient step on $\Delta\bm W=\alpha\bm B\bm A,$1 gives

$\Delta\bm W=\alpha\bm B\bm A,$2

The resulting first-order change in the merged weight is

$\Delta\bm W=\alpha\bm B\bm A,$3

where

$\Delta\bm W=\alpha\bm B\bm A,$4

Early LoRA therefore behaves like full fine-tuning whose gradient is right-multiplied by an input-side, positive-semidefinite, rank-$\Delta\bm W=\alpha\bm B\bm A,$5 preconditioner. For the $\Delta\bm W=\alpha\bm B\bm A,$6-th input coordinate, the dominant diagonal contribution is proportional to

$\Delta\bm W=\alpha\bm B\bm A,$7

where $\Delta\bm W=\alpha\bm B\bm A,$8 is the $\Delta\bm W=\alpha\bm B\bm A,$9-th column of $$\bm A\in\mathbb R^{r\times k},\qquad \bm B\in\mathbb R^{d\times r},\qquad r\ll \min(d,k).$$0. Column norms consequently act as coordinate-wise effective learning-rate multipliers. The off-diagonal terms $$\bm A\in\mathbb R^{r\times k},\qquad \bm B\in\mathbb R^{d\times r},\qquad r\ll \min(d,k).$$1 produce crosstalk between input coordinates.

For common random initialization with entry variance $$\bm A\in\mathbb R^{r\times k},\qquad \bm B\in\mathbb R^{d\times r},\qquad r\ll \min(d,k).$$2,

$$\bm A\in\mathbb R^{r\times k},\qquad \bm B\in\mathbb R^{d\times r},\qquad r\ll \min(d,k).$$3

Because $$\bm A\in\mathbb R^{r\times k},\qquad \bm B\in\mathbb R^{d\times r},\qquad r\ll \min(d,k).$$4, the initial gains can be small. Their relative fluctuations are of order $$\bm A\in\mathbb R^{r\times k},\qquad \bm B\in\mathbb R^{d\times r},\qquad r\ll \min(d,k).$$5, producing randomly unbalanced coordinate-wise learning rates, particularly at low rank. NoRA is designed to remove this scale imbalance while retaining the low-rank factorization.

3. Normalized down-projection parameterization

Let

$$\bm A\in\mathbb R^{r\times k},\qquad \bm B\in\mathbb R^{d\times r},\qquad r\ll \min(d,k).$$6

NoRA applies column-wise normalization across the rank dimension:

$$\bm A\in\mathbb R^{r\times k},\qquad \bm B\in\mathbb R^{d\times r},\qquad r\ll \min(d,k).$$7

where $$\bm A\in\mathbb R^{r\times k},\qquad \bm B\in\mathbb R^{d\times r},\qquad r\ll \min(d,k).$$8 is a small numerical-stability constant. Except for the numerical edge case controlled by $$\bm A\in\mathbb R^{r\times k},\qquad \bm B\in\mathbb R^{d\times r},\qquad r\ll \min(d,k).$$9,

$\bm x\in\mathbb R^k$0

for every input coordinate $\bm x\in\mathbb R^k$1.

The normalization has four defining properties:

• it is applied to the down-projection $\bm x\in\mathbb R^k$2;
• it is performed column-wise;
• it acts across the $\bm x\in\mathbb R^k$3 rows corresponding to the rank dimension;
• it does not normalize $\bm x\in\mathbb R^k$4, the input $\bm x\in\mathbb R^k$5, or the activation $\bm x\in\mathbb R^k$6.

The NoRA forward computation is

$\bm x\in\mathbb R^k$7

or, including the frozen pretrained branch,

$\bm x\in\mathbb R^k$8

Because $\bm x\in\mathbb R^k$9 depends on parameters but not on $\bm y=\bm W_0\bm x+\alpha\bm B\bm A\bm x.$0, the mapping remains linear in the input. After training, the normalized update can be merged into the base weight:

$\bm y=\bm W_0\bm x+\alpha\bm B\bm A\bm x.$1

This property distinguishes NoRA from activation normalization of the form

$\bm y=\bm W_0\bm x+\alpha\bm B\bm A\bm x.$2

which is input-dependent and generally nonlinear in $\bm y=\bm W_0\bm x+\alpha\bm B\bm A\bm x.$3. Such an activation-normalized branch cannot be exactly absorbed into one static linear weight matrix.

Under the common LoRA convention in which the effective update uses $\bm y=\bm W_0\bm x+\alpha\bm B\bm A\bm x.$4, the paper recommends

$\bm y=\bm W_0\bm x+\alpha\bm B\bm A\bm x.$5

In the unnormalized notation, this corresponds to effective scaling $\bm y=\bm W_0\bm x+\alpha\bm B\bm A\bm x.$6. With normalized columns, the diagonal of the initial effective preconditioner becomes

$\bm y=\bm W_0\bm x+\alpha\bm B\bm A\bm x.$7

under unit effective scaling. The normalization therefore removes the random column-norm contribution from the principal coordinate-wise gains. It does not eliminate crosstalk, since the off-diagonal inner products between normalized columns can remain nonzero.

NoRA is not a capacity increase. It uses the same rank and the same number of trainable parameters as LoRA. Its intended benefit arises from conditioning, initialization geometry, and optimization dynamics rather than from increasing the algebraic rank of the update.

4. Full NoRA, NoRA-init, and BIMI

The method has two principal variants.

Full NoRA

Full NoRA applies normalization throughout training. Each forward pass uses

$\bm y=\bm W_0\bm x+\alpha\bm B\bm A\bm x.$8

and computes

$\bm y=\bm W_0\bm x+\alpha\bm B\bm A\bm x.$9

The normalization is differentiated through during backpropagation. Since the forward path is invariant to positive rescaling of an individual column of $\bm A$0, radial changes in that column do not directly alter the effective projection. Gradients are predominantly directed along the column sphere, while the normalized projection maintains controlled column norms.

Full NoRA thus provides continuous scale control during optimization. The paper associates this with stronger and more uniformly scaled early gradients, improved conditioning, and reduced sensitivity to random initialization. The normalization can introduce a small training-time cost for column-norm computation and elementwise division, but it introduces no additional trainable parameters.

NoRA-init

NoRA-init applies normalization only at initialization:

$\bm A$1

Subsequent training uses ordinary LoRA:

$\bm A$2

NoRA-init therefore requires no repeated normalization and follows the standard LoRA computation after initialization. Its motivation is that $\bm A$3 is initially zero, so the initial geometry of $\bm A$4 controls the decisive first optimization phase. The paper reports that NoRA-init captures most of the benefit of full NoRA, while persistent normalization provides an additional improvement in supervised fine-tuning.

BIMI

Block Identity Matrix Initialization, or BIMI, is a deterministic realization of NoRA-init. If

$\bm A$5

then BIMI constructs

$\bm A$6

where $\bm A$7 consists of the first $\bm A$8 columns of $\bm A$9. Every column has unit norm:

$r$0

BIMI is used only for initialization; both $r$1 and $r$2 are subsequently trained. This differs from MiSS, which fixes a block-identity projection during training. The supplied experiments report that BIMI performs comparably to normalized random initialization.

The variants can be summarized as follows:

Method	Initialization	During training	Mergeable
LoRA	Random $r$3, $r$4	Unnormalized $r$5	Yes
NoRA-init	Normalized $r$6, $r$7	Ordinary LoRA	Yes
NoRA	Standard or other $r$8, $r$9	Normalize $\bm B$0 at each step	Yes

5. Optimization effects and empirical evaluation

The main mechanism proposed by NoRA is balanced coordinate-wise learning. In standard LoRA, the diagonal of the early preconditioner contains the random quantities $\bm B$1. NoRA fixes these column norms, so with unit effective scaling,

$\bm B$2

The paper reports that NoRA produces stronger gradient norms than standard LoRA and that these approach those of full fine-tuning. The effect is described as largely independent of rank after normalization. The remaining off-diagonal terms preserve low-rank crosstalk rather than eliminating all correlations among input coordinates.

The normalization dimension is important. The paper compares $\bm B$3, which normalizes columns across the rank dimension, with $\bm B$4, which normalizes rows across the input dimension. Only $\bm B$5 directly controls the column norms that determine the coordinate-wise gains.

On the reported supervised fine-tuning evaluation, rank-dimensional normalization consistently outperforms both no normalization and input-dimension row normalization:

Initialization	Method	GSM8K	Math	Average
$\bm B$6	None	47.68	10.86	29.27
	$\bm B$7	48.67	10.82	29.75
	$\bm B$8	60.12	14.20	37.16
$\bm B$9	None	50.41	11.68	31.05
	$r(k+d)$0	49.73	11.34	30.54
	$r(k+d)$1	59.96	13.93	36.95
$r(k+d)$2	None	48.19	11.02	29.61
	$r(k+d)$3	48.29	10.86	29.58
	$r(k+d)$4	58.98	14.34	36.66
BIMI	BIMI	59.89	14.24	37.07

The experiments cover language-model pretraining, supervised fine-tuning, and reinforcement learning with verifiable rewards.

Pretraining

The pretraining experiments use approximately 340-million-parameter models trained on SlimPajama with 10 billion tokens and a 0.5-million-token batch size. An additional configuration uses FineWeb-10BT, AdamW, peak learning rate $r(k+d)$5, 20,480 optimization steps, sequence length 2,048, global batch size 256, 1,024 warmup steps, cosine decay to 10% of the peak learning rate, gradient clipping at 1.0, and random seed 42.

For MLA, the reported average benchmark results are:

MLA method	LAMBADA PPL	WikiText PPL	LAMBADA accuracy	Average
$r(k+d)$6	48.73	32.24	30.62	41.17
$r(k+d)$7	61.83	32.89	28.70	40.36
$r(k+d)$8	50.77	32.42	29.87	40.64

Direct latent normalization performs slightly better in this experiment, but it is nonlinear in $r(k+d)$9 and is not exactly mergeable. NoRA-init retains linearity and mergeability.

For MHA with rank $r$00 applied to all attention projections, standard LoRA suffered severe optimization collapse, including a zero LAMBADA score and missing perplexities. NoRA-init avoided this collapse and produced better-scaled gradients, although its reported average remained below the full-weight baseline.

Supervised fine-tuning

The supervised fine-tuning experiments use Llama-3.2-3B with MetaMath and CodeFeedback data. Evaluation includes GSM8K, Math500, HumanEval, MBPP, MMLU, AGIEval, and ARC-Challenge. The principal configuration uses rank $r$01, one epoch, batch size 128, AdamW, learning rate $r$02, cosine scheduling, warmup ratio 0.3, zero dropout, and LoRA modules on $r$03.

Method	Trainable parameters	GSM8K	Math	HumanEval	MBPP	Average
Full fine-tuning	3B	65.12	17.96	36.15	49.05	42.07
PiSSA	48.6M	54.66	12.08	39.00	55.30	40.26
OFT	53.7M	56.10	13.02	38.40	54.20	40.43
RSLoRA	48.6M	57.05	12.32	39.65	56.10	41.28
MiSS	49.5M	60.80	14.60	39.60	55.80	42.70
LoRA	48.6M	50.94	10.38	37.20	53.20	37.93
NoRA-init	48.6M	59.89	14.24	39.60	55.80	42.38
NoRA	48.6M	61.63	14.46	42.10	55.30	43.37

Relative to standard LoRA, NoRA improves the reported average by 5.44 points. NoRA-init obtains most of this improvement, while persistent normalization contributes a further 0.99 points. The method also improves DoRA: DoRA obtains an average of 38.30, NoRA-init-DoRA 41.40, and NoRA-DoRA 43.34.

Catastrophic forgetting

Retention is evaluated on MMLU, AGIEval, and ARC-Challenge. The reported average change relative to the base model is:

Method	MMLU	AGIEval	ARC-C	Average change
Base model	55.10	23.99	42.92	0.00
Full fine-tuning	54.17	26.25	41.64	+0.02
LoRA	54.27	24.40	41.64	-0.56
PiSSA	54.43	23.99	42.75	-0.28
MiSS	53.40	24.33	41.98	-0.70
NoRA-init	54.09	25.10	42.24	-0.19
NoRA	54.00	25.32	42.75	+0.02

The results associate NoRA with a favorable stability–plasticity trade-off: task adaptation is improved while retained benchmark performance remains close to that of the base model.

Reinforcement learning with verifiable rewards

The RLVR experiments use DeepSeek-R1-Distill-Qwen1.5B, the DAPO objective, DAPO-Math-17K, eight GPUs, bfloat16, 1,024 steps, global batch size 128, learning rate $r$04, cosine scheduling without warmup, rank $r$05, $r$06, dropout 0.05, all attention and MLP projections, and eight sampled responses per prompt.

Method	AIME24	AIME25	AMC	HMMT	MATH500	Minerva	Overall average
Base	24.3	20.4	64.8	9.9	80.9	27.8	41.0
MiLoRA	4.2	0.0	19.6	0.0	44.5	11.7	18.0
PiSSA	0.0	0.0	0.0	0.0	0.6	0.1	0.2
LoRA	28.0	23.1	68.0	10.3	81.3	30.1	42.8
NoRA	26.7	23.5	72.7	10.3	84.5	32.0	44.4

NoRA improves over standard LoRA by 1.6 average points and over the base model by 3.4 points. The spectral initialization methods PiSSA and MiLoRA were highly unstable in this RLVR configuration, whereas NoRA does not rely on SVD or pretrained-weight spectral structure.

6. Computational properties, limitations, and relation to other methods

NoRA retains LoRA’s parameter count:

$r$07

with approximately

$r$08

trainable parameters. In the Llama-3.2-3B supervised fine-tuning experiments, both LoRA and NoRA use 48.6 million trainable parameters.

Full NoRA adds column-norm computation and elementwise division during training. The paper characterizes this as a minor overhead relative to the matrix multiplications involving $r$09 and $r$10, but does not provide a precise runtime percentage. NoRA-init avoids repeated normalization and has virtually no training-time overhead beyond initialization.

After training, NoRA can be merged:

$r$11

Consequently, it requires no additional inference-time parameters after merging, no normalization at inference, and no extra latency beyond the single matrix multiplication of the merged model.

NoRA’s main limitations are conceptual and empirical. Normalization does not remove low-rank crosstalk, because off-diagonal terms $r$12 remain. Scaling still matters: larger scaling can improve adaptation but may increase catastrophic forgetting. The method is not a capacity increase and therefore does not address limitations caused by insufficient rank directly. The experiments cover pretraining, supervised fine-tuning, and RLVR, but do not establish universal superiority across all architectures, ranks, datasets, quantization regimes, or optimizers. The paper also does not provide a broad numerical rank sweep or a precise failure threshold.

The distinction between NoRA and other low-rank methods is summarized below:

Method	Primary mechanism	Persistent normalization or constraint
NoRA	Column normalization of the LoRA down-projection	Yes, across the rank dimension
NoRA-init	Normalized down-projection initialization	No, after initialization
AutoLoRA	Layer-wise rank selection	No factor or update normalization
OLoRA	QR-based orthonormal initialization	No maintained orthonormality
Nested LoRA NoRA	Frozen SVD-derived outer factors and trainable inner factors	Not the normalized NoRA method
Non-linear NoRA	SiLU and structural dropout in a bottleneck adapter	No explicit normalization equation
NB-LoRA	Explicit singular-value and Schatten-norm bounds	Yes, at the update-matrix level
PoLAR	Stiefel-constrained direction factors and full core matrix	Yes, geometric orthonormality

NB-LoRA provides the clearest contrast with normalized NoRA. NoRA controls the norms of columns in one factor, which regulates the implicit early preconditioner. NB-LoRA instead parameterizes the complete update so that its singular values satisfy prescribed bounds, including nuclear, Frobenius, or spectral norm constraints. Factor normalization does not necessarily imply a tight bound on the complete update matrix; conversely, a norm-bound does not necessarily equalize or normalize individual factor columns.

The central contribution of normalized NoRA is therefore an optimization-geometric intervention rather than a new rank allocation, spectral decomposition, nonlinear adapter, or explicit matrix-norm constraint. Standard LoRA’s early update is controlled by the random matrix $r$13 through the hidden preconditioner

$r$14

NoRA replaces the random column-norm component of this preconditioner with controlled unit-norm columns. The resulting method preserves LoRA’s low parameter count, linear input mapping, weight merging, and zero inference-time overhead, while the reported experiments associate it with stronger early gradients, more stable training, improved adaptation, and reduced catastrophic forgetting.