Rescaled Fine-Tuning Matrix

• Rescaled fine-tuning matrix is a family of PEFT methods that adapt pre-trained models by explicitly scaling key components rather than relying solely on additive low-rank updates.
• It encompasses techniques such as SVFit, SpecLoRA, rsLoRA, and α-LoRA, each targeting different aspects like singular values, dominant directions, adapter gains, or base weights.
• These methods enhance training efficiency and generalization, reducing trainable parameters and preventing issues like gradient collapse even at higher update ranks.

Rescaled fine-tuning matrix denotes, in current parameter-efficient fine-tuning practice, a family of matrix reparameterizations in which adaptation is expressed through explicit scaling of structurally important components of a pre-trained model, rather than solely through an unconstrained additive update. In the recent literature, this idea appears in several mathematically distinct forms: SVFit trains only the top singular values of a pre-trained weight while freezing the associated singular vectors and residual subspace (Sun et al., 2024); SpecLoRA rescales top singular directions and combines that modulation with a standard LoRA residual (2505.23099); rsLoRA replaces LoRA’s conventional rank-dependent scaling by a rank-stabilized factor (Kalajdzievski, 2023); and $\alpha$-LoRA rescales the frozen base weight row-wise before adding a low-rank update (Firdoussi et al., 24 Oct 2025). This suggests that “rescaled fine-tuning matrix” is best read as an umbrella description for PEFT schemes that treat scaling itself as a primary adaptation variable.

1. Conceptual basis within parameter-efficient fine-tuning

The common reference point for rescaled fine-tuning is LoRA. In one standard formulation, a frozen weight matrix is updated by an additive low-rank residual,

$$\mathbf{W}\rightarrow \mathbf{W}+\Delta\mathbf{W}=\mathbf{W}+\mathbf{A}\mathbf{B},$$

with $\mathbf{A}\in\mathbb{R}^{n\times r}$, $\mathbf{B}\in\mathbb{R}^{r\times m}$, and $r\ll\{n,m\}$ (2505.23099). In another equivalent presentation for a linear layer, LoRA uses

$x_{\text{out}}=(W+\gamma_r BA)x_{\text{in}}+b,$

where $A\in\mathbb{R}^{r\times d_1}$, $B\in\mathbb{R}^{d_2\times r}$, and $\gamma_r$ is rank-dependent (Kalajdzievski, 2023).

Rescaled fine-tuning departs from this baseline by asking whether the most useful adaptation variable is not only the residual term $AB$, but also the scale of some pre-existing matrix structure. The motivation differs across papers. SVFit argues that random low-rank initialization may prevent the main components of $$\mathbf{W}\rightarrow \mathbf{W}+\Delta\mathbf{W}=\mathbf{W}+\mathbf{A}\mathbf{B},$$0 from being effectively updated in the initial stages of fine-tuning, leading to inefficient gradient descent, suboptimal local minima, and weaker generalization (Sun et al., 2024). SpecLoRA is motivated by an SVD analysis indicating that full fine-tuning mainly amplifies the top singular values and reorients dominant singular vectors while leaving the lower-spectrum components largely intact (2505.23099). rsLoRA studies the scaling factor internal to LoRA itself and argues that the usual division by rank causes gradient collapse at higher ranks (Kalajdzievski, 2023). $$\mathbf{W}\rightarrow \mathbf{W}+\Delta\mathbf{W}=\mathbf{W}+\mathbf{A}\mathbf{B},$$1-LoRA instead treats the contribution of the frozen source model as a quantity that should also be adapted, rather than fixed at the default choice $$\mathbf{W}\rightarrow \mathbf{W}+\Delta\mathbf{W}=\mathbf{W}+\mathbf{A}\mathbf{B},$$2 (Firdoussi et al., 24 Oct 2025).

Method	Rescaled object	Representative parameterization
SVFit	Top singular values $$\mathbf{W}\rightarrow \mathbf{W}+\Delta\mathbf{W}=\mathbf{W}+\mathbf{A}\mathbf{B},$$3	$$\mathbf{W}\rightarrow \mathbf{W}+\Delta\mathbf{W}=\mathbf{W}+\mathbf{A}\mathbf{B},$$4
SpecLoRA	Top singular directions / spectral mask $$\mathbf{W}\rightarrow \mathbf{W}+\Delta\mathbf{W}=\mathbf{W}+\mathbf{A}\mathbf{B},$$5	$$\mathbf{W}\rightarrow \mathbf{W}+\Delta\mathbf{W}=\mathbf{W}+\mathbf{A}\mathbf{B},$$6
rsLoRA	Adapter scaling factor $$\mathbf{W}\rightarrow \mathbf{W}+\Delta\mathbf{W}=\mathbf{W}+\mathbf{A}\mathbf{B},$$7	$$\mathbf{W}\rightarrow \mathbf{W}+\Delta\mathbf{W}=\mathbf{W}+\mathbf{A}\mathbf{B},$$8
$$\mathbf{W}\rightarrow \mathbf{W}+\Delta\mathbf{W}=\mathbf{W}+\mathbf{A}\mathbf{B},$$9-LoRA	Frozen base matrix $\mathbf{A}\in\mathbb{R}^{n\times r}$0	$\mathbf{A}\in\mathbb{R}^{n\times r}$1

Taken together, these formulations indicate that rescaling can be applied at several loci: inside the spectral decomposition of the base matrix, on the gain of the low-rank adapter, or on the base matrix itself.

2. Singular-value rescaling in SVFit

SVFit begins from the singular value decomposition of a pre-trained matrix $\mathbf{A}\in\mathbb{R}^{n\times r}$2,

$\mathbf{A}\in\mathbb{R}^{n\times r}$3

with singular values $\mathbf{A}\in\mathbb{R}^{n\times r}$4 in descending order (Sun et al., 2024). The method partitions the singular vectors into retained and residual components,

$\mathbf{A}\in\mathbb{R}^{n\times r}$5

$\mathbf{A}\in\mathbb{R}^{n\times r}$6

and defines

$\mathbf{A}\in\mathbb{R}^{n\times r}$7

The full reconstruction is written as

$\mathbf{A}\in\mathbb{R}^{n\times r}$8

The central operational choice is that SVFit freezes the singular vectors and trains only the top $\mathbf{A}\in\mathbb{R}^{n\times r}$9 singular values. The adapted matrix is

$\mathbf{B}\in\mathbb{R}^{r\times m}$0

so the low-rank principal part is rescaled while the residual term is frozen. The paper interprets the singular vectors as the matrix’s fundamental subspaces: $\mathbf{B}\in\mathbb{R}^{r\times m}$1 as the range space of $\mathbf{B}\in\mathbb{R}^{r\times m}$2, $\mathbf{B}\in\mathbb{R}^{r\times m}$3 as the null space of $\mathbf{B}\in\mathbb{R}^{r\times m}$4, $\mathbf{B}\in\mathbb{R}^{r\times m}$5 as the range space of $\mathbf{B}\in\mathbb{R}^{r\times m}$6, and $\mathbf{B}\in\mathbb{R}^{r\times m}$7 as the null space of $\mathbf{B}\in\mathbb{R}^{r\times m}$8 (Sun et al., 2024).

The rationale for focusing on the dominant singular spectrum is explicitly spectral. The paper states that the top 10% or even 1% of singular values contribute to over 99% of the total matrix sum, and uses a Fishstar image reconstruction experiment as a visual justification: reconstruction with the largest singular values preserves image quality, whereas the smallest singular values contribute little. This leads to the claim that a pre-trained matrix already contains a highly structured “core” in its dominant singular directions, and that fine-tuning should begin from those directions rather than from randomly initialized low-rank factors.

Experimentally, SVFit is compared with LoRA, DyLoRA, AdaLoRA, and PiSSA on GLUE, image classification, and DreamBooth-style subject-driven generation. For RoBERTa-base, LoRA uses 0.3M trainable parameters with average 85.2, while SVFit uses 0.018M with average 85.1. For RoBERTa-large, LoRA uses 0.8M, SVFit uses 0.036M, and SVFit reaches average 88.5 versus LoRA’s 87.8. For ViT-large, LoRA uses 0.8M trainable parameters and SVFit uses 0.036M; the paper notes that this is roughly a 22× reduction from 0.8M versus 0.036M, while the abstract summarizes the result more conservatively as 16 times fewer trainable parameters. On ViT-large, SVFit achieves average accuracy 88.7, compared with LoRA’s 84.9 and PiSSA’s 88.7; on ViT-base, SVFit gets 84.3 versus LoRA’s 77.6 (Sun et al., 2024).

A key practical property is that the module can be integrated back into the original matrix post-training, so there is no inference-time overhead.

3. Dominant-direction rescaling in SpecLoRA

SpecLoRA starts from an empirical study of full fine-tuning on LLaMA3-8B on Commonsense170K, examining attention matrices (Q/K/V) and MLP projections (Up/Down) through SVD (2505.23099). The reported pattern is highly structured: the pre-trained and fully fine-tuned spectra largely overlap; the largest singular values are amplified after fine-tuning; non-dominant singular values remain largely unchanged; dominant singular vectors are often nearly orthogonal before and after fine-tuning; and the remaining singular subspace stays highly aligned.

This spectral picture motivates a PEFT design in which the dominant directions of the frozen matrix are explicitly rescaled. Writing

$\mathbf{B}\in\mathbb{R}^{r\times m}$9

SpecLoRA focuses on the top-$r\ll\{n,m\}$0 left singular vectors $r\ll\{n,m\}$1. It introduces a diagonal rescaling matrix $r\ll\{n,m\}$2 and applies

$r\ll\{n,m\}$3

while keeping the rows beyond the first $r\ll\{n,m\}$4 unchanged,

$r\ll\{n,m\}$5

The resulting adapted weight is

$r\ll\{n,m\}$6

Thus SpecLoRA combines a spectrally guided rescaling of dominant singular directions with a standard LoRA residual.

For efficiency, the paper avoids explicit SVD in every forward pass by implementing the rescaling through a Hadamard-product mask,

$r\ll\{n,m\}$7

where $r\ll\{n,m\}$8 is a learnable spectral modulation mask built from a learnable vector $r\ll\{n,m\}$9. This preserves the PEFT objective of a low trainable footprint while making the spectral intervention computationally practical.

Reported results cover GLUE with DeBERTaV3-base, commonsense reasoning with LLaMA3-8B, and VTAB-1K with ViT-B/16. On GLUE, SpecLoRA achieves 89.48 average, compared with LoRA 88.13, DoRA 88.31, AdaLoRA 88.86, and full fine-tuning 88.27, while updating only 0.18% of parameters. On the eight-task commonsense suite, SpecLoRA reaches 85.5 average, compared with DoRA 85.0, LoRA$x_{\text{out}}=(W+\gamma_r BA)x_{\text{in}}+b,$0 82.8, MiLoRA 81.9, and AdaLoRA 81.4, at 0.35% trainable parameters. On VTAB-1K, SpecLoRA reports 76.7 average, compared with SSF 75.7, NOAH 75.5, AdaptFormer 74.7, LoRA 74.5, and full fine-tuning 68.9 (2505.23099).

The ablations are integral to the argument. Performance is reported to be fairly stable across a wide range of top-direction counts $x_{\text{out}}=(W+\gamma_r BA)x_{\text{in}}+b,$1; SpecLoRA outperforms LoRA at ranks $x_{\text{out}}=(W+\gamma_r BA)x_{\text{in}}+b,$2; and modifying bottom singular directions helps somewhat but underperforms modifying top singular directions. The paper presents this as direct evidence that the most useful adaptation lives in the leading spectral components.

4. Rank-stabilized rescaling in rsLoRA

rsLoRA addresses a different object: not the spectral structure of the base matrix, but the scaling factor attached to the LoRA adapter itself (Kalajdzievski, 2023). In conventional LoRA, the low-rank update is multiplied by

$x_{\text{out}}=(W+\gamma_r BA)x_{\text{in}}+b,$3

The paper argues that this choice is rank-mismatched because it suppresses the adapter too aggressively as rank grows. Its proposed correction is

$x_{\text{out}}=(W+\gamma_r BA)x_{\text{in}}+b,$4

so that

$x_{\text{out}}=(W+\gamma_r BA)x_{\text{in}}+b,$5

The theoretical argument is framed in terms of rank stability: forward outputs and backward gradients should remain on the same order of magnitude as rank changes. Under the initialization $x_{\text{out}}=(W+\gamma_r BA)x_{\text{in}}+b,$6 and iid entries of $x_{\text{out}}=(W+\gamma_r BA)x_{\text{in}}+b,$7 with variance independent of $x_{\text{out}}=(W+\gamma_r BA)x_{\text{in}}+b,$8, the analysis shows that the effective adapter contribution scales like

$x_{\text{out}}=(W+\gamma_r BA)x_{\text{in}}+b,$9

Keeping this quantity order-1 requires

$A\in\mathbb{R}^{r\times d_1}$0

and the paper states that this stability holds if and only if that asymptotic scaling is used. Under the conventional rule $A\in\mathbb{R}^{r\times d_1}$1, the effective magnitude behaves like $A\in\mathbb{R}^{r\times d_1}$2, which the paper describes as causing gradient collapse and slowed learning.

The empirical claim is correspondingly specific. With ranks such as $A\in\mathbb{R}^{r\times d_1}$3, standard LoRA tends to converge to nearly the same loss or perplexity, and sometimes larger ranks do slightly worse; rsLoRA improves as rank increases, both on Llama 2 fine-tuned on OpenOrca and on GPT-J fine-tuned on GSM8k (Kalajdzievski, 2023). The paper also reports that merely tuning the learning rate of low-rank LoRA does not match the performance of a properly scaled high-rank rsLoRA model, and that initialization-only fixes do not reproduce the benefits.

In this line of work, rescaling is therefore not a spectral prior but a statement about optimization dynamics. The method preserves the familiar merge-in property of LoRA: $A\in\mathbb{R}^{r\times d_1}$4 so training cost increases with rank, but inference cost does not change relative to ordinary LoRA after the adapter is merged.

5. Base-model rescaling in $A\in\mathbb{R}^{r\times d_1}$5-LoRA

$A\in\mathbb{R}^{r\times d_1}$6-LoRA shifts attention from the adapter to the frozen model itself. The paper argues that standard reparameterized fine-tuning typically leaves the base weight unchanged and only adds a trainable update, whereas the contribution of the base model should also be adapted (Firdoussi et al., 24 Oct 2025). In the linear high-dimensional binary classification setting, the target fine-tuning objective is

$A\in\mathbb{R}^{r\times d_1}$7

where the frozen source classifier $A\in\mathbb{R}^{r\times d_1}$8 is multiplied by $A\in\mathbb{R}^{r\times d_1}$9 before adding the trainable term $B\in\mathbb{R}^{d_2\times r}$0. The closed-form solution is

$B\in\mathbb{R}^{d_2\times r}$1

and the final fine-tuned classifier becomes

$B\in\mathbb{R}^{d_2\times r}$2

Defining the target-only classifier

$B\in\mathbb{R}^{d_2\times r}$3

the paper writes

$B\in\mathbb{R}^{d_2\times r}$4

For neural networks, the scalar $B\in\mathbb{R}^{d_2\times r}$5 is generalized to a row-wise vector $B\in\mathbb{R}^{d_2\times r}$6. For a weight matrix $B\in\mathbb{R}^{d_2\times r}$7,

$B\in\mathbb{R}^{d_2\times r}$8

This is called $B\in\mathbb{R}^{d_2\times r}$9-LoRA. In this formulation, LoRA corresponds to fixing $\gamma_r$0, whereas $\gamma_r$1-LoRA learns how strongly each row of the frozen base matrix should contribute.

The theory is developed through Random Matrix Theory in the high-dimensional regime $\gamma_r$2 and $\gamma_r$3. The paper derives a Gaussian limit for the test score,

$\gamma_r$4

and from that the asymptotic test accuracy

$\gamma_r$5

It also derives a unique maximizer $\gamma_r$6 and a worst parameter

$\gamma_r$7

with the additional claim that $\gamma_r$8 tends to increase with source-target alignment $\gamma_r$9 and becomes negligible as $AB$0 (Firdoussi et al., 24 Oct 2025).

On RoBERTa-base fine-tuned on six GLUE tasks, the reported results improve over standard LoRA: MNLI 85.77 to 86.12, QNLI 91.95 to 92.20, MRPC 88.40 to 89.46, RTE 74.01 to 77.62, SST-2 94.00 to 94.38, and QQP 88.80 to 88.86. The paper states that the increase in trainable parameters is only 0.02%, and describes a heuristic algorithm that updates $AB$1 every $AB$2 steps using a separate batch and a separate optimizer to avoid overfitting and encourage better generalization (Firdoussi et al., 24 Oct 2025).

6. Comparative interpretation, misconceptions, and scope

Taken together, these works suggest that a rescaled fine-tuning matrix is not a single standardized formula but a shared design principle: fine-tuning can be made more efficient or more faithful to pre-trained structure by scaling a carefully chosen part of the model rather than treating all degrees of freedom as equally mutable. The principal difference between the papers lies in what is being scaled. SVFit scales top singular values while freezing both singular vectors and the residual component (Sun et al., 2024). SpecLoRA scales dominant spectral directions and adds a low-rank residual (2505.23099). rsLoRA scales the adapter gain so that higher-rank updates remain effective (Kalajdzievski, 2023). $AB$3-LoRA scales the frozen base weight itself, row-wise, before adding the adapter (Firdoussi et al., 24 Oct 2025).

Several misconceptions are explicitly challenged by this literature. One is that all low-rank initialization schemes are functionally equivalent: SVFit argues that random initialization can misalign early optimization and that beginning from principal singular subspaces can better preserve the model’s original knowledge (Sun et al., 2024). Another is that PEFT should be understood purely as additive low-rank correction: SpecLoRA and $AB$4-LoRA both treat multiplicative modulation of the frozen model as an essential adaptation variable (2505.23099, Firdoussi et al., 24 Oct 2025). A third is that LoRA’s conventional $AB$5 factor is merely a harmless normalization; rsLoRA’s analysis states that it induces gradient collapse as rank increases, whereas $AB$6 is the rank-stable choice (Kalajdzievski, 2023). A fourth is that all spectral directions matter equally: SpecLoRA’s top-versus-bottom ablation reports that bottom-direction modification helps somewhat but still underperforms top-direction modification (2505.23099).

The broader significance is methodological. These papers converge on the view that fine-tuning is not well characterized as a uniform rewrite of the weight matrix. SpecLoRA describes it as controlled amplification and reorientation of top singular components; SVFit treats it as rescaling the matrix’s principal subspaces; $AB$7-LoRA formalizes it as learning the proper contribution of source knowledge to the target task; and rsLoRA shows that even when the parameterization remains additive, the scaling law itself governs whether rank is usable in practice. This suggests that “rescaled fine-tuning matrix” is best understood as a structural perspective on transfer learning: adaptation is concentrated in a small number of influential directions, gains, or gates, and the mathematical role of scaling is to expose those variables directly rather than leaving them implicit.