IniLoRA: Efficient Fine-Tuning for LLMs

• IniLoRA is a parameter-efficient fine-tuning method that initializes its low-rank branch to approximate pretrained weights, enabling more aligned adaptation.
• It employs a residual-plus-low-rank decomposition where a fixed residual is combined with optimized low-rank factors during fine-tuning.
• Empirical evaluations show that IniLoRA and its variants improve performance on both natural language understanding and generative tasks while managing computational overhead.

IniLoRA is a parameter-efficient fine-tuning method for LLMs and related pretrained transformers that modifies the initialization regime of Low-Rank Adaptation (LoRA). Rather than starting the trainable low-rank branch from a zero product, IniLoRA initializes the low-rank factors so that their product approximates the pretrained weight matrix, and then fine-tunes only that low-rank component while keeping a residual term fixed. The method is introduced in “Optimizing Fine-Tuning through Advanced Initialization Strategies for Low-Rank Adaptation” (Xue, 4 Oct 2025), which frames conventional LoRA initialization as an initialization bottleneck and proposes both a base decomposition-based method and two simpler initialization variants, IniLoRA-$\alpha$ and IniLoRA-$\beta$.

1. Conceptual background and motivation

IniLoRA is defined in the setting of parameter-efficient fine-tuning (PEFT), with LoRA as its immediate point of departure. For a pretrained weight matrix $W_0 \in \mathbb{R}^{d \times k}$, standard LoRA freezes $W_0$ and learns a low-rank update

$\Delta W = BA,$

where

$$A \in \mathbb{R}^{r \times k}, \qquad B \in \mathbb{R}^{d \times r}, \qquad r \ll \min(d,k).$$

The effective weight is conceptually

$W = W_0 + BA.$

The motivation for IniLoRA is the conventional LoRA initialization condition

$B^{(0)}A^{(0)} = 0.$

In practice, this means one low-rank factor is randomly initialized and the other is set to zero, so the low-rank branch is inactive at the start of training. The paper argues that this does not remove $W_0$, but it does cause the trainable adaptation path to begin from a degenerate update, which may make the initial optimization dynamics less aligned with full fine-tuning (Xue, 4 Oct 2025).

The core intuition is comparative. Full fine-tuning starts directly from the pretrained parameterization. Vanilla LoRA, by contrast, starts from a trainable low-rank branch whose product is exactly zero. IniLoRA is proposed to narrow that gap by making the low-rank path active from initialization and structurally aligned with the pretrained weights. This suggests that the method is less a change to LoRA’s low-rank parameterization than a redefinition of how the trainable subspace is activated at the beginning of adaptation.

2. Formal definition and decomposition

The defining operation in IniLoRA is to initialize the low-rank matrices so that their product approximates the original pretrained weight matrix $W_0$, rather than enforcing a zero product. The paper states that “IniLoRA approximates the original model weights by utilizing the product of two low-rank matrices” (Xue, 4 Oct 2025).

For each target weight matrix $\beta$0, IniLoRA constructs

$\beta$1

and optimizes them so that after $\beta$2 iterations,

$\beta$3

The approximation is obtained by minimizing the Frobenius reconstruction objective

$\beta$4

The paper gives the optimization updates explicitly as

$\beta$5

Because a rank-$\beta$6 product generally cannot exactly reconstruct a full-rank matrix when $\beta$7 is small, IniLoRA introduces a residual matrix

$\beta$8

The pretrained weight is then written as

$\beta$9

During downstream fine-tuning, $W_0 \in \mathbb{R}^{d \times k}$0 remains fixed, while only $W_0 \in \mathbb{R}^{d \times k}$1 and $W_0 \in \mathbb{R}^{d \times k}$2 are updated. This residual-plus-trainable-low-rank decomposition is the distinctive formulation of IniLoRA (Xue, 4 Oct 2025).

A common misconception is to treat IniLoRA as an SVD-based initialization scheme. The paper explicitly does not derive the method from SVD, principal singular vectors, or any closed-form low-rank factorization. Instead, it finds the factors by direct optimization of $W_0 \in \mathbb{R}^{d \times k}$3 (Xue, 4 Oct 2025).

3. Initialization statistics and method variants

Before the approximation stage, the base method initializes the low-rank factors from Gaussian statistics derived from the pretrained weights. For each layer $W_0 \in \mathbb{R}^{d \times k}$4, with weight matrix $W_0 \in \mathbb{R}^{d \times k}$5, the paper computes

$W_0 \in \mathbb{R}^{d \times k}$6

It then averages these values across layers: $W_0 \in \mathbb{R}^{d \times k}$7 These become the global initialization parameters for $W_0 \in \mathbb{R}^{d \times k}$8 and $W_0 \in \mathbb{R}^{d \times k}$9. The paper later notes that because the means are near zero, the experiments simplify this to near-zero-mean Gaussian sampling (Xue, 4 Oct 2025).

The paper introduces three named configurations.

Variant	Initialization mechanism	Pre-approximation step
IniLoRA	Gaussian from global weight statistics	Yes
IniLoRA-$W_0$0	$W_0$1, $W_0$2	Appears omitted
IniLoRA-$W_0$3	Kaiming initialization: K-N or K-U	No explicit approximation stage

Base IniLoRA uses the full approximation pipeline: initialize $W_0$4 and $W_0$5, optimize them to approximate $W_0$6, form

$W_0$7

freeze $W_0$8, and fine-tune only $W_0$9 and $\Delta W = BA,$0.

IniLoRA-$\Delta W = BA,$1 is motivated by an ablation showing that even zero approximation iterations can already be competitive with LoRA in some settings when the low-rank product is nonzero. It uses a broader Gaussian initialization

$\Delta W = BA,$2

with

$\Delta W = BA,$3

The reported interpretation is that increasing the standard deviation improves performance up to $\Delta W = BA,$4, after which performance degrades (Xue, 4 Oct 2025).

IniLoRA-$\Delta W = BA,$5 replaces Gaussian initialization with Kaiming initialization, either Kaiming normal (K-N) or Kaiming uniform (K-U). The paper does not provide a full closed-form formula, but states that both low-rank matrices are initialized from the same Kaiming distribution. This suggests a shift in emphasis from reconstructing $\Delta W = BA,$6 explicitly to choosing a nonzero initialization regime with more favorable variance propagation.

4. Training pipeline and computational characteristics

The practical pipeline in the paper applies low-rank adaptation to the query and value projection matrices in attention layers. For the base method, the preprocessing stage is performed per target weight matrix before downstream task fine-tuning (Xue, 4 Oct 2025).

The workflow is: compute global initialization statistics, initialize $\Delta W = BA,$7 and $\Delta W = BA,$8, run gradient descent on

$\Delta W = BA,$9

form the residual

$$A \in \mathbb{R}^{r \times k}, \qquad B \in \mathbb{R}^{d \times r}, \qquad r \ll \min(d,k).$$0

and then fine-tune only the low-rank factors while keeping $$A \in \mathbb{R}^{r \times k}, \qquad B \in \mathbb{R}^{d \times r}, \qquad r \ll \min(d,k).$$1 fixed. The approximation stage uses only pretrained weights; it does not require task data.

For the approximation stage, the reported configuration is: query and value modules, Adam optimizer, learning rate $$A \in \mathbb{R}^{r \times k}, \qquad B \in \mathbb{R}^{d \times r}, \qquad r \ll \min(d,k).$$2, StepLR scheduler, 20,000 total steps, and rank 8. The paper notes that convergence tends to happen around 4,000 steps (Xue, 4 Oct 2025). This is consistent with its later ablation showing rapid reconstruction-loss reduction in the first $$A \in \mathbb{R}^{r \times k}, \qquad B \in \mathbb{R}^{d \times r}, \qquad r \ll \min(d,k).$$3 iterations and a downstream-performance plateau thereafter.

A notable implementation detail is that the approximation is done once per pretrained model and the resulting factors can be cached locally. This makes the one-time decomposition cost potentially amortizable across repeated downstream fine-tuning runs.

The reported approximation overhead for query/value approximation at rank 8, concurrency 64, on an NVIDIA 4090 is substantial for larger models. RoBERTa Base requires 2.0 min and 0.8 GB; RoBERTa Large 4.1 min and 1.3 GB; LLaMA2-7B 25.8 min and 16.2 GB; Gemma-7B 18.2 min and 12.4 GB; LLaMA3-8B 15.7 min and 10.9 GB; and LLaMA2-13B 54.6 min and 24.4 GB (Xue, 4 Oct 2025). A practical implication is that base IniLoRA is most attractive when the same pretrained model is reused sufficiently often for cached approximations to offset the up-front cost.

5. Empirical evaluation

The paper evaluates IniLoRA on both NLU and generative, reasoning, and code tasks using RoBERTa Base, RoBERTa Large, LLaMA2-7B, LLaMA2-13B, LLaMA3-8B, and Gemma-7B (Xue, 4 Oct 2025). Benchmarks include GLUE tasks (CoLA, MRPC, QNLI, RTE, STS-B, SST-2), GSM8K, MATH, MMLU, and HumanEval with P@1 and P@10.

On GLUE, the reported gains are modest but positive on average. For RoBERTa Base, the average GLUE score improves from 85.2 with LoRA to 85.8 with IniLoRA; MRPC rises from 88.9 to 91.4, QNLI from 92.9 to 93.8, STS-B from 90.7 to 91.7, and SST-2 from 94.7 to 95.1, while RTE drops from 80.3 to 79.0. For RoBERTa Large, the average improves from 88.1 to 88.5; CoLA rises from 67.8 to 69.4, RTE from 87.7 to 88.1, STS-B from 91.9 to 92.1, and SST-2 from 96.5 to 97.0, while MRPC is essentially flat at 90.1 to 90.0 (Xue, 4 Oct 2025).

On LLM-style tasks, IniLoRA generally outperforms LoRA but not uniformly. For LLaMA2-7B, GSM8K improves from 20.5 to 22.5, MATH from 2.4 to 2.9, HumanEval P@1 from 14.2 to 15.3, and HumanEval P@10 from 18.3 to 18.6, while MMLU shifts from 43.6 to 43.4. For Gemma-7B, GSM8K improves from 64.5 to 66.9 and HumanEval P@1 from 36.8 to 38.0, but MATH drops from 24.8 to 23.9 and HumanEval P@10 from 64.3 to 64.0. For LLaMA3-8B, GSM8K rises from 59.6 to 60.1, MATH from 16.3 to 17.5, HumanEval P@1 from 40.8 to 41.9, and HumanEval P@10 from 50.0 to 55.3, while MMLU falls from 61.3 to 60.1. For LLaMA2-13B, GSM8K improves from 35.9 to 37.4 and HumanEval P@1 from 17.9 to 20.1, while MATH slightly declines from 5.5 to 5.3 (Xue, 4 Oct 2025).

The paper also reports strong rank and data-size robustness on LLaMA2-7B for GSM8K. Across ranks 8, 16, and 32, LoRA obtains 20.5, 21.8, and 25.2, while IniLoRA reaches 22.5, 25.8, and 27.2. Across MetaMathQA subsets of 10k, 50k, and 100k samples, IniLoRA remains ahead of LoRA at all three ranks, with larger gains at higher ranks and larger data sizes (Xue, 4 Oct 2025). This suggests that the benefits are not limited to low-data or low-rank regimes.

One of the most striking findings concerns the variants. On GSM8K with LLaMA2-7B, rank 8, LoRA scores 20.3, base IniLoRA 22.4, IniLoRA-$$A \in \mathbb{R}^{r \times k}, \qquad B \in \mathbb{R}^{d \times r}, \qquad r \ll \min(d,k).$$4 28.2, IniLoRA-$$A \in \mathbb{R}^{r \times k}, \qquad B \in \mathbb{R}^{d \times r}, \qquad r \ll \min(d,k).$$5 K-N 22.8, and IniLoRA-$$A \in \mathbb{R}^{r \times k}, \qquad B \in \mathbb{R}^{d \times r}, \qquad r \ll \min(d,k).$$6 K-U 23.4. At rank 16, the corresponding numbers are 22.2, 25.2, 30.6, 27.5, and 28.3. For LLaMA2-13B, rank 8, LoRA scores 35.9, base IniLoRA 37.4, IniLoRA-$$A \in \mathbb{R}^{r \times k}, \qquad B \in \mathbb{R}^{d \times r}, \qquad r \ll \min(d,k).$$7 45.6, IniLoRA-$$A \in \mathbb{R}^{r \times k}, \qquad B \in \mathbb{R}^{d \times r}, \qquad r \ll \min(d,k).$$8 K-N 38.2, and IniLoRA-$$A \in \mathbb{R}^{r \times k}, \qquad B \in \mathbb{R}^{d \times r}, \qquad r \ll \min(d,k).$$9 K-U 39.7; at rank 16 they are 37.9, 39.2, 48.1, 40.4, and 42.3 (Xue, 4 Oct 2025). Empirically, IniLoRA-$W = W_0 + BA.$0 is the strongest reported variant.

6. Interpretation, limitations, and relation to PEFT practice

The paper’s justification for IniLoRA is largely intuitive and empirical rather than theorem-driven. It argues that because the trainable low-rank path is initialized to approximate $W = W_0 + BA.$1, early weight adjustments more closely resemble those of full fine-tuning than those of vanilla LoRA. In this view, the decisive change is not merely nonzero initialization, but the immediate activation of a trainable branch already aligned with pretrained structure (Xue, 4 Oct 2025).

Several limitations are explicit. First, the theory is heuristic: the paper does not provide formal analyses of singular-value spectra, Hessians, gradient-flow equations, or convergence proofs. Second, the empirical improvements are not universal. There are ties and regressions on some MMLU, MATH, and isolated GLUE settings. Third, the strongest results come not from the base approximation method but from IniLoRA-$W = W_0 + BA.$2, and secondarily IniLoRA-$W = W_0 + BA.$3. This complicates a simple interpretation in which faithful approximation of $W = W_0 + BA.$4 is the sole source of performance gains. A plausible implication is that initialization scale and activation of a nonzero low-rank branch may matter at least as much as approximation fidelity.

The scope of the experiments is also limited. The paper adapts only the query and value projections, so broader module coverage is not studied. It discusses PEFT methods such as LoftQ, PiSSA, and MiLoRA, but does not provide an implementation or experiment combining IniLoRA with quantization methods such as QLoRA or LoftQ; compatibility with quantization is therefore plausible in principle but not demonstrated (Xue, 4 Oct 2025).

Within the PEFT landscape, IniLoRA’s main contribution is to elevate initialization from an implementation detail to a central design axis. The paper ultimately makes two intertwined claims: that LoRA initialization matters much more than commonly assumed, and that approximating or otherwise better activating pretrained weights at initialization can improve downstream performance (Xue, 4 Oct 2025). In that sense, IniLoRA is best understood both as a specific residual-plus-low-rank decomposition method and as a broader argument for revisiting the initialization mechanics of low-rank adaptation.