LiLoRA in LoRA: Continual Visual Tuning

• LiLoRA is a continual learning method for multimodal instruction tuning that employs low-rank matrix sharing and task-specific decompositions to preserve past knowledge.
• By factorizing task-specific LoRA modules into a shared matrix A and a compressed residual, LiLoRA achieves significant parameter efficiency over traditional DirLoRA approaches.
• Empirical results on diverse vision-language benchmarks demonstrate that LiLoRA improves key metrics such as AP, MAP, BWT, and MIF while reducing memory overhead.

Searching arXiv for the LiLoRA paper and closely related CA-LoRA work to ground the article in current preprints. {"query":"arXiv (Che et al., 8 Aug 2025) LoRA in LoRA continual visual instruction tuning", "max_results": 5} LoRA in LoRA (LiLoRA) is a continual-learning architecture expansion method for Continual Visual Instruction Tuning (CVIT) in multimodal LLMs (MLLMs). It is designed for sequential learning over heterogeneous vision-language instruction tasks, where catastrophic forgetting degrades performance on earlier tasks as the model adapts to later ones. LiLoRA preserves the isolation benefits of architecture expansion while reducing parameter growth by sharing the LoRA matrix $A$ across tasks, applying an additional low-rank decomposition to the task-specific matrix $B$, and constraining shared-basis drift with a cosine-regularized stability loss (Che et al., 8 Aug 2025).

1. CVIT setting and the rationale for expansion

In the CVIT formulation, the model is exposed to a task stream

$\mathcal{T} = \{\tau_1, \tau_2, \dots, \tau_n\},$

where each task $\tau_t$ has a dataset

$$\mathcal{D}_t = \{(X^{\text{ins}}, X^{\text{inputs}}, X^{\text{gt}})\},$$

consisting of instruction text, multimodal inputs, and the ground-truth response. Training is sequential: when task $t$ arrives, the model updates its parameters using only $\mathcal{D}_t$ (Che et al., 8 Aug 2025).

The central difficulty is catastrophic forgetting. Parameters that were useful for earlier tasks are overwritten while learning new tasks, so accuracy on old tasks drops as training proceeds. The problem is particularly acute in CVIT because tasks are often diverse and not naturally class-incremental; they can differ in reasoning style, output format, and domain.

LiLoRA is motivated by the observation that architecture expansion is a natural way to mitigate forgetting. Rather than forcing all tasks to share exactly the same capacity, expansion allocates task-specific parameters, which isolates task-specific knowledge and reduces interference. The difficulty, as the paper states, is that prior CVIT expansion methods often expand whole layers or large modules per task, leading to substantial parameter overhead and poor scalability. LiLoRA is introduced to preserve the isolation benefit of expansion while making it much more parameter-efficient (Che et al., 8 Aug 2025).

2. Baseline LoRA and the nested low-rank structure of LiLoRA

LiLoRA is built on the standard LoRA parameterization. For a frozen weight matrix $W_0 \in \mathbb{R}^{d \times k}$, LoRA introduces

$$B \in \mathbb{R}^{d \times r}, \qquad A \in \mathbb{R}^{r \times k},$$

with $r \ll \min(d,k)$, and defines

$B$0

The paper notes the usual initialization pattern: $B$1 is initialized to zeros, while $B$2 is drawn from a standard Gaussian distribution (Che et al., 8 Aug 2025).

A direct continual-learning extension, termed DirLoRA, assigns each task its own LoRA module: $B$3 This prevents interference by construction, but the parameter cost grows linearly with the number of tasks because every task stores a full pair $B$4.

LiLoRA reduces this redundancy in two stages. First, it shares the matrix $B$5 globally across tasks: $B$6 The justification given in the paper is empirical: via CKA similarity analysis, the LoRA matrices $B$7 learned by different tasks are highly similar, while $B$8 matrices are much less similar. This makes $B$9 the reusable factor and $\mathcal{T} = \{\tau_1, \tau_2, \dots, \tau_n\},$0 the task-adaptive factor.

Second, LiLoRA factorizes the task-specific $\mathcal{T} = \{\tau_1, \tau_2, \dots, \tau_n\},$1 again. For task $\mathcal{T} = \{\tau_1, \tau_2, \dots, \tau_n\},$2, it introduces

$\mathcal{T} = \{\tau_1, \tau_2, \dots, \tau_n\},$3

with $\mathcal{T} = \{\tau_1, \tau_2, \dots, \tau_n\},$4, and writes

$\mathcal{T} = \{\tau_1, \tau_2, \dots, \tau_n\},$5

where $\mathcal{T} = \{\tau_1, \tau_2, \dots, \tau_n\},$6 and $\mathcal{T} = \{\tau_1, \tau_2, \dots, \tau_n\},$7 are shared across tasks, while $\mathcal{T} = \{\tau_1, \tau_2, \dots, \tau_n\},$8 are task-specific. A learnable fusion coefficient $\mathcal{T} = \{\tau_1, \tau_2, \dots, \tau_n\},$9, initialized as

$\tau_t$0

balances the shared basis and the task-specific residual, yielding the final update

$\tau_t$1

This is the literal sense in which LiLoRA is “LoRA in LoRA”: the outer LoRA uses $\tau_t$2 and $\tau_t$3, and then $\tau_t$4 is itself expanded into a shared basis plus a low-rank task-specific correction. A plausible implication is that LiLoRA treats the LoRA factors themselves as structured objects, rather than as monolithic task-local parameters (Che et al., 8 Aug 2025).

3. Objective function, stability regularization, and sequential optimization

For task $\tau_t$5, the effective LiLoRA update is

$\tau_t$6

The training objective combines the task loss with a basis-stability regularizer: $\tau_t$7 where $\tau_t$8 is a regularization weight and $\tau_t$9 is described as an autoregressive loss for the instruction-tuning objective (Che et al., 8 Aug 2025).

The regularizer addresses a specific continual-learning failure mode. Since $$\mathcal{D}_t = \{(X^{\text{ins}}, X^{\text{inputs}}, X^{\text{gt}})\},$$0 is shared across sequential tasks, updating it for a new task can disrupt representations needed by previous tasks. LiLoRA therefore computes

$$\mathcal{D}_t = \{(X^{\text{ins}}, X^{\text{inputs}}, X^{\text{gt}})\},$$1

and defines

$$\mathcal{D}_t = \{(X^{\text{ins}}, X^{\text{inputs}}, X^{\text{gt}})\},$$2

If the current task’s residual representation is similar to the previous one, $$\mathcal{D}_t = \{(X^{\text{ins}}, X^{\text{inputs}}, X^{\text{gt}})\},$$3 is large and the penalty on changing $$\mathcal{D}_t = \{(X^{\text{ins}}, X^{\text{inputs}}, X^{\text{gt}})\},$$4 is weaker; if it is dissimilar, the penalty is stronger. The mechanism therefore preserves shared representations over time by conditioning permissible basis drift on similarity between consecutive task-specific subspaces (Che et al., 8 Aug 2025).

The training procedure is sequential. The pretrained MLLM backbone $$\mathcal{D}_t = \{(X^{\text{ins}}, X^{\text{inputs}}, X^{\text{gt}})\},$$5 is frozen. For the first task, $$\mathcal{D}_t = \{(X^{\text{ins}}, X^{\text{inputs}}, X^{\text{gt}})\},$$6 and $$\mathcal{D}_t = \{(X^{\text{ins}}, X^{\text{inputs}}, X^{\text{gt}})\},$$7 are initialized, together with task-specific $$\mathcal{D}_t = \{(X^{\text{ins}}, X^{\text{inputs}}, X^{\text{gt}})\},$$8. For each later task $$\mathcal{D}_t = \{(X^{\text{ins}}, X^{\text{inputs}}, X^{\text{gt}})\},$$9, previously learned shared parameters are retained, new task-specific matrices $t$0 are added, the task update $t$1 is computed, and optimization proceeds with

$t$2

The shared parameters $t$3 and $t$4 are updated throughout training but constrained by $t$5; task-specific matrices are added as tasks arrive and retained for future use.

At inference time, LiLoRA can be merged into the pretrained weights, so there is no extra runtime overhead. For a given task, the model uses that task’s corresponding low-rank residual together with the shared components, but these can be folded into the weights before inference (Che et al., 8 Aug 2025).

4. Parameter efficiency and empirical evaluation

The parameter-efficiency argument in LiLoRA is explicit. Relative to DirLoRA, which stores a full $t$6 per task, LiLoRA reuses one shared $t$7, one shared basis $t$8, and only stores a reduced-rank pair $t$9 per task. The paper does not derive a symbolic closed-form parameter-count formula in the text, but it states the qualitative comparison:

• DirLoRA per task: $\mathcal{D}_t$0
• LiLoRA per task: $\mathcal{D}_t$1, plus shared overhead $\mathcal{D}_t$2 amortized across tasks

Since $\mathcal{D}_t$3, the task-specific storage is much smaller than a full $\mathcal{D}_t$4 matrix. This suggests that LiLoRA’s main compression lever is not eliminating task-specificity, but compressing the task-specific component into a lower-rank residual (Che et al., 8 Aug 2025).

Experiments are conducted on the CVIT Benchmark with six instruction datasets: ScienceQA, TextVQA, Flickr30k, ImageNet, GQA, and VQAv2. Two sequential settings are used: Single-type instruction and Five-type instruction. The base model is LLaVA-v1.5-7B without instruction tuning. LiLoRA adapters are inserted into the FFN layers of the LLM and the projection layer between the vision encoder and the LLM. Reported hyperparameters are shared rank $\mathcal{D}_t$5, task-specific rank $\mathcal{D}_t$6 by default, Adam optimizer, learning rate $\mathcal{D}_t$7, batch size 64, and one epoch per task (Che et al., 8 Aug 2025).

The paper reports four metrics: $\mathcal{D}_t$8

$\mathcal{D}_t$9

These correspond respectively to Average Performance, Mean Average Performance, Backward Transfer, and Mean Instruction Following.

The main empirical claim is that LiLoRA consistently outperforms all baselines on both sequential settings. Against SMoLoRA in the Single-type setting, LiLoRA improves by $W_0 \in \mathbb{R}^{d \times k}$0 AP, $W_0 \in \mathbb{R}^{d \times k}$1 MAP, $W_0 \in \mathbb{R}^{d \times k}$2 BWT, and $W_0 \in \mathbb{R}^{d \times k}$3 MIF. In the Five-type setting, it improves over SMoLoRA by $W_0 \in \mathbb{R}^{d \times k}$4 AP, $W_0 \in \mathbb{R}^{d \times k}$5 MAP, $W_0 \in \mathbb{R}^{d \times k}$6 BWT, and $W_0 \in \mathbb{R}^{d \times k}$7 MIF. The paper also emphasizes that LiLoRA remains close to the upper-bound DirLoRA while using much less memory, and that it again outperforms SeqLo