---
title: 'LiLoRA in LoRA: Continual Visual Tuning'
url: https://www.emergentmind.com/topics/lora-in-lora-lilora
type: topic
---

# LiLoRA in LoRA: Continual Visual Tuning

Searching arXiv for the LiLoRA paper and closely related CA-LoRA work to ground the article in current preprints.
{"query":"arXiv 2508.06202 LoRA in LoRA continual visual instruction tuning", "max_results": 5}
LoRA in LoRA (LiLoRA) is a continual-learning architecture expansion method for Continual Visual Instruction Tuning (CVIT) in multimodal large language models (MLLMs). It is designed for sequential learning over heterogeneous vision-language instruction tasks, where catastrophic forgetting degrades performance on earlier tasks as the model adapts to later ones. LiLoRA preserves the isolation benefits of architecture expansion while reducing parameter growth by sharing the LoRA matrix \(A\) across tasks, applying an additional low-rank decomposition to the task-specific matrix \(B\), and constraining shared-basis drift with a cosine-regularized stability loss [2508.06202].

## 1. CVIT setting and the rationale for expansion

In the CVIT formulation, the model is exposed to a task stream
\[
\mathcal{T} = \{\tau_1, \tau_2, \dots, \tau_n\},
\]
where each task \(\tau_t\) has a dataset
\[
\mathcal{D}_t = \{(X^{\text{ins}}, X^{\text{inputs}}, X^{\text{gt}})\},
\]
consisting of instruction text, multimodal inputs, and the ground-truth response. Training is sequential: when task \(t\) arrives, the model updates its parameters using only \(\mathcal{D}_t\) [2508.06202].

The central difficulty is catastrophic forgetting. Parameters that were useful for earlier tasks are overwritten while learning new tasks, so accuracy on old tasks drops as training proceeds. The problem is particularly acute in CVIT because tasks are often diverse and not naturally class-incremental; they can differ in reasoning style, output format, and domain.

LiLoRA is motivated by the observation that architecture expansion is a natural way to mitigate forgetting. Rather than forcing all tasks to share exactly the same capacity, expansion allocates task-specific parameters, which isolates task-specific knowledge and reduces interference. The difficulty, as the paper states, is that prior CVIT expansion methods often expand whole layers or large modules per task, leading to substantial parameter overhead and poor scalability. LiLoRA is introduced to preserve the isolation benefit of expansion while making it much more parameter-efficient [2508.06202].

## 2. Baseline LoRA and the nested low-rank structure of LiLoRA

LiLoRA is built on the standard LoRA parameterization. For a frozen weight matrix \(W_0 \in \mathbb{R}^{d \times k}\), LoRA introduces
\[
B \in \mathbb{R}^{d \times r}, \qquad A \in \mathbb{R}^{r \times k},
\]
with \(r \ll \min(d,k)\), and defines
\[
W' = W_0 + \Delta W = W_0 + BA.
\]
The paper notes the usual initialization pattern: \(B\) is initialized to zeros, while \(A\) is drawn from a standard Gaussian distribution [2508.06202].

A direct continual-learning extension, termed DirLoRA, assigns each task its own LoRA module:
\[
\Delta W_i = B_iA_i.
\]
This prevents interference by construction, but the parameter cost grows linearly with the number of tasks because every task stores a full pair \((B_i, A_i)\).

LiLoRA reduces this redundancy in two stages. First, it shares the matrix \(A\) globally across tasks:
\[
\Delta W_i = B_iA.
\]
The justification given in the paper is empirical: via CKA similarity analysis, the LoRA matrices \(A\) learned by different tasks are highly similar, while \(B\) matrices are much less similar. This makes \(A\) the reusable factor and \(B\) the task-adaptive factor.

Second, LiLoRA factorizes the task-specific \(B_i\) again. For task \(\tau_i\), it introduces
\[
\tilde{B}_i \in \mathbb{R}^{d \times \tilde r}, \qquad \tilde{A}_i \in \mathbb{R}^{\tilde r \times r},
\]
with \(\tilde r < r\), and writes
\[
\Delta W_i = (B_0 + \tilde{B}_i\tilde{A}_i)A,
\]
where \(B_0\) and \(A\) are shared across tasks, while \(\tilde{B}_i,\tilde{A}_i\) are task-specific. A learnable fusion coefficient \(\alpha \in (0,1)\), initialized as
\[
\alpha \sim \text{Sigmoid}(\mathcal{N}(0,1)),
\]
balances the shared basis and the task-specific residual, yielding the final update
\[
\Delta W_i = (\alpha B_0 + (1-\alpha)\tilde{B}_i\tilde{A}_i)A.
\]

This is the literal sense in which LiLoRA is “LoRA in LoRA”: the outer LoRA uses \(A\) and \(B\), and then \(B\) is itself expanded into a shared basis plus a low-rank task-specific correction. A plausible implication is that LiLoRA treats the LoRA factors themselves as structured objects, rather than as monolithic task-local parameters [2508.06202].

## 3. Objective function, stability regularization, and sequential optimization

For task \(\tau_i\), the effective LiLoRA update is
\[
\Delta W_i = (\alpha B_0 + (1-\alpha)\tilde{B}_i\tilde{A}_i)A,
\qquad
W_i' = W_0 + \Delta W_i.
\]
The training objective combines the task loss with a basis-stability regularizer:
\[
\min \ \mathcal{L}_{\text{task}} + \lambda \mathcal{L}_{\text{reg}},
\]
where \(\lambda\) is a regularization weight and \(\mathcal{L}_{\text{task}}\) is described as an autoregressive loss for the instruction-tuning objective [2508.06202].

The regularizer addresses a specific continual-learning failure mode. Since \(B_0\) is shared across sequential tasks, updating it for a new task can disrupt representations needed by previous tasks. LiLoRA therefore computes
\[
sim_t = \cos\left(\tilde{B}_t\tilde{A}_t,\ \tilde{B}_{t-1}\tilde{A}_{t-1}\right),
\]
and defines
\[
\mathcal{L}_{reg} = (1-sim_t)\cdot \left\|B_0^t - B_0^{t-1}\right\|^2.
\]
If the current task’s residual representation is similar to the previous one, \(sim_t\) is large and the penalty on changing \(B_0\) is weaker; if it is dissimilar, the penalty is stronger. The mechanism therefore preserves shared representations over time by conditioning permissible basis drift on similarity between consecutive task-specific subspaces [2508.06202].

The training procedure is sequential. The pretrained MLLM backbone \(M\) is frozen. For the first task, \(B_0\) and \(A\) are initialized, together with task-specific \(\tilde{B}_1,\tilde{A}_1\). For each later task \(\tau_t\), previously learned shared parameters are retained, new task-specific matrices \(\tilde{B}_t,\tilde{A}_t\) are added, the task update \(\Delta W_t\) is computed, and optimization proceeds with
\[
\mathcal{L}_{\text{task}} + \lambda \mathcal{L}_{reg}.
\]
The shared parameters \(B_0\) and \(A\) are updated throughout training but constrained by \(\mathcal{L}_{reg}\); task-specific matrices are added as tasks arrive and retained for future use.

At inference time, LiLoRA can be merged into the pretrained weights, so there is no extra runtime overhead. For a given task, the model uses that task’s corresponding low-rank residual together with the shared components, but these can be folded into the weights before inference [2508.06202].

## 4. Parameter efficiency and empirical evaluation

The parameter-efficiency argument in LiLoRA is explicit. Relative to DirLoRA, which stores a full \((B_i, A_i)\) per task, LiLoRA reuses one shared \(A\), one shared basis \(B_0\), and only stores a reduced-rank pair \((\tilde{B}_i,\tilde{A}_i)\) per task. The paper does not derive a symbolic closed-form parameter-count formula in the text, but it states the qualitative comparison:
- DirLoRA per task: \(dr + rk\)
- LiLoRA per task: \(d\tilde r + \tilde r r\), plus shared overhead \(dr + rk\) amortized across tasks

Since \(\tilde r < r\), the task-specific storage is much smaller than a full \(d \times r\) matrix. This suggests that LiLoRA’s main compression lever is not eliminating task-specificity, but compressing the task-specific component into a lower-rank residual [2508.06202].

Experiments are conducted on the CVIT Benchmark with six instruction datasets: ScienceQA, TextVQA, Flickr30k, ImageNet, GQA, and VQAv2. Two sequential settings are used: Single-type instruction and Five-type instruction. The base model is LLaVA-v1.5-7B without instruction tuning. LiLoRA adapters are inserted into the FFN layers of the LLM and the projection layer between the vision encoder and the LLM. Reported hyperparameters are shared rank \(r = 128\), task-specific rank \(\tilde r = r/2\) by default, Adam optimizer, learning rate \(2\times 10^{-5}\), batch size 64, and one epoch per task [2508.06202].

The paper reports four metrics:
\[
\mathrm{AP}_{k} = \frac{1}{k}\sum_{j=1}^{k} a_{k,j}, \qquad
\mathrm{MAP}_{k} = \frac{1}{k}\sum_{i=1}^{k} \mathrm{AP}_i,
\]
\[
\mathrm{BWT}_{k} = \frac{1}{k-1}\sum_{j=1}^{k-1}(a_{k,j}-a_{j,j}), \qquad
\mathrm{MIF}_{k} = \frac{1}{k}\sum_{j=1}^{k}\left(\frac{1}{n}\sum_{i=1}^{n}\mathcal{B}_j(o_i^j)\right).
\]
These correspond respectively to Average Performance, Mean Average Performance, Backward Transfer, and Mean Instruction Following.

The main empirical claim is that LiLoRA consistently outperforms all baselines on both sequential settings. Against SMoLoRA in the Single-type setting, LiLoRA improves by \(+1.44\) AP, \(+2.85\) MAP, \(+0.10\) BWT, and \(+0.45\) MIF. In the Five-type setting, it improves over SMoLoRA by \(+1.92\) AP, \(+2.58\) MAP, \(+4.56\) BWT, and \(+0.04\) MIF. The paper also emphasizes that LiLoRA remains close to the upper-bound DirLoRA while using much less memory, and that it again outperforms SeqLo

Source: https://www.emergentmind.com/topics/lora-in-lora-lilora