---
title: Continual Visual Instruction Tuning
url: https://www.emergentmind.com/topics/continual-visual-instruction-tuning-cvit
type: topic
---

# Continual Visual Instruction Tuning

Searching arXiv for recent CVIT papers to ground the article.
Continual Visual Instruction Tuning (CVIT) denotes the continual adaptation of multimodal large language models (MLLMs) or large vision-language models (LVLMs) to a sequence or stream of visual instruction tasks, rather than a single static multitask corpus. In this setting, the model receives new image-text instruction data over time and must acquire new capabilities while preserving previously learned visual understanding and instruction-following behavior [2311.16206]. The term is used in two closely related senses in the literature: a sequential task-by-task regime, in which datasets arrive one after another, and an online streaming regime, in which multimodal data continuously arrive with non-stationary distributions [2506.02011]. Across both regimes, CVIT is characterized by catastrophic forgetting, interference across heterogeneous tasks, and the practical constraints of parameter efficiency, memory, latency, and, increasingly, safety preservation [2411.02564].

## 1. Emergence and scope of the field

The first benchmark explicitly targeting continual instruction tuning in large multimodal models was introduced in "Continual Instruction Tuning for Large Multimodal Models" [2311.16206]. That study established that catastrophic forgetting remains present when instruction-tuned multimodal systems are updated sequentially, and it framed CVIT as a distinct continual-learning problem in which each task is represented as an image–instruction–output mapping rather than a fixed classifier head. It also showed that multi-task joint instruction tuning provides a stronger initialization for subsequent continual learning than pure image-text alignment alone.

Subsequent work broadened both the problem definition and the evaluation protocols. "Continual LLaVA" introduced COAST, a benchmark spanning domain-incremental, capability-incremental, and dataset-incremental streams for LVLMs [2411.02564]. "SMoLoRA" argued that CVIT involves a dual form of forgetting—visual understanding and instruction following—and extended benchmarking to unseen downstream tasks and diverse instruction types [2411.13949]. "LLaVA-c" shifted emphasis from task-specific retention alone to the preservation of general multimodal capability, identifying base model degradation as a distinct failure mode during continual adaptation [2506.08666].

By 2025, the scope of CVIT had further expanded along three axes. First, "OASIS" formulated online CVIT as a streaming problem in which full-data training can create training delay and impair real-time adaptation [2506.02011]. Second, "LiLoRA" focused on parameter-efficient architecture expansion tailored to sequential CVIT in MLLMs [2508.06202]. Third, "MCITlib" reframed the area more broadly as Multimodal Continual Instruction Tuning (MCIT), emphasizing benchmarking, reproducibility, and leakage-resistant evaluation protocols for multimodal continual instruction tuning research [2508.07307].

A parallel but adjacent line of work is represented by "SwitchCIT," which studies continual instruction tuning for text-generation tasks rather than vision-language benchmarks [2407.11780]. Its routing-based design is directly relevant to CVIT conceptually, but its experiments do not involve visual encoders, image tasks, or multimodal datasets.

## 2. Formal problem settings

In the sequential formulation, CVIT is defined over a task stream such as
\[
[\mathcal{T}_1,\ldots,\mathcal{T}_N]
\]
with corresponding datasets
\[
[\mathcal{D}_1,\ldots,\mathcal{D}_N].
\]
At time step \(i\), only \(\mathcal{D}_i\) is available, except in replay-based methods that retain a small memory buffer [2311.16206]. For instruction tuning, each dataset is typically written as
\[
\mathcal{D}_i=\{(\boldsymbol{t}^i_j,\boldsymbol{v}^i_j,\boldsymbol{o}^i_j)\}_{j=1}^{N_i},
\]
where \(\boldsymbol{t}^i_j\) is the text instruction, \(\boldsymbol{v}^i_j\) is the image, and \(\boldsymbol{o}^i_j\) is the text output [2311.16206]. A closely related notation used in later work represents each example as \(\{X^{ins},X^{vis},X^{ans}\}\) [2411.13949].

A second formulation emphasizes sequential visual instruction datasets for LVLMs:
\[
\{\mathcal{D}_t\}_{t=1}^{T}, \qquad \mathcal{D}_t=\{(x_t^i,y_t^i,r_t^i)\}_{i=1}^{|\mathcal{D}_t|},
\]
where \(x_t^i\) is an input image, \(y_t^i\) is a textual instruction, and \(r_t^i\) is the target response [2411.02564]. In both formulations, the model is expected to improve on the current task without substantially degrading prior-task performance.

The online formulation introduced by OASIS is stricter. There, multi-modal data arrive “in an online streaming manner,” future data are unknown, and non-stationary batch distributions make fixed per-batch selection policies problematic [2506.02011]. In that setting, training on all incoming data can create a serious training delay: the model spends too much time processing earlier batches and adapts too slowly to later tasks. This shifts the emphasis from continual learning under task boundaries to continual learning under stream responsiveness constraints.

A further extension appears in safety-aligned CVIT. "Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMs" distinguishes **pre-SA CVIT**, where the model is not yet safety-aligned, from **post-SA CVIT**, where a previously aligned MLLM must preserve both downstream task performance and safety behavior during continual adaptation [2511.20158]. This adds a second objective beyond ordinary retention.

## 3. Principal failure modes

The canonical difficulty in CVIT is catastrophic forgetting: updating shared backbone parameters for a new task degrades performance on earlier tasks [2508.06202]. In multimodal systems, the same backbone is reused across heterogeneous tasks such as VQA, classification, captioning, and reasoning, and task-specific gradients can conflict. The problem is accentuated by differences in instruction formats and by the need to maintain visual-language alignment over time [2508.06202].

Later work has shown that task accuracy alone is not a sufficient description of failure. "SMoLoRA" identifies **dual catastrophic forgetting**, consisting of **visual understanding forgetting** and **instruction-following forgetting** [2411.13949]. The first appears as declining accuracy on earlier visual tasks after later tasks are learned. The second appears when the model no longer answers in the required format or style, even if some task knowledge remains. This is why SMoLoRA introduces **Mean Instruction Following (MIF)** as an explicit evaluation metric [2411.13949].

"LLaVA-c" isolates another phenomenon, **base model degradation** [2506.08666]. In that account, continual fine-tuning on highly task-specific datasets can overfit the model to narrow instruction styles and degrade general multimodal instruction-following ability. The paper’s qualitative examples describe models that begin answering every prompt in a task-biased style such as single-word responses or captions, even when the prompt requires open-ended reasoning. This diagnosis differs from ordinary forgetting because the loss of general capability can occur even when some task-specific scores remain competitive.

"OASIS" highlights an additional failure mode specific to online CVIT: **training delay** under large-scale streaming data [2506.02011]. In this view, the principal issue is not only forgetting but also adaptation latency. Fixed top-\(k\) selection policies are criticized because informativeness varies across batches, so a rigid quota can waste budget on low-value batches and undersample high-value ones. OASIS also emphasizes intra-batch redundancy: near-duplicate examples can consume training budget without improving diversity [2506.02011].

In safety-aligned settings, post-SA CVIT introduces **safety degradation** as a coequal concern. HPA reports that continual fine-tuning can progressively weaken safety-aligned behavior, increasing the likelihood that the model answers harmful multimodal prompts instead of refusing or responding safely [2511.20158]. This reframes CVIT as a joint preservation problem over task capability and alignment.

## 4. Methodological families

CVIT methods span replay, regularization, architecture expansion, routing/expert selection, incremental embedding adaptation, consolidation, and online sample selection. Early evidence showed that **replay** and **model expansion** are robust strategies, while **regularization-based methods** perform well mainly when the starting model has already undergone multi-task joint instruction tuning [2311.16206]. Later work concentrated on parameter-efficient variants adapted to MLLM architectures.

| Work | Core mechanism | Salient property |
|---|---|---|
| Continual Instruction Tuning for Large Multimodal Models | Replay, EProj, regularization, task similarity | First benchmark; similarity-informed CL [2311.16206] |
| Continual LLaVA | Intrinsic and contextual increment embeddings | Rehearsal-free PEFT on COAST [2411.02564] |
| SMoLoRA | Separable VU/IF routing over LoRA experts | Targets dual forgetting [2411.13949] |
| LLaVA-c | SAC + UIR | Balances task retention and general capability [2506.08666] |
| OASIS | ORIS + SIREN | Adaptive online sample selection [2506.02011] |
| LiLoRA | Shared \(A\), decomposed \(B\), stability loss | Parameter-efficient architecture expansion [2508.06202] |
| HPA | Parameter partition, balanced selection, orthogonal adjustment | Post-SA CVIT for safety-aligned MLLMs [2511.20158] |

"Continual LLaVA" freezes the LVLM backbone and learns only a small set of low-rank **increment embeddings** [2411.02564]. Its intrinsic increment embeddings are selected from a pool of learnable proxy-increment pairs using instruction similarity computed by a frozen Sentence-BERT encoder, while contextual increment embeddings aggregate prior task summaries through a learnable weighted sum. The adapted projection is written as
\[
W' = W_0 + \Delta\theta_t^i + \Delta\delta_t^i.
\]
Only one output projection in the attention block is reparameterized, and the reported tunable parameter count is about **0.75M** versus **6.76B** for full-model sequential finetuning [2411.02564].

"SMoLoRA" adopts a separable mixture-of-LoRA design to isolate visual and instruction competencies [2411.13949]. It decomposes adaptation into a **visual understanding (VU) module** and an **instruction following (IF) module**:
\[
\Delta W^{vu} = \sum_{i=1}^{M} G^{vu}(z^{vu})_{i}\cdot \Delta W_i, \qquad
\Delta W^{if} = \sum_{j=M+1}^{N} G^{if}(z^{if})_{j}\cdot \Delta W_j.
\]
The layer output fuses the two branches adaptively:
\[
x_l = W_0x_{l-1}+\mathcal{F}(\Delta W^{vu}x_{l-1},\Delta W^{if}x_{l-1}).
\]
The VU router uses the instance representation, whereas the IF router uses the instruction embedding directly [2411.13949].

"LiLoRA" is an architecture-expansion method grounded in LoRA factorization [2508.06202]. Standard LoRA is
\[
W' = W_0 + BA,
\]
with low-rank matrices \(A\) and \(B\). LiLoRA shares the LoRA matrix \(A\) across tasks and further decomposes \(B\) into a shared basis plus a task-specific low-rank residual:
\[
\Delta W_i = \bigl(\alpha B_0 + (1-\alpha)\tilde B_i \tilde A_i\bigr)A.
\]
A cosine-regularized basis stability loss,
\[
\mathcal{L}_{reg} = (1 - sim_t)\,\left\|B_0^{t} - B_0^{t-1}\right\|^2,
\]
is used to preserve the shared basis when adjacent task residuals are dissimilar [2508.06202].

"LLaVA-c" modifies LLaVA-1.5 with **Spectral-Aware Consolidation (SAC)** and **Unsupervised Inquiry Regularization (UIR)** [2506.08666]. SAC treats the update \(\Delta\bm{\theta}\) spectrally via SVD and replaces uniform interpolation with a sliding-window average over the singular spectrum, while UIR regularizes hidden representations on unlabeled inquiry text:
\[
\mathcal{L}^{t}_{\text{uir}} = \mathbb{E}_{\bm{x}\sim\mathcal{D}_{\text{inquiry}}}
\left\|h[f_{\bm{\theta}_t}(\bm{x})]-h[f_{\bm{\theta}_{t-1}}(\bm{x})]\right\|_2.
\]
The method is explicitly designed to prevent base-model degradation while maintaining new-task plasticity [2506.08666].

"OASIS" addresses a different axis of the problem: which samples to process under online data streams [2506.02011]. Its **Online Relative Informativeness Selection (ORIS)** scores each sample using a Fisher Information–based measure on last-layer gradients,
\[
I_i^{(t)} = \mathrm{tr}\left[\left(\nabla_{\theta_L} L(d_i^{(t)})\right)\left(\nabla_{\theta_L} L(d_i^{(t)})\right)^\top\right],
\]
then normalizes scores with an exponential moving average and variance to compute relative informativeness across batches. Its **Similarity-aware Information Redundancy Elimination (SIREN)** iteratively subtracts similarity-weighted contributions from already selected samples:
\[
\widetilde I_i^{(t)} = I_i^{(t)} - \sum_{h\in H}\cos(g_i,g_h)\cdot I_h.
\]
The implementation requires only a single forward pass plus last-layer gradients [2506.02011].

For post-safety-alignment CVIT, HPA partitions parameters into safety-focused and task-focused groups using Hessian-inspired sensitivity approximations, selects layerwise masks to retain balanced subsets from old and new models, and applies orthogonal parameter adjustment by subtracting the projection of the update onto the previous weight space [2511.20158]. This design is explicitly post-training: it modifies weights after ordinary task fine-tuning rather than changing the original training pipeline.

A related but non-visual design pattern is provided by SwitchCIT, which uses a lightweight switch network—OPT-125M plus a 2-layer MLP classifier of **154K parameters**—to route an instruction to the corresponding task-specific LoRA expert [2407.11780]. Because its experiments are text-only, it does not establish CVIT results directly, but it demonstrates a routing-based PEFT template relevant to multimodal continual instruction tuning.

## 5. Benchmarks and evaluation protocols

CVIT benchmarking has diversified rapidly. The earliest benchmark in [2311.16206] defined two settings. **Benchmark 1** starts from BLIP2 and uses the order
\[
\text{Flickr30k} \rightarrow \text{TextCaps} \rightarrow \text{VQA v2} \rightarrow \text{OCR-VQA} \rightarrow \text{GQA}.
\]
**Benchmark 2** starts from InstructBLIP and uses
\[
\text{Multi-task} \rightarrow \text{Flickr30k} \rightarrow \text{VizWiz} \rightarrow \text{TextVQA} \rightarrow \text{GQA}.
\]
The distinction was designed to test whether prior multi-task instruction tuning facilitates continual learning [2311.16206].

COAST, introduced with Continual LLaVA, defines three stream types [2411.02564]. **COAST-domain** uses ChartQA, DocVQA, IconQA, and MedicalQA, each with **20,000** training samples and **5,000** evaluation samples. **COAST-capability** uses conversation, detail description, complex reasoning, and referring question answering, again with **20,000/5,000** train/eval splits. **COAST-dataset** chains VQAv2, VizWiz, ScienceQA, TextVQA, GQA, OCR-VQA, ImageNet, RefCOCO, RefCOCO+, and RefCOCOg [2411.02564].

The CVIT Benchmark used by SMoLoRA and LiLoRA contains six instruction datasets—ScienceQA, TextVQA, Flickr30k, ImageNet, GQA, and VQAv2—and is evaluated under **single-type instruction** and **five-type instruction** protocols [2411.13949; 2508.06202]. SMoLoRA further extends evaluation with downstream transfer to VizWiz, TextCaps, OCRVQA, and Places365, with both zero-shot and few-shot settings [2411.13949].

OASIS evaluates on three major CVIT benchmarks: **MICVIT**, **COAST**, and **Adapt** [2506.02011]. MICVIT includes seven tasks: Bongard-OpenWorld, NLVR2, Co-Instruct-DB, Bongard-HOI, PatternCom, DVQA, and HQ-Edit. COAST is instantiated there with ChartQA, DocVQA, IconQA, and MedicalQA. Adapt contains M3IT, MANTIS, LaMM, and VisionFLAN [2506.02011].

MCITlib introduces two carefully selected benchmarks intended to reduce leakage from pretraining or prior instruction tuning [2508.07307]. **UCIT** consists of ImageNet-R, ArxivQA, VizWiz-Caption, IconQA, CLEVR-Math, and Flickr30k, while **MLLM-DCL** covers remote sensing, medical, autonomous driving, science, and finance domains through datasets such as RSVQA, PathVQA, DriveLM, FinVis, and AI2D/Sciverse/MapQA/TQA [2508.07307]. The paper explicitly identifies benchmark leakage and inconsistent comparison protocols as major evaluation problems.

Metric design reflects the multidimensional nature of CVIT. OASIS reports **\(A_{last}\)** and **\(A_{avg}\)** [2506.02011]. SMoLoRA and LiLoRA use **AP**, **MAP**, **BWT**, and **MIF** [2411.13949; 2508.06202]. Continual LLaVA uses **Average Accuracy (AA)** and **Average Forgetting (AF)**, with GPT-4o–assisted evaluation for free-form outputs [2411.02564]. MCITlib follows **MFT**, **MFN**, **MAA**, and **BWT**, and argues that MFN and MAA are often more informative than BWT alone because some methods reduce forgetting by learning less [2508.07307]. Safety-aligned CVIT adds **ASR**, **MASR**, and **DASR** as safety-retention metrics [2511.20158].

## 6. Representative empirical findings

The foundational empirical result is that catastrophic forgetting persists in continual multimodal instruction tuning, but its severity depends strongly on initialization and method class [2311.16206]. In the earliest benchmark study, sequential fine-tuning caused severe forgetting; replay and model expansion were consistently effective; and regularization-based methods performed well mainly when the model had already been jointly instruction-tuned on multiple tasks. The same study also found that similar task pairs forget each other less and transfer better,

Source: https://www.emergentmind.com/topics/continual-visual-instruction-tuning-cvit