Continual Visual Instruction Tuning

• Continual Visual Instruction Tuning (CVIT) is the process of sequentially adapting multimodal models to new visual instructions while preserving past knowledge.
• Key methodologies include replay, regularization, architecture expansion, and parameter-efficient techniques to mitigate catastrophic forgetting and maintain performance.
• Recent benchmarks and metrics, such as COAST and Mean Instruction Following (MIF), assess CVIT's responsiveness to online data streams and safety alignment challenges.

Searching arXiv for recent CVIT papers to ground the article. Continual Visual Instruction Tuning (CVIT) denotes the continual adaptation of multimodal LLMs (MLLMs) or large vision-LLMs (LVLMs) to a sequence or stream of visual instruction tasks, rather than a single static multitask corpus. In this setting, the model receives new image-text instruction data over time and must acquire new capabilities while preserving previously learned visual understanding and instruction-following behavior (He et al., 2023). The term is used in two closely related senses in the literature: a sequential task-by-task regime, in which datasets arrive one after another, and an online streaming regime, in which multimodal data continuously arrive with non-stationary distributions (Lee et al., 27 May 2025). Across both regimes, CVIT is characterized by catastrophic forgetting, interference across heterogeneous tasks, and the practical constraints of parameter efficiency, memory, latency, and, increasingly, safety preservation (Cao et al., 2024).

1. Emergence and scope of the field

The first benchmark explicitly targeting continual instruction tuning in large multimodal models was introduced in "Continual Instruction Tuning for Large Multimodal Models" (He et al., 2023). That study established that catastrophic forgetting remains present when instruction-tuned multimodal systems are updated sequentially, and it framed CVIT as a distinct continual-learning problem in which each task is represented as an image–instruction–output mapping rather than a fixed classifier head. It also showed that multi-task joint instruction tuning provides a stronger initialization for subsequent continual learning than pure image-text alignment alone.

Subsequent work broadened both the problem definition and the evaluation protocols. "Continual LLaVA" introduced COAST, a benchmark spanning domain-incremental, capability-incremental, and dataset-incremental streams for LVLMs (Cao et al., 2024). "SMoLoRA" argued that CVIT involves a dual form of forgetting—visual understanding and instruction following—and extended benchmarking to unseen downstream tasks and diverse instruction types (Wang et al., 2024). "LLaVA-c" shifted emphasis from task-specific retention alone to the preservation of general multimodal capability, identifying base model degradation as a distinct failure mode during continual adaptation (Liu et al., 10 Jun 2025).

By 2025, the scope of CVIT had further expanded along three axes. First, "OASIS" formulated online CVIT as a streaming problem in which full-data training can create training delay and impair real-time adaptation (Lee et al., 27 May 2025). Second, "LiLoRA" focused on parameter-efficient architecture expansion tailored to sequential CVIT in MLLMs (Che et al., 8 Aug 2025). Third, "MCITlib" reframed the area more broadly as Multimodal Continual Instruction Tuning (MCIT), emphasizing benchmarking, reproducibility, and leakage-resistant evaluation protocols for multimodal continual instruction tuning research (Guo et al., 10 Aug 2025).

A parallel but adjacent line of work is represented by "SwitchCIT," which studies continual instruction tuning for text-generation tasks rather than vision-language benchmarks (Wu et al., 2024). Its routing-based design is directly relevant to CVIT conceptually, but its experiments do not involve visual encoders, image tasks, or multimodal datasets.

2. Formal problem settings

In the sequential formulation, CVIT is defined over a task stream such as

$[\mathcal{T}_1,\ldots,\mathcal{T}_N]$

with corresponding datasets

$[\mathcal{D}_1,\ldots,\mathcal{D}_N].$

At time step $i$, only $\mathcal{D}_i$ is available, except in replay-based methods that retain a small memory buffer (He et al., 2023). For instruction tuning, each dataset is typically written as

$$\mathcal{D}_i=\{(\boldsymbol{t}^i_j,\boldsymbol{v}^i_j,\boldsymbol{o}^i_j)\}_{j=1}^{N_i},$$

where $\boldsymbol{t}^i_j$ is the text instruction, $\boldsymbol{v}^i_j$ is the image, and $\boldsymbol{o}^i_j$ is the text output (He et al., 2023). A closely related notation used in later work represents each example as $\{X^{ins},X^{vis},X^{ans}\}$ (Wang et al., 2024).

A second formulation emphasizes sequential visual instruction datasets for LVLMs: $$\{\mathcal{D}_t\}_{t=1}^{T}, \qquad \mathcal{D}_t=\{(x_t^i,y_t^i,r_t^i)\}_{i=1}^{|\mathcal{D}_t|},$$ where $[\mathcal{D}_1,\ldots,\mathcal{D}_N].$0 is an input image, $[\mathcal{D}_1,\ldots,\mathcal{D}_N].$1 is a textual instruction, and $[\mathcal{D}_1,\ldots,\mathcal{D}_N].$2 is the target response (Cao et al., 2024). In both formulations, the model is expected to improve on the current task without substantially degrading prior-task performance.

The online formulation introduced by OASIS is stricter. There, multi-modal data arrive “in an online streaming manner,” future data are unknown, and non-stationary batch distributions make fixed per-batch selection policies problematic (Lee et al., 27 May 2025). In that setting, training on all incoming data can create a serious training delay: the model spends too much time processing earlier batches and adapts too slowly to later tasks. This shifts the emphasis from continual learning under task boundaries to continual learning under stream responsiveness constraints.

A further extension appears in safety-aligned CVIT. "Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMs" distinguishes pre-SA CVIT, where the model is not yet safety-aligned, from post-SA CVIT, where a previously aligned MLLM must preserve both downstream task performance and safety behavior during continual adaptation (Wang et al., 25 Nov 2025). This adds a second objective beyond ordinary retention.

3. Principal failure modes

The canonical difficulty in CVIT is catastrophic forgetting: updating shared backbone parameters for a new task degrades performance on earlier tasks (Che et al., 8 Aug 2025). In multimodal systems, the same backbone is reused across heterogeneous tasks such as VQA, classification, captioning, and reasoning, and task-specific gradients can conflict. The problem is accentuated by differences in instruction formats and by the need to maintain visual-language alignment over time (Che et al., 8 Aug 2025).

Later work has shown that task accuracy alone is not a sufficient description of failure. "SMoLoRA" identifies dual catastrophic forgetting, consisting of visual understanding forgetting and instruction-following forgetting (Wang et al., 2024). The first appears as declining accuracy on earlier visual tasks after later tasks are learned. The second appears when the model no longer answers in the required format or style, even if some task knowledge remains. This is why SMoLoRA introduces Mean Instruction Following (MIF) as an explicit evaluation metric (Wang et al., 2024).

"LLaVA-c" isolates another phenomenon, base model degradation (Liu et al., 10 Jun 2025). In that account, continual fine-tuning on highly task-specific datasets can overfit the model to narrow instruction styles and degrade general multimodal instruction-following ability. The paper’s qualitative examples describe models that begin answering every prompt in a task-biased style such as single-word responses or captions, even when the prompt requires open-ended reasoning. This diagnosis differs from ordinary forgetting because the loss of general capability can occur even when some task-specific scores remain competitive.

"OASIS" highlights an additional failure mode specific to online CVIT: training delay under large-scale streaming data (Lee et al., 27 May 2025). In this view, the principal issue is not only forgetting but also adaptation latency. Fixed top-$[\mathcal{D}_1,\ldots,\mathcal{D}_N].$3 selection policies are criticized because informativeness varies across batches, so a rigid quota can waste budget on low-value batches and undersample high-value ones. OASIS also emphasizes intra-batch redundancy: near-duplicate examples can consume training budget without improving diversity (Lee et al., 27 May 2025).

In safety-aligned settings, post-SA CVIT introduces safety degradation as a coequal concern. HPA reports that continual fine-tuning can progressively weaken safety-aligned behavior, increasing the likelihood that the model answers harmful multimodal prompts instead of refusing or responding safely (Wang et al., 25 Nov 2025). This reframes CVIT as a joint preservation problem over task capability and alignment.

4. Methodological families

CVIT methods span replay, regularization, architecture expansion, routing/expert selection, incremental embedding adaptation, consolidation, and online sample selection. Early evidence showed that replay and model expansion are robust strategies, while regularization-based methods perform well mainly when the starting model has already undergone multi-task joint instruction tuning (He et al., 2023). Later work concentrated on parameter-efficient variants adapted to MLLM architectures.

Work	Core mechanism	Salient property
Continual Instruction Tuning for Large Multimodal Models	Replay, EProj, regularization, task similarity	First benchmark; similarity-informed CL (He et al., 2023)
Continual LLaVA	Intrinsic and contextual increment embeddings	Rehearsal-free PEFT on COAST (Cao et al., 2024)
SMoLoRA	Separable VU/IF routing over LoRA experts	Targets dual forgetting (Wang et al., 2024)
LLaVA-c	SAC + UIR	Balances task retention and general capability (Liu et al., 10 Jun 2025)
OASIS	ORIS + SIREN	Adaptive online sample selection (Lee et al., 27 May 2025)
LiLoRA	Shared $[\mathcal{D}_1,\ldots,\mathcal{D}_N].$4, decomposed $[\mathcal{D}_1,\ldots,\mathcal{D}_N].$5, stability loss	Parameter-efficient architecture expansion (Che et al., 8 Aug 2025)
HPA	Parameter partition, balanced selection, orthogonal adjustment	Post-SA CVIT for safety-aligned MLLMs (Wang et al., 25 Nov 2025)

"Continual LLaVA" freezes the LVLM backbone and learns only a small set of low-rank increment embeddings (Cao et al., 2024). Its intrinsic increment embeddings are selected from a pool of learnable proxy-increment pairs using instruction similarity computed by a frozen Sentence-BERT encoder, while contextual increment embeddings aggregate prior task summaries through a learnable weighted sum. The adapted projection is written as

$[\mathcal{D}_1,\ldots,\mathcal{D}_N].$6

Only one output projection in the attention block is reparameterized, and the reported tunable parameter count is about 0.75M versus 6.76B for full-model sequential finetuning (Cao et al., 2024).

"SMoLoRA" adopts a separable mixture-of-LoRA design to isolate visual and instruction competencies (Wang et al., 2024). It decomposes adaptation into a visual understanding (VU) module and an instruction following (IF) module: $[\mathcal{D}_1,\ldots,\mathcal{D}_N].$7 The layer output fuses the two branches adaptively: $[\mathcal{D}_1,\ldots,\mathcal{D}_N].$8 The VU router uses the instance representation, whereas the IF router uses the instruction embedding directly (Wang et al., 2024).

"LiLoRA" is an architecture-expansion method grounded in LoRA factorization (Che et al., 8 Aug 2025). Standard LoRA is

$[\mathcal{D}_1,\ldots,\mathcal{D}_N].$9

with low-rank matrices $i$0 and $i$1. LiLoRA shares the LoRA matrix $i$2 across tasks and further decomposes $i$3 into a shared basis plus a task-specific low-rank residual: $i$4 A cosine-regularized basis stability loss,

$i$5

is used to preserve the shared basis when adjacent task residuals are dissimilar (Che et al., 8 Aug 2025).

"LLaVA-c" modifies LLaVA-1.5 with Spectral-Aware Consolidation (SAC) and Unsupervised Inquiry Regularization (UIR) (Liu et al., 10 Jun 2025). SAC treats the update $i$6 spectrally via SVD and replaces uniform interpolation with a sliding-window average over the singular spectrum, while UIR regularizes hidden representations on unlabeled inquiry text: $i$7 The method is explicitly designed to prevent base-model degradation while maintaining new-task plasticity (Liu et al., 10 Jun 2025).

"OASIS" addresses a different axis of the problem: which samples to process under online data streams (Lee et al., 27 May 2025). Its Online Relative Informativeness Selection (ORIS) scores each sample using a Fisher Information–based measure on last-layer gradients,

$i$8

then normalizes scores with an exponential moving average and variance to compute relative informativeness across batches. Its Similarity-aware Information Redundancy Elimination (SIREN) iteratively subtracts similarity-weighted contributions from already selected samples: $i$9 The implementation requires only a single forward pass plus last-layer gradients (Lee et al., 27 May 2025).

For post-safety-alignment CVIT, HPA partitions parameters into safety-focused and task-focused groups using Hessian-inspired sensitivity approximations, selects layerwise masks to retain balanced subsets from old and new models, and applies orthogonal parameter adjustment by subtracting the projection of the update onto the previous weight space (Wang et al., 25 Nov 2025). This design is explicitly post-training: it modifies weights after ordinary task fine-tuning rather than changing the original training pipeline.

A related but non-visual design pattern is provided by SwitchCIT, which uses a lightweight switch network—OPT-125M plus a 2-layer MLP classifier of 154K parameters—to route an instruction to the corresponding task-specific LoRA expert (Wu et al., 2024). Because its experiments are text-only, it does not establish CVIT results directly, but it demonstrates a routing-based PEFT template relevant to multimodal continual instruction tuning.

5. Benchmarks and evaluation protocols

CVIT benchmarking has diversified rapidly. The earliest benchmark in (He et al., 2023) defined two settings. Benchmark 1 starts from BLIP2 and uses the order

$\mathcal{D}_i$0

Benchmark 2 starts from InstructBLIP and uses

$\mathcal{D}_i$1

The distinction was designed to test whether prior multi-task instruction tuning facilitates continual learning (He et al., 2023).

COAST, introduced with Continual LLaVA, defines three stream types (Cao et al., 2024). COAST-domain uses ChartQA, DocVQA, IconQA, and MedicalQA, each with 20,000 training samples and 5,000 evaluation samples. COAST-capability uses conversation, detail description, complex reasoning, and referring question answering, again with 20,000/5,000 train/eval splits. COAST-dataset chains VQAv2, VizWiz, ScienceQA, TextVQA, GQA, OCR-VQA, ImageNet, RefCOCO, RefCOCO+, and RefCOCOg (Cao et al., 2024).

The CVIT Benchmark used by SMoLoRA and LiLoRA contains six instruction datasets—ScienceQA, TextVQA, Flickr30k, ImageNet, GQA, and VQAv2—and is evaluated under single-type instruction and five-type instruction protocols (Wang et al., 2024, Che et al., 8 Aug 2025). SMoLoRA further extends evaluation with downstream transfer to VizWiz, TextCaps, OCRVQA, and Places365, with both zero-shot and few-shot settings (Wang et al., 2024).

OASIS evaluates on three major CVIT benchmarks: MICVIT, COAST, and Adapt (Lee et al., 27 May 2025). MICVIT includes seven tasks: Bongard-OpenWorld, NLVR2, Co-Instruct-DB, Bongard-HOI, PatternCom, DVQA, and HQ-Edit. COAST is instantiated there with ChartQA, DocVQA, IconQA, and MedicalQA. Adapt contains M3IT, MANTIS, LaMM, and VisionFLAN (Lee et al., 27 May 2025).

MCITlib introduces two carefully selected benchmarks intended to reduce leakage from pretraining or prior instruction tuning (Guo et al., 10 Aug 2025). UCIT consists of ImageNet-R, ArxivQA, VizWiz-Caption, IconQA, CLEVR-Math, and Flickr30k, while MLLM-DCL covers remote sensing, medical, autonomous driving, science, and finance domains through datasets such as RSVQA, PathVQA, DriveLM, FinVis, and AI2D/Sciverse/MapQA/TQA (Guo et al., 10 Aug 2025). The paper explicitly identifies benchmark leakage and inconsistent comparison protocols as major evaluation problems.

Metric design reflects the multidimensional nature of CVIT. OASIS reports $\mathcal{D}_i$2 and $\mathcal{D}_i$3 (Lee et al., 27 May 2025). SMoLoRA and LiLoRA use AP, MAP, BWT, and MIF (Wang et al., 2024, Che et al., 8 Aug 2025). Continual LLaVA uses Average Accuracy (AA) and Average Forgetting (AF), with GPT-4o–assisted evaluation for free-form outputs (Cao et al., 2024). MCITlib follows MFT, MFN, MAA, and BWT, and argues that MFN and MAA are often more informative than BWT alone because some methods reduce forgetting by learning less (Guo et al., 10 Aug 2025). Safety-aligned CVIT adds ASR, MASR, and DASR as safety-retention metrics (Wang et al., 25 Nov 2025).

6. Representative empirical findings

The foundational empirical result is that catastrophic forgetting persists in continual multimodal instruction tuning, but its severity depends strongly on initialization and method class (He et al., 2023). In the earliest benchmark study, sequential fine-tuning caused severe forgetting; replay and model expansion were consistently effective; and regularization-based methods performed well mainly when the model had already been jointly instruction-tuned on multiple tasks. The same study also found that similar task pairs forget each other less and transfer better,