Multimodal Continual Instruction Tuning

• Multimodal Continual Instruction Tuning (MCIT) is a sequential approach that aligns vision–language models with human intent across tasks while avoiding catastrophic forgetting.
• It leverages unimodal continual learning techniques like modular parameter isolation, replay, and gradient projection to maintain and update prior expertise.
• MCIT employs dynamic routing and format-aware protocols to optimize scalability and adaptation, achieving significant accuracy gains on complex benchmarks.

Multimodal Continual Instruction Tuning (MCIT) defines a class of algorithms and systems designed to incrementally align Multimodal LLMs (MLLMs) with human intent across a sequence of vision-and-language tasks, without catastrophic forgetting, while accommodating cross-modal instruction-following protocols. Unlike standard instruction tuning, which typically assumes joint access to all tasks and supports only static multi-task optimization, MCIT targets sequential task streams where prior data is not revisited, placing stringent demands on stability, scalability, and flexible adaptation. The field is motivated by practical needs: real-world MLLM deployments must learn new tasks or domains as they arise, without erasing prior expertise or incurring prohibitive retraining costs. MCIT leverages, adapts, and extends numerous strategies from unimodal continual learning—modular parameter isolation, replay, spectral consolidation, architecture growth, prompt routing, gradient projection—while tackling unique challenges arising from cross-modal interaction, output format diversity, and reasoning trace retention.

1. Problem Formulation and Benchmarks

MCIT is defined as the rehearsal-free, sequential tuning of a pretrained MLLM on an incoming stream of instruction-following tasks. Each task provides multimodal samples, typically tuples (image, instruction text, target answer sequence). The key desiderata are:

• Retention: The model must maintain high performance (instruction-following accuracy, general knowledge, or reasoning trace quality) on previously learned tasks.
• Adaptation: The system should exhibit strong plasticity to novel tasks and instructions, including those with novel output protocols or stylistic requirements.
• Parameter/Budget Efficiency: The total additional parameters and compute overhead should remain controllable as the number of tasks grows.
• Absence of Catastrophic Forgetting: Catastrophic forgetting, observable as a drop in test performance on old tasks after training on new ones, is the dominant failure mode in MCIT, often much more pronounced than in unimodal continual learning.

Several large-scale MCIT benchmarks have been proposed to enable rigorous evaluation, including CoIN and UCIT (eight and six vision–language tasks, respectively) and MLLM-CTBench (sixteen datasets across six domains with answer and CoT reasoning analysis) (Chen et al., 2024, Guo et al., 31 Jul 2025, Guo et al., 17 Mar 2025). The standard evaluation metrics in the field include:

• Average Accuracy (ACC) / Mean Final Accuracy (MFN): Mean performance on all tasks after the final stage.
• Backward Transfer (BWT): $$\mathrm{BWT} = \frac1{T-1}\sum_{i=1}^{T-1} (A_{T,i} - A_{i,i})$$—negative values measure aggregate forgetting.
• Mean Average Accuracy (MAA): Time-smoothed average over all training steps.
• Chain-of-Thought Reasoning Score: For models expected to generate reasoning traces.

MCIT research demonstrates that, even for well-calibrated MLLMs, naive sequential application of LoRA or other PEFT modules can cause BWT as low as $-32.6\%$ and mean accuracy drops exceeding 25 points (Chen et al., 2024).

2. Causes of Catastrophic Forgetting in MCIT

Unlike text-only continual learning, MCIT exposes additional forms of interference:

• Format-induced interference: New tasks with distinct answer styles (MCQ, bounding-box, short text, detailed explanation) bias outputs on prior tasks into incorrect formats, masking underlying knowledge retention (Chen et al., 5 May 2025, Shi et al., 1 Jun 2026).
• Routing and parameter drift: Modular architectures (e.g. MoELoRA) aggregate LoRA updates across all experts, causing updates for new tasks to "wash out" earlier expertise. t-SNE analyses reveal rapid collapse of certain LoRA parameter matrices ($A_j$) into shared subspaces, highlighting redundancy and interference (Zhang et al., 31 May 2025).
• Router and expert drift: Mixture-of-Experts and sparse expert routing degrade as both routers and expert parameters drift (router drift leads to old examples being assigned to incorrect experts, expert drift corrupts previously specialized functionality) (Xie et al., 2 Feb 2026).
• Instruction and modality balancing: Tasks vary in their reliance on vision vs language components; insufficient dynamic allocation leads to modality-imbalance, overfitting, and insufficient adaptation (Ge et al., 13 Jun 2025).

3. Model Architectures and Parameterization Strategies

A diverse set of parameter-efficient and modular architectures for MCIT have been proposed, with distinct strengths and trade-offs:

Architecture/Method	Key Idea(s)	Forgetting Mitigation Mechanism
MoELoRA (Chen et al., 2024)	Mixture-of-experts over multiple LoRA adapters per layer, soft router	Parameter isolation; modularity
BranchLoRA (Zhang et al., 31 May 2025)	Asymmetric split: shared trunk $A$, multiple $B_j$ branches; task-specific routers	Sparse aggregation & tuning-freezing
SAME (Xie et al., 2 Feb 2026)	Spectral-aware routing and curvature-aware expert scaling	Subspace preconditioning; expert freezing
D-MoLE (Ge et al., 13 Jun 2025)	Dynamic layerwise expert allocation via gradient-norm proxy	Architectural evolution under budget
HiDe-LLaVA (Guo et al., 17 Mar 2025)	Task-specific expansion only at top transformer layer, task-general merging elsewhere	Layerwise specialization
ProtoAda (Shi et al., 1 Jun 2026)	Format-aware prototypes guide expert routing and adapter consolidation	Format-protocol encoding; geometric reuse
CRAM (Tang et al., 1 Jun 2026)	Centroid-routing and adaptive-rank MoE experts per instruction cluster	Group isolation; rank-efficient expansion

Typically, LoRA-based PEFT remains the foundation, with advances focusing on (i) disentangling shared vs. task-specific adaptation, (ii) flexible sparse expert gating and dynamic expansion, and (iii) format and semantic–aware routing. Recent innovations, e.g. BranchLoRA, yield a shared low-rank trunk $A$ plus per-task $B_j$, leveraging sparse top-$k$ aggregation to sharply reduce cross-task interference and parameter count (Zhang et al., 31 May 2025). Dynamic freezing of previously specialized branches further prevents overwriting old knowledge.

Adaptive mechanisms such as D-MoLE's curriculum-driven expert schedule and adaptive-rank instantiation in CRAM enable the system to re-allocate parameter capacity precisely where each new task is most demanding, yielding both higher accuracy and parameter budget efficiency (Ge et al., 13 Jun 2025, Tang et al., 1 Jun 2026).

4. Overcoming Forgetting: Regularization, Replay, and Gradient Control

MCIT methods also adapt classical continual learning strategies to the multimodal domain:

• Regularization: EWC and related Fisher-matrix or synaptic-importance penalties, sometimes with task-similarity-informed weighting, provide limited gains unless ample initial multi-task tuning has occurred (He et al., 2023, Guo et al., 31 Jul 2025).
• Replay: Experience replay buffers (e.g. 1% per task) sharply reduce forgetting; however, storage and privacy costs may be prohibitive (He et al., 2023, Guo et al., 31 Jul 2025).
• Parameter regularization for essential forgetting: SEFE introduces mask-based regularization penalizing updates to the few (top 2%) largest-magnitude, knowledge-critical adapter parameters from prior tasks (Chen et al., 5 May 2025).
• Gradient-guidance: MCIT-specific approaches, such as Dynamic Gradient Guidance, treat catastrophic forgetting as missing old-task gradients, estimating them geometrically (by parameter difference to previous optima) and fusing with current gradients and replay gradients under a Bernoulli schedule. This yields state-of-the-art final accuracy with minimal expansion (Li et al., 19 Nov 2025).
• Null-space projection: Approaches such as DRAPE project update gradients into the null space of dominant subspaces spanned by prior-task features, preserving both reasoning and perceptual span (Hu et al., 11 May 2026).

Empirically, regularization is most effective on strong, multi-task pretrained models; replay-based and architectural expansion methods (MoE, ProtoAda, BranchLoRA) consistently yield the lowest BWT and highest average accuracy in more challenging sequences.

5. Role of Output Format and Protocol in MCIT

Format drift—where a model begins to output, e.g., a string instead of a bounding box, or chooses the wrong answer style for a task—has been shown to dominate catastrophic forgetting in MCIT regimes. For instance, SEFE explicitly separates "superficial" (format) from "essential" (knowledge) forgetting, showing that only after format stabilization (via Answer Style Diversification) can true knowledge loss be measured (Chen et al., 5 May 2025). ProtoAda further demonstrates that format–blind expert routing (purely semantic similarity) yields severe cross-format interference, necessitating dual-format + semantic prototype construction for reliable task–adapter assignment (Shi et al., 1 Jun 2026).

Explicitly encoding output protocols (length, entropy, template consistency) into per-task prototypes, and using these for expert/router assignment and adapter consolidation, significantly reduces format-induced forgetting and boosts retention on benchmarks with diverse answer styles.

6. Inference Strategies, Automatic Routing, and Scalability

A major practical challenge in MCIT is inference without explicit task ID. Solutions include:

• Prototype-based selector: At test time, input features (modality-specific or fused) are matched to stored per-task or per-group prototypes using cosine similarity, with routers/gates/task-specific adapters auto-selected with accuracy exceeding 95% (Zhang et al., 31 May 2025, Shi et al., 1 Jun 2026, Hu et al., 11 May 2026).
• Automatic router and expert gating: Many approaches leverage top-k aggregation with learned routers, or instance-level gating via cross-modal autoencoders and CLIP-space features, to enable dynamic or instance-specific expert selection (Ge et al., 13 Jun 2025, Hu et al., 11 May 2026).
• Dual-track merging: Recent work such as MAny provides training-free merging of cross-modal projectors and LoRA updates, via prototype-guided weighted fusion (CPM) and closed-form recursive least-squares in parameter space (LPM), offering scalable multi-task consolidation with minimal storage (Gao et al., 15 Apr 2026).

These mechanisms ensure scalability in both parameter count and computational latency, even as the number of adapted tasks grows into the tens or hundreds.

7. Empirical Performance and Practical Guidance

Recent MCIT methods have demonstrated robust empirical gains on benchmarks such as CoIN, UCIT, and TriGap, summarized as follows:

Method	Final Avg Acc (UCIT, LLaVA-7B)	BWT (Forgetting)	Notable Contributions
LoRA-FT	61.24%	–18.78	Baseline PEFT, no forgetting control
MoE-LoRA	60.79%	–19.88	Mixture of LoRA experts
BranchLoRA	44.20% (CoIN)	–20.98 (CoIN)	Asymmetric sharing, sparse agg, freezing
SEFE	66.54%	–11.33	ASD + RegLoRA regularization
ProtoAda	74.66%	–5.61	Format/semantic prototypes, geometric consolidation
SAME	73.07%	–7.14	Spectral router/expert stabilization
CRAM	74.73%	–4.26	Centroid routing, adaptive-rank MoE
DISCO	75.87%	–7.45	Adapter extension+fused inference

Best-in-class results (average accuracy, lowest BWT) are obtained by architectures that (a) hierarchically separate task- and format-specific from shared generalizable components, (b) dynamically grow or consolidate capacity, and (c) employ routing or selection informed by both input semantics and output protocol.

Practical recommendations derived from benchmark studies (Guo et al., 31 Jul 2025, Ge et al., 13 Jun 2025, Xie et al., 2 Feb 2026):

• Base model selection: Strong, wide-coverage pretrained MLLMs are considerably more robust to forgetting and benefit more from regularization or replay.
• Architecture: Parameter-isolation or dynamically expandable architectures (MoE variants, ProtoAda, D-MoLE) yield the best retention with scalable overhead.
• Regularization and replay: Preferred for deployments with strict resource or privacy constraints; less effective on smaller or unadapted base models.
• Format protocol awareness: Mandatory for high-heterogeneity task sequences to prevent format or answer-style collapse.
• Evaluation: Always combine final answer metrics with reasoning trace or knowledge-alignment assessment; probe both superficial and essential forms of forgetting.

Potential future directions include adaptive expert importance weighting, online subspace refinement, generative or class-balanced replay for further robustness, and scaling to orders-of-magnitude more diverse and streaming task populations (Zhang et al., 31 May 2025, Shi et al., 1 Jun 2026, Gao et al., 15 Apr 2026).