MCITlib: Multimodal Continual Tuning

• MCITlib is an open-source Python library providing a unified framework for continual instruction tuning of multimodal large language models with efficient PEFT strategies.
• It implements eight state-of-the-art MCIT algorithms that leverage techniques like LoRA, modality-specific prompts, and gradient projection to mitigate catastrophic forgetting.
• The library offers modular training pipelines, leakage-free benchmarks, and standardized metrics to support reproducible multimodal continual learning research.

MCITlib is an open-source Python library providing a unified framework for continual instruction tuning of Multimodal LLMs (MLLMs). MCITlib implements eight representative multimodal continual instruction-tuning (MCIT) algorithms, offering modular training and evaluation pipelines and two systematically designed, leakage-free benchmarks. Built atop HuggingFace Transformers and PEFT, it enables reproducible evaluation of continual learning (CL) methods in multimodal settings, currently focusing on vision-and-language scenarios. MCITlib emphasizes parameter-efficient fine-tuning (PEFT), robust evaluation against catastrophic forgetting, and extensibility for future MCIT research (Guo et al., 10 Aug 2025).

1. Architecture and Workflow

MCITlib is organized into four principal modules:

1. Data: The benchmarks/ucit and benchmarks/mllm_dcl submodules supply PyTorch Dataset/DataLoader abstractions for all tasks, with a YAML-style specification for task sequences.
2. Models: Wrappers are provided for widely used MLLMs (currently LLaVA-1.5-7B), exposing a standardized forward interface: $\text{forward}(\text{text}, \text{image}) \to$ logits.
3. Methods: Eight MCIT algorithms are implemented as classes (LoRAFT, OLoRA, [MoELoRA](https://www.emergentmind.com/topics/moelora), ModalPrompt, CLMoE, HiDe, SEFE, [DISCO](https://www.emergentmind.com/topics/distributed-scion-disco)), each inheriting from a base MCITMethod. The base class defines hooks:
   • initialize_peft() to attach adapters or prompts,
   • before_task()/after_task() to capture parameters or update regularization state,
   • loss(outputs, targets) for combining cross-entropy and any CL regularization or replay.
4. Core: The training and evaluation engine comprises Trainer and Evaluator classes. They manage the rehearsal-free incremental training loop and compute per-task or aggregate CL metrics (MFT, MFN, MAA, BWT).

A standard MCITlib workflow consists of: instantiating a benchmark which returns an ordered dataset sequence, selecting a method with configured PEFT hyperparameters, initializing a trainer and evaluator with the chosen method and benchmark, then executing training and metric computation. Logging occurs after each task stage, producing cumulative and task-wise scores.

2. MCIT Algorithms

Eight algorithms are unified under MCITlib, supporting various parameter-efficient and regularization strategies. All methods share a base cross-entropy loss: $$\mathcal{L}_{\rm CE} = -\sum_{i=1}^N y_i \log p_{\theta}(y_i \mid x_i, I_i).$$

The following summarizes each technique and its principal mathematical formulation:

Method	Core Principle	Distinctive Features / Loss Terms
LoRA-FT	Low-rank PEFT adapters	$\mathcal{L} = \mathcal{L}_{\rm CE}$
OLoRA	Orthogonal LoRA directions	$$\mathcal{L}_{\rm ortho} = \lambda_{\rm ortho} \|A_t^\top A_{<t}\|_F^2$$
MoELoRA	Mixture-of-LoRA-Experts	Weighted sum by router, balance loss $\alpha\,\mathcal{L}_{\rm balance}$
ModalPrompt	Modality-specific prompts	Prepend $P_{\text{img}}$, $P_{\text{txt}}$ at each layer
CL-MoE	MoE with momentum updates	Dual-momentum expert weights, loss as in MoELoRA
HiDe-LLaVA	Hierarchical decoupling	Gradient projection for shared/task-specific params
SEFE	Fisher-based reg. (EWC style)	$\mathcal{L}_{\rm EWC}$ penalizes deviation in "essential" params
DISCO	Adapter per task, lookup	Fresh adapter per task, zero loss for CL, cosine-similarity inference

• LoRA-FT (Hu et al., ICLR '22): Attaches low-rank adapters $\Delta W = AB$ to projections; no explicit CL penalty.
• OLoRA (Wang et al., arXiv '23): Adds orthogonalization regularization to new LoRA bases to avoid interference.
• MoELoRA (Chen et al., NeurIPS '24): Maintains $K$ LoRA experts. A router $$\mathcal{L}_{\rm CE} = -\sum_{i=1}^N y_i \log p_{\theta}(y_i \mid x_i, I_i).$$0 computes a dynamic convex combination for each input, with $$\mathcal{L}_{\rm CE} = -\sum_{i=1}^N y_i \log p_{\theta}(y_i \mid x_i, I_i).$$1 encouraging expert usage diversity.
• ModalPrompt (Zeng et al., arXiv '24): Learns modality-specific prefix tokens, prepended at each transformer layer.
• CL-MoE (Huai et al., CVPR '25): Extends MoE via momentum updates for expert parameters to stabilize routing.
• HiDe-LLaVA (Guo et al., arXiv '25): Projects shared parameter gradients orthogonal to past-task directions, isolating task-specialization.
• SEFE (Chen et al., arXiv '25): Applies diagonal Fisher-matrix EWC for selectivity in preserving "essential" parameters from previous tasks.
• DISCO (Guo et al., arXiv '25): Adds a new LoRA adapter per task, storing them in an embedding-keyed bank; at inference, the nearest adapter is used, fully isolating task parameters.

3. Benchmarks and Metrics

MCITlib provides two extensively curated benchmarks, specifically designed to minimize pre-training leakage and support robust, multimodal continual evaluation.

UCIT: This benchmark comprises six sequential tasks—ImageNet-R (robust classification), ArxivQA (text+figure QA), VizWiz-Caption (assistive captioning), IconQA (diagrammatic multiple choice), CLEVR-Math (compositional reasoning + arithmetic), and Flickr30k (captioning). Metrics include accuracy (QA), BLEU, and CIDEr (captioning), normalized to a percentage.

MLLM-DCL: Spanning five domains over eight datasets, tasks include remote sensing (RSVQA), medical (PathVQA), autonomous driving (DriveLM), science (AI2D, Sciverse), and finance (FinVis, MapQA, TQA), all formulated as QA or chart interpretation. Top-1 accuracy is the central metric.

MCITlib enforces a standardized CL metrics suite:

• Mean Finetune (MFT): $$\mathcal{L}_{\rm CE} = -\sum_{i=1}^N y_i \log p_{\theta}(y_i \mid x_i, I_i).$$2
• Mean Final (MFN): $$\mathcal{L}_{\rm CE} = -\sum_{i=1}^N y_i \log p_{\theta}(y_i \mid x_i, I_i).$$3
• Mean Average (MAA): $$\mathcal{L}_{\rm CE} = -\sum_{i=1}^N y_i \log p_{\theta}(y_i \mid x_i, I_i).$$4
• Backward Transfer (BWT): $$\mathcal{L}_{\rm CE} = -\sum_{i=1}^N y_i \log p_{\theta}(y_i \mid x_i, I_i).$$5

4. Empirical Results and Observations

Comprehensive experiments under MCITlib's uniform hyperparameter regime reveal distinctive trade-offs among methods:

• LoRA-FT achieves significant MFN gains over zero-shot baselines (+29.6% UCIT, +18.2% MLLM-DCL), indicating some MLLM robustness to forgetting.
• Regularization (OLoRA, SEFE) and prompt-based (ModalPrompt) methods reduce BWT, often with modestly reduced MFT, encapsulating the "learn less, forget less" phenomenon.
• DISCO attains the best MFN and MAA on both benchmarks through strict adapter isolation but induces linearly increasing storage requirements.
• SEFE offers competitive BWT and MAA performance with negligible parameter overhead, representing a favorable trade-off for many applications.
• Taskwise forgetting analysis notes that UCIT's CLEVR-Math stage experiences the most forgetting, while robustness in MLLM-DCL is highest for remote sensing and least for medical QA.

This suggests that parameter decoupling mechanisms, as in DISCO, may be optimal where storage is unbounded, while selective consolidation (SEFE) excels in resource-constrained contexts.

5. Extensibility and Ongoing Evolution

MCITlib is designed for extensibility along three main axes:

• Benchmarks: New leakage-controlled datasets, including audio+vision and underexplored long-tail domains, can be integrated to capture broader multimodal CL challenges.
• Methods: The MCITMethod API allows rapid prototyping and plug-in of additional CL techniques, including memory replay, adapter fusion, and meta-continual algorithms.
• Base Models: Additional MLLMs such as InternVL and Vision-LLAMA are supported via simple registration of new model wrappers.

Community contributions are facilitated through GitHub pull requests, enabling rapid response to field advances.

6. Code Usage, Design Practices, and Best Practices

A prototypical MCITlib usage pipeline involves:

• Installation via pip;
• Pre-trained MLLM checkpoint loading;
• Method and benchmark instantiation;
• Trainer execution for continual instruction tuning;
• Evaluator computation of all four CL metrics.

$$\mathcal{L}_{\rm CE} = -\sum_{i=1}^N y_i \log p_{\theta}(y_i \mid x_i, I_i).$$6

$$\mathcal{L}_{\rm CE} = -\sum_{i=1}^N y_i \log p_{\theta}(y_i \mid x_i, I_i).$$7

Design priorities include: a rehearsal-free setting (task replay is disallowed); strict PEFT baselining; hyperparameter uniformity; leakage-minimized benchmarks (to avoid pre-training confounds); and a unified API for extensible research workflows. These conventions provide robust, fair comparison and reproducibility across MCIT approaches.

7. Significance and Research Context

MCITlib represents a reproducible and extensible platform for researchers investigating continual instruction tuning in MLLMs. By systematically implementing representative methods and leakage-controlled benchmarks, MCITlib enables fine-grained comparisons of catastrophic forgetting, backward transfer, and efficient adaptation in multimodal, parameter-efficient contexts. Its modular architecture and evolving dataset/method/model support provide infrastructure crucial for advancing state-of-the-art in multimodal continual learning (Guo et al., 10 Aug 2025).