MLLM‑CTBench: Multimodal Continual Benchmark

• MLLM‑CTBench is a benchmark for continual instruction tuning in multimodal LLMs, evaluating both final answer accuracy and detailed chain-of-thought reasoning.
• It organizes 16 public datasets into 7 tasks across 6 domains, allowing systematic analysis of sequential adaptation and catastrophic forgetting.
• Empirical findings reveal that stronger base models exhibit greater robustness, with reasoning chains degrading more slowly than final answer outputs.

MLLM‑CTBench is a benchmark for continual instruction tuning of multimodal LLMs (MLLMs), introduced to study sequential adaptation under catastrophic forgetting while retaining fine-grained visibility into reasoning quality rather than only final-answer accuracy (Guo et al., 31 Jul 2025). It organizes 16 public datasets into 7 tasks spanning 6 domains, evaluates 8 continual learning algorithms across 4 methodological categories, and couples macro-level continual learning metrics with micro-level Chain-of-Thought (CoT) analysis via a dedicated multimodal CoT evaluator. Its central empirical claims are that stronger base MLLMs exhibit greater robustness to forgetting, reasoning chains degrade more slowly than final answers, algorithm efficacy is strongly conditioned on both model capability and task order, and KL-regularized reinforcement learning materially improves stability in continual settings (Guo et al., 31 Jul 2025).

1. Problem setting and rationale

MLLM‑CTBench is situated in the setting of continual instruction tuning (CIT), where tasks arrive sequentially as $\mathcal{T}_1,\dots,\mathcal{T}_S$, the model trains on task $\mathcal{T}_i$ using only data from that task, and it must simultaneously acquire competence on the current task and retain performance on prior tasks (Guo et al., 31 Jul 2025). In MLLMs, this problem is sharpened by multimodal inputs, domain-specific reasoning, and the stability–plasticity trade-off induced by repeated instruction tuning.

The benchmark was designed in response to three limitations identified in prior MLLM continual learning benchmarks: superficial evaluation centered on final-answer accuracy, limited comparison across algorithm classes and training paradigms, and task suites that are too easy for contemporary MLLMs (Guo et al., 31 Jul 2025). The benchmark therefore emphasizes unsaturated tasks and explicit CoT analysis. This design is consistent with the broader benchmark-design view that high diagnostic value requires challenging, low-redundancy, diverse evaluation data rather than merely large test sets (Li et al., 13 Mar 2025).

A defining feature of MLLM‑CTBench is its treatment of CoT not as incidental generated text but as an evaluable object. In the benchmark’s framing, final answers alone cannot distinguish retained factual knowledge from retained reasoning procedure. This motivates parallel tracking of answer-level degradation and CoT-level degradation, leading to the reported hierarchical forgetting pattern in which reasoning traces deteriorate less rapidly than answer correctness (Guo et al., 31 Jul 2025).

2. Task construction and continual scenario

MLLM‑CTBench defines 7 tasks from 16 datasets across 6 domains. The task suite mixes text-only and multimodal settings to expose cross-domain and cross-modality interference during continual tuning (Guo et al., 31 Jul 2025).

Task	Source datasets	Domain
Math QA	TRACE	Math
Economics QA	TRACE	Economics
Science VQA	AI2D, ScienceQA	Science
Math VQA	IconQA, GeoQA, CHARTX, MMMU	Math
Medicine VQA	VQA‑RAD, VQA‑Med‑2021, PMC‑VQA, PathVQA	Medicine
OCR VQA	ChartOCR, CROHME, ESTVQA	OCR
Arts VQA	AQUA	Arts

The benchmark specifies the following train/test allocations: Math QA uses 10K / 0 train and 0.5K / 0 test samples; Economics QA 5K / 0 and 0.5K / 0; Science VQA 9K / 4K and 1K / 0.5K; Math VQA 8.3K / 8.3K and 0.9K / 0.9K; Medicine VQA 9K / 6.9K and 1K / 1K; OCR VQA 12K / 12.1K and 1.4K / 1.4K; and Arts VQA 9K / 7K and 1K / 0.9K (Guo et al., 31 Jul 2025).

To expose order sensitivity, the benchmark uses two permutations. Order‑A is: Math QA $\rightarrow$ Arts VQA $\rightarrow$ Math VQA $\rightarrow$ Economics QA $\rightarrow$ Medicine VQA $\rightarrow$ OCR VQA $\rightarrow$ Science VQA. Order‑B is the reverse sequence (Guo et al., 31 Jul 2025). The model is evaluated on all 7 tasks after each training stage, yielding post-task performance matrices $P_{i,j}$ for final answers and $C_{i,j}$ for CoT scores.

The benchmark starts from pretrained MLLMs, specifically LLaVA‑1.5‑7B and Qwen2.5‑VL‑3B, and studies zero-shot behavior, direct fine-tuning, multi-task joint training, sequential supervised fine-tuning under continual learning algorithms, and reinforcement learning with GRPO on the Qwen backbone (Guo et al., 31 Jul 2025).

3. Evaluation architecture: answer accuracy and CoT analysis

MLLM‑CTBench employs a two-tier evaluation stack. At the macro level, it evaluates final answers using task-appropriate extraction and scoring; at the micro level, it scores the generated reasoning chain along three dimensions: logical coherence, visual grounding fidelity, and domain knowledge retention (Guo et al., 31 Jul 2025).

The macro continual metrics are Average Performance (AP) after the final task and Backward Transfer (BWT):

$\mathcal{T}_i$0

$\mathcal{T}_i$1

Here, $\mathcal{T}_i$2 denotes accuracy on task $\mathcal{T}_i$3 after training task $\mathcal{T}_i$4, and $\mathcal{T}_i$5 (Guo et al., 31 Jul 2025). Negative BWT indicates catastrophic forgetting.

At the CoT level, the benchmark first considers a general-purpose MLLM-as-judge based on Qwen‑VL‑2.5‑32B, then introduces a dedicated evaluator to improve reliability (Guo et al., 31 Jul 2025). The specialized evaluator uses Qwen2.5‑VL‑7B as its base model and is trained in two stages: supervised fine-tuning to imitate GPT‑4 absolute CoT scores, followed by GRPO-based RL using GPT‑4 pairwise preferences as reward signals. The paper reports that this evaluator achieves consistently higher Spearman’s $\mathcal{T}_i$6, Pearson correlation $\mathcal{T}_i$7, and Kendall’s $\mathcal{T}_i$8 than the general evaluator across Qwen SFT, Qwen RL, and LLaVA SFT traces (Guo et al., 31 Jul 2025).

CoT scores are tracked analogously to answer accuracy. Let $\mathcal{T}_i$9 denote the CoT score for task $\rightarrow$0 after training task $\rightarrow$1; CoT AP and CoT BWT are then computed by substituting $\rightarrow$2 for $\rightarrow$3 in the continual metrics (Guo et al., 31 Jul 2025). This yields the benchmark’s central reasoning-level diagnostic: answer forgetting and reasoning forgetting need not coincide.

4. Training paradigms and continual learning algorithms

MLLM‑CTBench compares two training paradigms. The first is supervised fine-tuning (SFT) using standard next-token prediction on instruction–answer pairs. The second is reinforcement learning with GRPO, which optimizes expected reward over generated responses while imposing KL regularization to stabilize policy updates (Guo et al., 31 Jul 2025).

For task $\rightarrow$4, the SFT objective is written as

$\rightarrow$5

and the GRPO objective as

$\rightarrow$6

The benchmark also evaluates GRPO w/o KL to isolate the role of the KL term in forgetting (Guo et al., 31 Jul 2025).

The 8 continual learning algorithms are grouped into 4 categories (Guo et al., 31 Jul 2025). The regularization-based methods are EWC, MAS, LwF, and freeze, where the freeze setting uses freeze-first-8 or freeze-last-8 transformer layers together with a frozen vision encoder. The replay-based methods are ER and DER. The architectural expansion method is L2P, which learns from a prompt pool. The model-fusion method is MagMaX, which performs post-hoc parameter merging.

The benchmark evaluates both full-parameter tuning and LoRA-based adaptation. For RL, the vision encoder is frozen, LoRA is applied on the LLM, and GRPO uses KL regularization in the main setting (Guo et al., 31 Jul 2025).

5. Empirical findings

The benchmark’s first empirical conclusion is that stronger base models forget less. Under sequential SFT with Order‑A, LLaVA‑1.5 reaches AP = 35.60 with BWT $\rightarrow$7, whereas Qwen2.5‑VL reaches AP = 57.71 with BWT $\rightarrow$8 (Guo et al., 31 Jul 2025). The benchmark interprets this as evidence that stronger general capability confers greater robustness to continual interference.

The second conclusion is the benchmark’s most distinctive one: reasoning chains degrade more slowly than final answers. For LLaVA‑1.5, Order‑A SFT yields answer-level BWT = -15.37 but CoT-level BWT = -6.54. For Qwen2.5‑VL, the corresponding values are -6.43 and -3.74 (Guo et al., 31 Jul 2025). This supports the hierarchical forgetting hypothesis, according to which procedural reasoning habits are more stable than final-step factual recall or output execution.

The third conclusion is that algorithm effectiveness depends on both model capability and task order. Replay methods are particularly strong on weaker backbones: on LLaVA‑1.5, ER reduces forgetting by about 88% relative to naive SFT, with BWT around -1.8 versus -15.3; on Qwen2.5‑VL, the relative improvement is smaller, with BWT -3.23 versus -6.43 (Guo et al., 31 Jul 2025). Regularization methods are reported as more effective on Qwen than on LLaVA, while aggressive layer freezing can be counterproductive: for Qwen, freezing the first or last 8 layers often increases forgetting by about 20% relative to standard SFT (Guo et al., 31 Jul 2025). L2P and MagMaX provide stable forgetting reduction across scales, but with different operational trade-offs: prompt-pool growth and retrieval overhead for L2P, and low-overhead post-hoc merging for MagMaX (Guo et al., 31 Jul 2025).

Task order matters at the per-task level, but aggregate order effects are smaller. The benchmark reports that some tasks can suffer markedly different forgetting depending on sequence position, yet overall AP and BWT across Order‑A and Order‑B differ by only about 1% on average (Guo et al., 31 Jul 2025). This suggests that order sensitivity is real but not uniformly catastrophic at the benchmark level.

The fourth conclusion concerns RL. In the Qwen2.5‑VL Order‑A comparison, SFT yields AP $\rightarrow$9 and BWT $\rightarrow$0; GRPO with KL lowers overall AP to $\rightarrow$1 but improves BWT to $\rightarrow$2; GRPO w/o KL drops to AP $\rightarrow$3 and BWT $\rightarrow$4 (Guo et al., 31 Jul 2025). The benchmark’s interpretation is explicit: the KL term acts as an implicit memory that constrains policy drift, and without it RL can forget more severely than plain SFT.

6. Position in the benchmark landscape, limitations, and extensions

MLLM‑CTBench occupies a specific niche within multimodal evaluation: it is not a general capability benchmark, a domain benchmark, or a benchmark-efficiency framework, but a continual adaptation benchmark with CoT-aware diagnostics (Guo et al., 31 Jul 2025). This differentiates it from domain-specific multimodal benchmarks such as MMFCTUB, which targets credit table understanding with cross-table constraints and mask-and-recovery metrics (Yakun et al., 8 Jan 2026), and M3CoTBench, which evaluates CoT reasoning in medical image understanding using correctness, efficiency, impact, and consistency metrics (Jiang et al., 13 Jan 2026). A plausible implication is that MLLM‑CTBench provides the continual-learning substrate that such specialized benchmarks do not attempt to supply.

The benchmark also interacts conceptually with broader benchmark methodology. The Information Density principle argues that difficult, low-redundancy, diverse benchmarks provide more useful signal to model developers than superficially large but weakly diagnostic test suites (Li et al., 13 Mar 2025). MLLM‑CTBench’s reliance on unsaturated tasks and explicit reasoning analysis is congruent with that principle. Separately, the MITV interview strategy shows that on several multimodal benchmarks, reliable ranking can often be recovered from a small adaptive subset of questions, with strong rank alignment sometimes reached using 2–5% of questions (Wen et al., 1 Jun 2025). This suggests that future continual-learning evaluation protocols could combine MLLM‑CTBench’s CoT-aware diagnostics with more sample-efficient benchmark administration.

The benchmark’s stated limitations are substantial (Guo et al., 31 Jul 2025). Experiments are confined to 3B–7B models and to two MLLM families, so scaling conclusions remain provisional. Only two task orders are evaluated rather than the full combinatorial space of permutations. CoT evaluation relies on an automatic MLLM-as-judge, even if fine-tuned and correlation-validated, so human judgment is not fully replaced. The six domains are broad but not exhaustive; the benchmark does not include modalities such as audio or 3D, nor domains such as code/data tables or robotics. These limitations motivate the paper’s future directions: scaling-law studies for forgetting, CoT-aware continual learning algorithms, broader continual RL methods, task-order optimization, and improved multimodal judges (Guo et al., 31 Jul 2025).

In that sense, MLLM‑CTBench is best understood as both an evaluation resource and an empirical argument. Its design asserts that continual learning in MLLMs cannot be adequately characterized by endpoint accuracy alone, and its results support the stronger claim that reasoning traces, answer outputs, optimization regime, and task sequence should be analyzed as partially separable variables in multimodal continual adaptation (Guo et al., 31 Jul 2025).