---
title: 'MLLM‑CTBench: Multimodal Continual Benchmark'
url: https://www.emergentmind.com/topics/mllm-ctbench
type: topic
---

# MLLM‑CTBench: Multimodal Continual Benchmark

MLLM‑CTBench is a benchmark for continual instruction tuning of multimodal large language models (MLLMs), introduced to study sequential adaptation under catastrophic forgetting while retaining fine-grained visibility into reasoning quality rather than only final-answer accuracy [2508.08275]. It organizes 16 public datasets into 7 tasks spanning 6 domains, evaluates 8 continual learning algorithms across 4 methodological categories, and couples macro-level continual learning metrics with micro-level Chain-of-Thought (CoT) analysis via a dedicated multimodal CoT evaluator. Its central empirical claims are that stronger base MLLMs exhibit greater robustness to forgetting, reasoning chains degrade more slowly than final answers, algorithm efficacy is strongly conditioned on both model capability and task order, and KL-regularized reinforcement learning materially improves stability in continual settings [2508.08275].

## 1. Problem setting and rationale

MLLM‑CTBench is situated in the setting of **continual instruction tuning (CIT)**, where tasks arrive sequentially as $\mathcal{T}_1,\dots,\mathcal{T}_S$, the model trains on task $\mathcal{T}_i$ using only data from that task, and it must simultaneously acquire competence on the current task and retain performance on prior tasks [2508.08275]. In MLLMs, this problem is sharpened by multimodal inputs, domain-specific reasoning, and the stability–plasticity trade-off induced by repeated instruction tuning.

The benchmark was designed in response to three limitations identified in prior MLLM continual learning benchmarks: superficial evaluation centered on final-answer accuracy, limited comparison across algorithm classes and training paradigms, and task suites that are too easy for contemporary MLLMs [2508.08275]. The benchmark therefore emphasizes unsaturated tasks and explicit CoT analysis. This design is consistent with the broader benchmark-design view that high diagnostic value requires challenging, low-redundancy, diverse evaluation data rather than merely large test sets [2503.10079].

A defining feature of MLLM‑CTBench is its treatment of CoT not as incidental generated text but as an evaluable object. In the benchmark’s framing, final answers alone cannot distinguish retained factual knowledge from retained reasoning procedure. This motivates parallel tracking of answer-level degradation and CoT-level degradation, leading to the reported hierarchical forgetting pattern in which reasoning traces deteriorate less rapidly than answer correctness [2508.08275].

## 2. Task construction and continual scenario

MLLM‑CTBench defines **7 tasks** from **16 datasets** across **6 domains**. The task suite mixes text-only and multimodal settings to expose cross-domain and cross-modality interference during continual tuning [2508.08275].

| Task | Source datasets | Domain |
|---|---|---|
| Math QA | TRACE | Math |
| Economics QA | TRACE | Economics |
| Science VQA | AI2D, ScienceQA | Science |
| Math VQA | IconQA, GeoQA, CHARTX, MMMU | Math |
| Medicine VQA | VQA‑RAD, VQA‑Med‑2021, PMC‑VQA, PathVQA | Medicine |
| OCR VQA | ChartOCR, CROHME, ESTVQA | OCR |
| Arts VQA | AQUA | Arts |

The benchmark specifies the following train/test allocations: Math QA uses **10K / 0** train and **0.5K / 0** test samples; Economics QA **5K / 0** and **0.5K / 0**; Science VQA **9K / 4K** and **1K / 0.5K**; Math VQA **8.3K / 8.3K** and **0.9K / 0.9K**; Medicine VQA **9K / 6.9K** and **1K / 1K**; OCR VQA **12K / 12.1K** and **1.4K / 1.4K**; and Arts VQA **9K / 7K** and **1K / 0.9K** [2508.08275].

To expose order sensitivity, the benchmark uses two permutations. **Order‑A** is: Math QA $\rightarrow$ Arts VQA $\rightarrow$ Math VQA $\rightarrow$ Economics QA $\rightarrow$ Medicine VQA $\rightarrow$ OCR VQA $\rightarrow$ Science VQA. **Order‑B** is the reverse sequence [2508.08275]. The model is evaluated on all 7 tasks after each training stage, yielding post-task performance matrices $P_{i,j}$ for final answers and $C_{i,j}$ for CoT scores.

The benchmark starts from pretrained MLLMs, specifically **LLaVA‑1.5‑7B** and **Qwen2.5‑VL‑3B**, and studies zero-shot behavior, direct fine-tuning, multi-task joint training, sequential supervised fine-tuning under continual learning algorithms, and reinforcement learning with GRPO on the Qwen backbone [2508.08275].

## 3. Evaluation architecture: answer accuracy and CoT analysis

MLLM‑CTBench employs a two-tier evaluation stack. At the macro level, it evaluates final answers using task-appropriate extraction and scoring; at the micro level, it scores the generated reasoning chain along three dimensions: **logical coherence**, **visual grounding fidelity**, and **domain knowledge retention** [2508.08275].

The macro continual metrics are **Average Performance (AP)** after the final task and **Backward Transfer (BWT)**:
$$
AP = \frac{1}{N} \sum_{j=1}^{N} P_{N,j},
$$
$$
BWT = \frac{1}{N-1} \sum_{j=1}^{N-1} \bigl(P_{N,j} - P_{j,j}\bigr).
$$
Here, $P_{i,j}$ denotes accuracy on task $j$ after training task $i$, and $N=7$ [2508.08275]. Negative BWT indicates catastrophic forgetting.

At the CoT level, the benchmark first considers a general-purpose MLLM-as-judge based on **Qwen‑VL‑2.5‑32B**, then introduces a dedicated evaluator to improve reliability [2508.08275]. The specialized evaluator uses **Qwen2.5‑VL‑7B** as its base model and is trained in two stages: supervised fine-tuning to imitate GPT‑4 absolute CoT scores, followed by GRPO-based RL using GPT‑4 pairwise preferences as reward signals. The paper reports that this evaluator achieves consistently higher **Spearman’s $\rho$**, **Pearson correlation $r$**, and **Kendall’s $\tau$** than the general evaluator across Qwen SFT, Qwen RL, and LLaVA SFT traces [2508.08275].

CoT scores are tracked analogously to answer accuracy. Let $C_{i,j}$ denote the CoT score for task $j$ after training task $i$; CoT AP and CoT BWT are then computed by substituting $C$ for $P$ in the continual metrics [2508.08275]. This yields the benchmark’s central reasoning-level diagnostic: answer forgetting and reasoning forgetting need not coincide.

## 4. Training paradigms and continual learning algorithms

MLLM‑CTBench compares **two training paradigms**. The first is **supervised fine-tuning (SFT)** using standard next-token prediction on instruction–answer pairs. The second is **reinforcement learning with GRPO**, which optimizes expected reward over generated responses while imposing KL regularization to stabilize policy updates [2508.08275].

For task $\mathcal{T}_i$, the SFT objective is written as
$$
\mathcal{L}_{\mathcal{T}_i} = \frac{1}{N_i} \sum_{j=1}^{N_i} \ell \bigl( f_\theta(X^{\text{img}}_{i,j}, X^{\text{ins}}_{i,j}), X^{\text{ans}}_{i,j} \bigr),
$$
and the GRPO objective as
$$
\mathcal{L}_{\text{GRPO}} =
\mathbb{E}_{(s,a)\sim \pi_{\theta_{\text{old}}}}
\bigg[
\frac{\pi_\theta(a\mid s)}{\pi_{\theta_{\text{old}}}(a\mid s)} A^\pi(s,a)
-
\beta\, \mathrm{KL}\bigl(\pi_\theta \,\|\, \pi_{\theta_{\text{old}}}\bigr)
\bigg].
$$
The benchmark also evaluates **GRPO w/o KL** to isolate the role of the KL term in forgetting [2508.08275].

The **8 continual learning algorithms** are grouped into **4 categories** [2508.08275]. The regularization-based methods are **EWC**, **MAS**, **LwF**, and **freeze**, where the freeze setting uses **freeze-first-8** or **freeze-last-8** transformer layers together with a frozen vision encoder. The replay-based methods are **ER** and **DER**. The architectural expansion method is **L2P**, which learns from a prompt pool. The model-fusion method is **MagMaX**, which performs post-hoc parameter merging.

The benchmark evaluates both **full-parameter tuning** and **LoRA-based adaptation**. For RL, the **vision encoder is frozen**, LoRA is applied on the language model, and GRPO uses KL regularization in the main setting [2508.08275].

## 5. Empirical findings

The benchmark’s first empirical conclusion is that **stronger base models forget less**. Under sequential SFT with **Order‑A**, **LLaVA‑1.5** reaches **AP = 35.60** with **BWT $\approx -15.37$**, whereas **Qwen2.5‑VL** reaches **AP = 57.71** with **BWT $\approx -6.43$** [2508.08275]. The benchmark interprets this as evidence that stronger general capability confers greater robustness to continual interference.

The second conclusion is the benchmark’s most distinctive one: **reasoning chains degrade more slowly than final answers**. For **LLaVA‑1.5**, Order‑A SFT yields answer-level **BWT = -15.37** but CoT-level **BWT = -6.54**. For **Qwen2.5‑VL**, the corresponding values are **-6.43** and **-3.74** [2508.08275]. This supports the **hierarchical forgetting hypothesis**, according to which procedural reasoning habits are more stable than final-step factual recall or output execution.

The third conclusion is that **algorithm effectiveness depends on both model capability and task order**. Replay methods are particularly strong on weaker backbones: on **LLaVA‑1.5**, **ER** reduces forgetting by about **88%** relative to naive SFT, with BWT around **-1.8** versus **-15.3**; on **Qwen2.5‑VL**, the relative improvement is smaller, with BWT **-3.23** versus **-6.43** [2508.08275]. Regularization methods are reported as more effective on Qwen than on LLaVA, while aggressive layer freezing can be counterproductive: for Qwen, freezing the first or last 8 layers often **increases forgetting by about 20%** relative to standard SFT [2508.08275]. **L2P** and **MagMaX** provide stable forgetting reduction across scales, but with different operational trade-offs: prompt-pool growth and retrieval overhead for L2P, and low-overhead post-hoc merging for MagMaX [2508.08275].

Task order matters at the per-task level, but aggregate order effects are smaller. The benchmark reports that some tasks can suffer markedly different forgetting depending on sequence position, yet overall **AP** and **BWT** across Order‑A and Order‑B differ by only about **1% on average** [2508.08275]. This suggests that order sensitivity is real but not uniformly catastrophic at the benchmark level.

The fourth conclusion concerns RL. In the Qwen2.5‑VL **Order‑A** comparison, **SFT** yields **AP $\approx 55.87$** and **BWT $\approx -6.34$**; **GRPO with KL** lowers overall AP to **$\approx 46.82$** but improves BWT to **$\approx -1.91$**; **GRPO w/o KL** drops to **AP $\approx 42.62$** and **BWT $\approx -17.29$** [2508.08275]. The benchmark’s interpretation is explicit: the KL term acts as an implicit memory that constrains policy drift, and without it RL can forget more severely than plain SFT.

## 6. Position in the benchmark landscape, limitations, and extensions

MLLM‑CTBench occupies a specific niche within multimodal evaluation: it is not a general capability benchmark, a domain benchmark, or a benchmark-efficiency framework, but a **continual adaptation benchmark with CoT-aware diagnostics** [2508.08275]. This differentiates it from domain-specific multimodal benchmarks such as **MMFCTUB**, which targets credit table understanding with cross-table constraints and mask-and-recovery metrics [2601.04643], and **M3CoTBench**, which evaluates CoT reasoning in medical image understanding using correctness, efficiency, impact, and consistency metrics [2601.08758]. A plausible implication is that MLLM‑CTBench provides the continual-learning substrate that such specialized benchmarks do not attempt to supply.

The benchmark also interacts conceptually with broader benchmark methodology. The **Information Density** principle argues that difficult, low-redundancy, diverse benchmarks provide more useful signal to model developers than superficially large but weakly diagnostic test suites [2503.10079]. MLLM‑CTBench’s reliance on unsaturated tasks and explicit reasoning analysis is congruent with that principle. Separately, the **MITV** interview strategy shows that on several multimodal benchmarks, reliable ranking can often be recovered from a small adaptive subset of questions, with strong rank alignment sometimes reached using **2–5%** of questions [2506.00883]. This suggests that future continual-learning evaluation protocols could combine MLLM‑CTBench’s CoT-aware diagnostics with more sample-efficient benchmark administration.

The benchmark’s stated limitations are substantial [2508.08275]. Experiments are confined to **3B–7B** models and to **two MLLM families**, so scaling conclusions remain provisional. Only **two task orders** are evaluated rather than the full combinatorial space of permutations. CoT evaluation relies on an automatic MLLM-as-judge, even if fine-tuned and correlation-validated, so human judgment is not fully replaced. The six domains are broad but not exhaustive; the benchmark does not include modalities such as audio or 3D, nor domains such as code/data tables or robotics. These limitations motivate the paper’s future directions: scaling-law studies for forgetting, CoT-aware continual learning algorithms, broader continual RL methods, task-order optimization, and improved multimodal judges [2508.08275].

In that sense, MLLM‑CTBench is best understood as both an evaluation resource and an empirical argument. Its design asserts that continual learning in MLLMs cannot be adequately characterized by endpoint accuracy alone, and its results support the stronger claim that reasoning traces, answer outputs, optimization regime, and task sequence should be analyzed as partially separable variables in multimodal continual adaptation [2508.08275].

Source: https://www.emergentmind.com/topics/mllm-ctbench