Continual Bench: Sequential Evaluation

• Continual Bench is a framework defined by sequential evaluation protocols that incorporate time-ordering, limited data access, and measured metrics such as plasticity and forgetting.
• It utilizes protocols like support-set sequencing and task incremental setups to assess key performance indicators including forward transfer and retention across varied modalities.
• The approach extends to systems engineering through continuous benchmarking infrastructures that automate performance tracking on evolving hardware and code bases.

“Continual Bench” designates a class of benchmark frameworks built to evaluate systems under sequential change rather than a single static train–test split. The literature suggests two closely related usages. In continual learning, a benchmark specifies ordered tasks, restricted access to past data, and metrics for plasticity, retention, transfer, and forgetting across modalities such as few-shot classification, robotic control, image editing, video–language understanding, coding agents, biomedical NLP, and multimodal unlearning (Antoniou et al., 2020, Wołczyk et al., 2021, Liu et al., 14 May 2026, Tang et al., 2024, Joshi et al., 13 Jun 2025, Zeng et al., 17 Mar 2026, Wang et al., 7 May 2026). In systems and software engineering, the term also denotes continuous benchmarking infrastructures that extend continuous integration to performance testing on evolving hardware, code bases, and agent requirements (Alt et al., 2024, Vogelsang et al., 17 Apr 2026, Saxena et al., 13 Nov 2025).

1. Conceptual foundations

A continual benchmark differs from a static benchmark by making temporal order part of the problem definition. In the continual few-shot learning framework, a task is a sequence of support sets, $\mathcal{G}=\{S_n\}_{n=1}^{N_G}$, followed by a target set $T$, under a strict sequential constraint: each support set is sampled without replacement, deleted after use, and the learner never has access to more than one support set at a time (Antoniou et al., 2020). This formalism places few-shot learning, continual few-shot learning, and full continual learning on a spectrum defined by support-set size and sequence length rather than treating them as unrelated regimes.

The same sequential principle appears in modality-specific definitions. In continual image editing, the objective is to train a single diffusion-based image editing model to acquire new editing capabilities over time, task by task, without access to all past data and without forgetting earlier edits (Liu et al., 14 May 2026). In multimodal instruction tuning, the model is tuned sequentially on tasks $$\mathcal{T}_1 \rightarrow \cdots \rightarrow \mathcal{T}_N$$ and evaluated on all tasks seen so far, so the benchmark measures not only present-task accuracy but also retention across the entire sequence (Guo et al., 31 Jul 2025). In continual unlearning, the sequential object is no longer task acquisition but repeated deletion requests: after each unlearning step, the model must forget the current target data, maintain forgetting of all previously forgotten data, preserve non-target utility, and remain stable over long horizons (Wang et al., 7 May 2026).

A second meaning of the term shifts from model adaptation to infrastructure. Continuous benchmarking in high-performance computing is defined as the performance analogue of continuous integration: benchmarks are triggered by commits, standardized workloads are rerun on real hardware, and results are stored and visualized over time to expose regressions and trends (Alt et al., 2024). A closely related systems view separates workflow, platform, machine, and implementation layers so that benchmarking becomes researcher-agnostic and machine-agnostic at the entry point, even when execution remains HPC-specific (Vogelsang et al., 17 Apr 2026). This suggests that “Continual Bench” is best understood as a family resemblance term uniting sequential evaluation protocols rather than a single benchmark instance.

2. Sequential protocols and benchmark construction

Across domains, continual benchmarks are defined as much by protocol as by data. CIE-Bench makes image editing explicitly continual by fixing a base model, using rank-48 LoRA adapters for all methods, training one task at a time for 10 epochs per task, withholding previous-task data except in a Data Rehearsal baseline, and testing multiple task permutations such as CVLEFT, FVELTC, and EFLTVC (Liu et al., 14 May 2026). MLLM-CTBench adopts a task-incremental continual instruction-tuning setup with $N=7$ tasks, repeated under two task orders, so that answer accuracy and chain-of-thought quality can be tracked after each stage of tuning (Guo et al., 31 Jul 2025). MedCL-Bench extends this logic to biomedical NLP with ten datasets, eight pre-specified randomized task orders, and evaluation after each stage on all tasks seen so far (Zeng et al., 17 Mar 2026).

Several benchmarks make streaming rather than replay the core abstraction. CLEAR organizes ten years of Flickr imagery into chronological buckets and argues that conventional iid train–test splits overestimate continual-learning performance; its streaming protocol trains on bucket $i$, evaluates on bucket $i+1$, and then repurposes today’s test data as tomorrow’s train data (Lin et al., 2022). In human state monitoring, the continual unit is not a class subset but subject identity: each experience contains data from two new subjects, the label space remains fixed, and the held-out test subject remains constant throughout the sequence (Matteoni et al., 2022). In continual semi-supervised learning for activity recognition and crowd counting, the only labeled data are the initial supervised folds $S$, while subsequent validation and test streams are unlabeled for training and split into temporal sessions for self-training updates (Shahbaz et al., 2021).

Chronological ordering also structures software-engineering variants. SWE-Bench-CL reorganizes SWE-Bench Verified into per-repository sequences ordered by GitHub issue creation time and then by difficulty, producing 273 tasks across 8 sequences and enabling evaluation of coding agents under a repository’s natural evolution (Joshi et al., 13 Jun 2025). In continual reinforcement learning, Continual World fixes a shared state and action space across Meta-World tasks and trains a single agent sequentially for a fixed budget per task (Wołczyk et al., 2021). The more recent Continual Bench environment for CRL goes further by embedding six manipulation tasks in a single MuJoCo tabletop world with unified dynamics $P^u$, so only the reward $R^\tau$ and initial distribution $\rho_0^\tau$ change across tasks (Liu et al., 12 Jul 2025).

3. Representative benchmark families

The benchmark landscape now spans a wide range of modalities, task structures, and continual units.

Benchmark	Domain	Sequential unit
"Defining Benchmarks for Continual Few-Shot Learning" (Antoniou et al., 2020)	Few-shot classification	Support sets within one CFSL episode
"The CLEAR Benchmark: Continual LEArning on Real-World Imagery" (Lin et al., 2022)	Real-world image classification	Chronological buckets from 2004–2014
"Continual World: A Robotic Benchmark For Continual Reinforcement Learning" (Wołczyk et al., 2021)	Robotic manipulation	Fixed task sequences such as CW10 and CW20
"Continual Reinforcement Learning by Planning with Online World Models" (Liu et al., 12 Jul 2025)	MuJoCo continual RL	Six reward-defined tasks in one unified world
"ViLCo-Bench: VIdeo Language COntinual learning Benchmark" (Tang et al., 2024)	Video–language episodic memory	Query-incremental MQ, NLQ, and VQ sub-tasks
"ACE-LoRA: Adaptive Orthogonal Decoupling for Continual Image Editing" (Liu et al., 14 May 2026)	Diffusion-based image editing	ERP, Focus, Light, Text, Tryon, Causal
"T2I-ConBench: Text-to-Image Benchmark for Continual Post-training" (Huang et al., 22 May 2025)	Text-to-image diffusion	Item customization, domain enhancement, mixed sequences
"MLLM-CBench: A Comprehensive Benchmark for Continual Instruction Tuning of Multimodal LLMs with Chain-of-Thought Reasoning Analysis" (Guo et al., 31 Jul 2025)	Multimodal instruction tuning	Seven tasks across sixteen datasets
"MedCL-Bench: Benchmarking stability-efficiency trade-offs and scaling in biomedical continual learning" (Zeng et al., 17 Mar 2026)	Biomedical NLP	Ten tasks under eight task orders
"ICU-Bench: Benchmarking Continual Unlearning in Multimodal LLMs" (Wang et al., 7 May 2026)	Multimodal unlearning on documents	100 forget tasks in 10 batches

Two additional families broaden the scope further. "SWE-Bench-CL: Continual Learning for Coding Agents" reformulates repository maintenance as sequential issue solving with explicit memory and transfer analysis (Joshi et al., 13 Jun 2025). The CSSL benchmarks "Continual Activity Recognition" and "Continual Crowd Counting" move continual evaluation into unlabeled streams, where the learner must update from pseudo-labels rather than supervised task batches (Shahbaz et al., 2021). Human state monitoring benchmarks derived from WESAD and ASCERTAIN turn subject arrival into the source of non-stationarity, thereby emphasizing domain-incremental accumulation rather than class-incremental forgetting (Matteoni et al., 2022).

The diversity of these suites is not merely taxonomic. CIE-Bench covers granularity from global panoramic reconstruction to single-word text editing (Liu et al., 14 May 2026), ViLCo-Bench uses ten-minute egocentric videos and query-incremental tasks rather than class-incremental labels (Tang et al., 2024), and ICU-Bench forces deletion across full-image, masked-image, text-only, and description views of privacy-critical documents (Wang et al., 7 May 2026). This suggests that contemporary continual benchmarks increasingly encode domain structure directly into the benchmark protocol instead of treating sequentiality as a generic wrapper around iid datasets.

4. Metrics, diagnostics, and evaluation logic

A defining property of continual benchmarks is that they combine task-quality metrics with temporal summaries. In CIE-Bench, per-sample editing quality is measured by Instruction Following and Perceptual Naturalness, each scored from 1 to 5 by a specialized MLLM-based evaluator, and the primary score is

$T$0

with $T$1 (Liu et al., 14 May 2026). In T2I-ConBench, retention of pretrained generality is assessed with FID and a complex-composition subset of T2I-CompBench, target-task performance with Unique-Sim or domain HPS, forgetting with Unique-Forget or Domain-Forget, and cross-task generalization with Item+Item, Item+Domain, and Domain+Domain VQA-based composition tests (Huang et al., 22 May 2025). MLLM-CTBench adds a second axis beyond answer accuracy by scoring chain-of-thought traces for logical coherence, visual grounding fidelity, and domain knowledge retention (Guo et al., 31 Jul 2025).

Temporal summaries usually adopt continual-learning notation. CIE-Bench reports Last, Avg, Imm., and Backward Transfer, with

$T$2

so less-negative values indicate less forgetting (Liu et al., 14 May 2026). MLLM-CTBench and MedCL-Bench use Average Performance and BWT in analogous forms, the latter also adding Forward Transfer relative to zero-shot backbone performance (Guo et al., 31 Jul 2025, Zeng et al., 17 Mar 2026). CLEAR uses an accuracy matrix $T$3 and emphasizes Next-domain Accuracy, the average of the superdiagonal under a streaming protocol that always evaluates on the near future (Lin et al., 2022). Continual World makes forward transfer a primary metric through normalized area under the learning curve and separately reports forgetting and average performance across seen robotic tasks (Wołczyk et al., 2021).

Some benchmarks introduce domain-specific efficiency or stability measures. Continual few-shot learning proposes Across-Task Memory,

$T$4

to quantify the size of explicit stored representations relative to raw data, and pairs it with MACs to expose memory–compute trade-offs (Antoniou et al., 2020). SWE-Bench-CL augments ACC, Forgetting, FT, BWT, and AULC with Tool-use Efficiency, a Composite Continual Learning Score, and CL-$T$5, which combines CL-Plasticity and CL-Stability into a single stability–plasticity summary (Joshi et al., 13 Jun 2025). ICU-Bench introduces Retain Stability Rate for inter-batch drift on retain sets, Forgetting Rebound for historical forget recovery, and Generation Quality scored on a $T$6–$T$7 scale to detect collapse even when forgetting appears numerically strong (Wang et al., 7 May 2026).

The expansion of metric families reflects a shift in what benchmarks are expected to diagnose. A single end-of-sequence accuracy number is no longer considered sufficient when the target phenomenon may involve order sensitivity, evaluator misalignment, replay cost, loss of generic capability, or the re-emergence of previously deleted information.

5. Continuous benchmarking infrastructures and evolving evaluation pipelines

Outside continual learning proper, continuous benchmarking treats performance tracking as an operational workflow. In the HPC setting, the pipeline is built around GitLab CI, a custom GitLab runner attached to a test cluster, Slurm submission via sbatch, LIKWID and NVIDIA Nsight Compute for measurements, InfluxDB for time-series storage, Kadi4Mat for FAIR artifact storage, and Grafana or Plotly for visualization (Alt et al., 2024). The purpose is not episodic task retention but systematic re-benchmarking of representative workloads whenever code or hardware changes, so that regressions become visible as time-series rather than anecdotal observations.

A more abstract formulation separates workflow, architecture, and implementation. The workflow layer specifies stages such as prepare, build, execute, transfer, annotate, analyze, and plot; the architecture layer specializes them to a CI platform and a target machine; the implementation layer provides benchmark-aware command logic parameterized by hierarchical key–value configurations (Vogelsang et al., 17 Apr 2026). This architecture supports researcher-agnostic operation, role-based maintenance by machine and software experts, and reuse of centralized configurations across groups. The underlying idea is that benchmarking itself should become a versioned, automated artifact rather than a one-off experimental script.

A third extension appears in enterprise LLM-agent evaluation, where even the benchmark set is generated continuously. "Continuous Benchmark Generation for Evaluating Enterprise-scale LLM Agents" formalizes inputs $T$8 from migrated services and developer-authored Knowledge Base documents and outputs benchmark items

$T$9

where $$\mathcal{T}_1 \rightarrow \cdots \rightarrow \mathcal{T}_N$$0 is the pre-migration state, $$\mathcal{T}_1 \rightarrow \cdots \rightarrow \mathcal{T}_N$$1 is a KB task document, and $$\mathcal{T}_1 \rightarrow \cdots \rightarrow \mathcal{T}_N$$2 is the set of migration-related diff hunks relevant to that task (Saxena et al., 13 Nov 2025). Keyword matching and LLM-generated regexes link high-level intent to concrete hunks, and evaluation proceeds with line-edit precision/recall and per-KB precision/recall. Here, “continual benchmark” no longer means only sequential model updating; it also means that the benchmark definition itself evolves with requirements, services, and organizational documentation.

6. Limitations, misconceptions, and open problems

A recurring misconception is that all continual benchmarks primarily measure catastrophic forgetting. Several suites explicitly dispute that view. Continual World argues that the field has overly focused on forgetting and that forward transfer should be prioritized in robotic continual RL (Wołczyk et al., 2021). Human state monitoring benchmarks show the opposite regime from many class-incremental image benchmarks: because the setting is domain-incremental, forgetting can be “easily tackled even with a simple finetuning,” while existing strategies struggle to accumulate knowledge on a fixed held-out subject (Matteoni et al., 2022). CLEAR similarly argues that iid train–test protocols artificially inflate performance and that streaming evaluation on the near future is more realistic (Lin et al., 2022).

Another limitation is evaluator dependence. CIE-Bench relies on Qwen3.5-plus with task-specific prompts and notes dependence on the evaluator’s visual understanding and prompt design (Liu et al., 14 May 2026). MLLM-CTBench uses a dedicated multimodal chain-of-thought evaluator because generic judges correlate less well with human ratings, yet still acknowledges evaluator dependence and limited order coverage (Guo et al., 31 Jul 2025). ICU-Bench’s synthetic document domains provide controllable privacy-critical data, but only two domains are covered and real-world multimodal document noise remains outside the benchmark (Wang et al., 7 May 2026).

Task order and compute are now first-class variables rather than secondary details. MedCL-Bench shows that continual methods occupy distinct retention–compute frontiers and that robustness must be assessed across eight task orders, not a single sequence (Zeng et al., 17 Mar 2026). T2I-ConBench reports that no approach excels on all fronts, that even joint “oracle” training does not succeed for every task, and that cross-task generalization remains unsolved (Huang et al., 22 May 2025). ViLCo-Bench highlights a different bottleneck: ten-minute videos make replay memory and multimodal alignment far more difficult than in image-based or short-video CL (Tang et al., 2024).

Continuous benchmarking infrastructures have their own unresolved issues. The HPC pipeline in (Alt et al., 2024) still emphasizes single-node continuous benchmarking and relies mainly on visual inspection rather than statistical regression alarms, while the layered CI-beNNch design notes that true executor independence is constrained by HPC security and accounting policies and that metadata lookup remains a practical concern (Vogelsang et al., 17 Apr 2026). The enterprise benchmark-generation pipeline depends on high-quality Knowledge Base documents and incurs LLM cost for pattern synthesis, so benchmark validity becomes partly a documentation and workflow problem rather than only a data problem (Saxena et al., 13 Nov 2025).

Taken together, these limitations indicate that “Continual Bench” is not converging toward a single canonical template. Instead, the field is moving toward domain-specific, protocol-heavy, and metric-rich evaluation suites in which sequentiality, order, memory, evaluator design, and infrastructure are all part of the benchmark definition.