Continual Learning Benchmarks Overview

• Continual learning benchmarks are frameworks comprising datasets, protocols, and metrics that assess models’ ability to incrementally learn across evolving tasks without forgetting.
• They cover diverse modalities such as images, videos, text, and reinforcement learning environments to critically analyze stability-plasticity trade-offs.
• Protocols range from synthetic splits to streaming and dynamic task discovery, offering actionable insights for robust algorithm evaluation and real-world deployment.

Continual learning benchmarks are frameworks, datasets, and protocols designed to systematically measure the ability of machine learning models to acquire, transfer, and retain knowledge across evolving task streams without catastrophic forgetting. These benchmarks enable rigorous, reproducible, and multifaceted evaluation of continual learning (CL) algorithms across diverse settings, task modalities, and real-world phenomena. They address fundamental questions regarding stability-plasticity trade-offs, domain and class incrementality, task similarity, resource constraints, and realistic deployment conditions.

1. Taxonomy of Continual Learning Benchmarks

Contemporary CL benchmarks span a wide spectrum of data modalities, task granularities, and evaluation settings, reflecting the breadth of the continual learning landscape.

• Image classification: Classic synthetic protocols (Permuted-MNIST, Split-CIFAR), curriculum-based heterogeneous streams ("M2I"/"I2M") (Faber et al., 2023), and temporally structured real-world datasets ("CLEAR") (Lin et al., 2022).
• Video and spatio-temporal data: Action recognition streams (UCF101, confidence-driven rehearsal) (Castagnolo et al., 2023).
• Time series and signals: Domain-incremental benchmarks for physiological monitoring (WESAD, ASCERTAIN) (Matteoni et al., 2022).
• Dialogue and NLP: Modular multi-domain dialogue (Task-Oriented Dialogue, 37 domains) (Madotto et al., 2020), biomedical multi-task NLP (MedCL-Bench, 10 tasks, 5 families) (Zeng et al., 17 Mar 2026).
• Few-shot and instance recognition: CFSL and instance-level extensions (Omniglot, SlimageNet64) (Kowadlo et al., 2022).
• Reinforcement learning: Robotic manipulation (Continual World, Meta-World), multi-domain game sequences and household environments (CORA) (Wołczyk et al., 2021, Powers et al., 2021).
• Code generation and software engineering: Chronologically ordered patch sequences from real GitHub projects (SWE-Bench-CL) (Joshi et al., 13 Jun 2025).
• Generative modeling: Continual learning of generative models (CLoG: GANs, Diffusion, label- and concept-conditioned) (Zhang et al., 2024).
• LLMs: Intrinsically challenging tasks and alignment-aware continual learning in LLMs (TRACE) (Wang et al., 2023).
• Semi-supervised/unsupervised streaming: Continual semi-supervised frameworks for activity recognition and crowd counting (CSSL) (Shahbaz et al., 2021).
• Dynamic, method-adaptive protocols: MDP-based, algorithm-conditioned dynamic benchmarks (CLDyB) leveraging MCTS for automatically mining hard task sequences (Chen et al., 6 Mar 2025).

Benchmarks are further distinguished by their incrementality scenario (class-, domain-, or task-incremental), the presence or absence of task-ID at test time, realism of drift (synthetic vs. real-world), and degree of temporal granularity (batch, sessional, or pure streaming).

2. Protocols and Dataset Construction Strategies

Evaluation setup is critical for credible CL benchmarking and influences algorithm ranking, interpretability, and transferability to real deployments.

2.1 Task Construction

• Synthetic Splits: Uniform random partitioning (Split-MNIST, Split-CIFAR), permutations (Permuted-MNIST), or domain-altering transformations (RotatedMNIST).
• Curriculum-Ordered Streams: Structured sequences graded by visual or semantic complexity (e.g., MNIST → TinyImageNet in M2I; TinyImageNet → MNIST in I2M) (Faber et al., 2023).
• Real-World Temporal Streams: Chronological slices with smoothly evolving distributions (CLEAR: 11 YFCC100M buckets by year (Lin et al., 2022); CLOC: 39M Flickr images, albums ordered by upload time (Cai et al., 2021)).
• Dynamic Task Discovery: CLDyB models CL benchmarking as a Markov Decision Process, dynamically mining difficult, algorithm-specific class subsets via MCTS and functional clustering (Chen et al., 6 Mar 2025).

2.2 Evaluation Protocols

• Classical (offline) protocol: IID class/task splits; train/test per batch; cumulative metrics aggregated post hoc.
• Streaming protocol: Models are evaluated on each incoming batch/period before being allowed to update, providing a "test on tomorrow’s data" lens (CLEAR (Lin et al., 2022), CLOC (Cai et al., 2021)).
• Dynamic protocol (CLDyB): Task order is adaptively chosen via an MDP-based planner, tailored to maximize forgetting or minimize performance given the current learner state (Chen et al., 6 Mar 2025).
• Domain-specific conventions:
  • LLMs: trace alignment and reasoning preservation post-task sequence (TRACE (Wang et al., 2023)).
  • RL: cycles of evaluation after each episode or task, metrics normalized by maximum episode return (CORA (Powers et al., 2021)).
  • Dialogue and biomedical NLP: curriculum over heterogeneous tasks and domain permutations with order-robustness reporting (MedCL-Bench (Zeng et al., 17 Mar 2026)).

2.3 Dataset Curation

Modern CL benchmarks employ advanced visio-linguistic pipelines (e.g., CLIP-based interactive filtering in CLEAR), human verification (SWE-Bench-CL), and crowd-sourcing for annotation quality and privacy screening (Lin et al., 2022, Joshi et al., 13 Jun 2025).

3. Metrics and Multi-Attribute Evaluation

Evaluating continual learning models necessitates metrics that measure not just one-step accuracy but all facets of continual adaptation.

Metric	Formal Definition
Average Accuracy ($A_T$)	$\frac{1}{T}\sum_{i=1}^T R_{i,i}$
Forgetting ($F$)	$\frac{1}{T-1}\sum_{k=1}^{T-1} F_k$, $F_k = \max_{l\le T-1} R_{l,k} - R_{T,k}$
Backward Transfer (BWT)	$$\frac{1}{T(T-1)/2}\sum_{i=2}^T\sum_{j=1}^{i-1}\left(R_{i,j}-R_{j,j}\right)$$
Forward Transfer (FWT)	$$\frac{1}{T(T-1)/2}\sum_{i=1}^{T-1}\sum_{j=i+1}^T\left(R_{i,j}-b_j\right)$$
Streaming Next-Domain Acc.	$\frac{1}{T-1}\sum_{i=1}^{T-1} R_{i,i+1}$
Model Size/Sample Storage	Ratio of parameter or memory expansion per task (e.g., $MS=(1/N)\sum_k Mem(\theta_1)/Mem(\theta_k)$)
Compute Efficiency	Fraction of multiply–add operations or GPU-hour cost compared to baseline
CL-F$_\beta$ (SWE-Bench-CL)	Harmonic mean of plasticity and stability: $\frac{1}{T}\sum_{i=1}^T R_{i,i}$0
Composite CL Score (MAVT)	$\frac{1}{T}\sum_{i=1}^T R_{i,i}$1 (weighted aggregation over accuracy, transfer, efficiency, etc.) (Díaz-Rodríguez et al., 2018)

Domain- and scenario-specific metrics include area under learning curve (AULC), per-task semantic drift (SWE-Bench-CL), alignment deltas in LLMs, and generative quality metrics (FID, AFQ, AIQ, FR in CLoG (Zhang et al., 2024)).

4. Analysis of Baselines, Performance Patterns, and Method-Specific Findings

Most benchmarks implement a broad taxonomy of CL strategies:

• Regularization-based: EWC, SI, MAS, LwF—parameter anchoring via Fisher information, surrogate loss, or distillation. These excel in short task streams but degrade rapidly with longer or more heterogeneous sequences due to the intersection of parameter constraints shrinking with each task (Liu et al., 2023).
• Replay-based: Sample or generative replay methods (ER, DER++, iCaRL, GEM, AGEM). Empirically robust—iCaRL on SplitCIFAR100 up to 50 tasks; DER++ dominant in video action streams—yet buffer size and sample selection (e.g., GSS for gradient diversity) are critical for efficacy (Castagnolo et al., 2023, Liu et al., 2023).
• Parameter isolation: Task-specific heads, adapters, or LoRA layers (AdapterCL, PackNet, C-LoRA). These can yield perfect retention but with linear parameter growth; AdapterCL is state-of-the-art in dialogue, biomedical NLP, and LLM settings for stability-efficiency trade-off (Madotto et al., 2020, Zeng et al., 17 Mar 2026).
• Dynamic/planned approaches: Dynamic curriculum (CLDyB) exposes and exacerbates model-dependent failure modes not visible in static splits and increases the diagnostic power of evaluation sequences (Chen et al., 6 Mar 2025).
• Specialized neural architectures: Task-wrapped networks, prompt-based ensembling, or compositional/hierarchical modules are emerging as efficient alternatives, especially for parameter- and compute-constrained settings (HiDe-Prompt (Chen et al., 6 Mar 2025); TCL (Zeng et al., 17 Mar 2026)).
• Online/streaming-optimized methods: Population Learning Rate Search (PoLRS), ADRep dynamic buffer, and online test-then-train protocols are necessary for evaluating and optimizing streaming learners (Cai et al., 2021, Lin et al., 2022).

Common findings:

• Regularization is brittle under scaling (sequence length, heterogeneity, model size).
• Replay (with optimized buffer management—reservoir, GSS, confidence-driven selection) dominates most vision and video benchmarks but at the cost of memory.
• Parameter isolation provides the highest "retention per cost" on large LLMs or biomedical NLP (Zeng et al., 17 Mar 2026).
• Task order and granularity (fine/coarse, direct/inverse curriculum) affect both baseline performance and the ability to trade off stability and plasticity (Faber et al., 2023).
• Streaming/online protocols, as in CLEAR and CLOC, better capture real-world generalization and drift than classical train/test splits (Lin et al., 2022, Cai et al., 2021).

5. Benchmark-Specific Innovations and Practical Recommendations

Recent benchmarks have introduced novel methodologies tailored to expose the limitations of CL algorithms and better match real-world deployment needs.

• CLEAR (Lin et al., 2022): Emphasizes temporal smoothness and natural distribution drift; leverages CLIP embeddings and crowdsourcing for scalable, high-precision curation; recommends streaming protocols, abundant unlabeled data per time-bucket, and biasing replay towards recency.
• MedCL-Bench (Zeng et al., 17 Mar 2026): First biomedical NLP benchmark with multi-order protocol, stability–efficiency Pareto front reporting, and per-task family analysis; strongly recommends adapters for high AP/cost ratio, and reporting order robustness.
• CLDyB (Chen et al., 6 Mar 2025): Introduces dynamic, method-challenging task ordering via MDPs and MCTS, enabling continuous self-evolution of the benchmark in response to model capabilities and surfacing method-dependent vulnerabilities.
• SWE-Bench-CL (Joshi et al., 13 Jun 2025): Employs human-verified chronological patch sequences, dependency annotations, interactive agentic evaluation, and the CL-F$\frac{1}{T}\sum_{i=1}^T R_{i,i}$2 metric to operationalize the stability-plasticity trade-off in code generation.
• TRACE (Wang et al., 2023): Focuses on the preservation of LLM alignment, instruction-following, and reasoning via cross-modal, multilingual, reasoning-augmented curriculum; introduces measures for general ability and safety retention in addition to classical CL metrics.
• CLoG (Zhang et al., 2024): Systematizes continual learning of generative models, establishing unified protocols, metrics (FID, AFQ, FR), and fair GAN/Diffusion baselines; highlights the high risk of mode collapse and buffer mismatch in generative replay.

Recommendations across benchmarks emphasize:

• Favor streaming or online protocols over classical IID splits.
• Always report order robustness—use multiple task permutations where possible.
• For rehearsal, optimize buffer composition, favor recent or uncertain samples, and avoid naive replay under severe distributional imbalance.
• Employ adapters or modular extensions when compute and parameter efficiency are critical.
• Make use of curriculum structure (when available) and analyze forward and backward transfer explicitly, not just final accuracy.
• Public release of code, data, and evaluation scripts are essential for reproducibility and adoption.

6. Limitations, Open Problems, and Future Directions

Despite recent advances, systematic gaps remain:

• Static benchmarks can lead to performance saturation and do not reflect algorithm-dependent sequence difficulty; dynamic versions such as CLDyB address this but at increased computational complexity and methodological overhead.
• Real-world applicability is limited by simplistic task and drift models; large-scale, temporally evolving, privacy-conscious, and multimodal benchmarks (CLEAR, SWE-Bench-CL, MedCL-Bench) provide partial solutions.
• Faithful modeling of concept drift, task autodiscovery, data contamination in pretraining, and unsupervised/semi-supervised continual adaptation remain major unsolved challenges.
• No single method or class of methods achieves efficient stability–plasticity trade-off across all benchmark types and domains—algorithms require domain- and resource-specific tuning.
• Emerging research advocates for meta-learning of task relations, game-theoretic dynamic regularization, and unified multi-attribute value frameworks for more practically relevant algorithm assessment.

Continual learning benchmarks are thus not static artifacts but evolving frameworks—driven by advances in neural architectures, data curation, and evaluation protocol design—crucial for the advancement of robust, adaptive, and general AI systems.