---
title: 'CL-Bench: Continual Learning Benchmark'
url: https://www.emergentmind.com/topics/continual-learning-bench-cl-bench
type: topic
---

# CL-Bench: Continual Learning Benchmark

Searching arXiv for the benchmark and closely related papers to ground the article in current literature.
Continual Learning Bench (CL-Bench) is a benchmark for evaluating whether frontier LLM-based systems improve through sequential experience in realistic, stateful environments. It is introduced as “the first difficult, expert-validated benchmark specifically designed to test whether frontier LLM-based systems genuinely improve through sequential experience” and is organized around environments in which tasks share a learnable latent structure that a stateful system can discover online but a stateless one cannot [2606.05661]. In the benchmark’s framing, continual learning is not merely long-context recall or memory retrieval, but the ability to use prior online experience to improve later performance in the same environment. CL-Bench spans six domains—software engineering, signal processing, disease outbreak forecasting, database querying, strategic game-playing, and demand forecasting—and evaluates both stateless and stateful agent architectures with a metric designed to isolate online learning from baseline model capability [2606.05661].

## 1. Conceptual scope and benchmark motivation

CL-Bench is motivated by a gap between existing evaluations of language-model systems and the practical notion of continual learning required in long-lived deployments. The benchmark argues that static one-shot benchmarks primarily measure capabilities acquired during pretraining, while many long-context or memory evaluations focus on recall fidelity rather than on whether an agent actually improves through repeated interaction in the same environment [2606.05661]. The benchmark therefore targets a narrower but more deployment-relevant question: whether an LLM-based system performs better on future instances because it learned from prior instances in that same task.

The benchmark formalizes a distinction between stateless and stateful systems. For a sequence \(x_1,\dots,x_N\), with history \(\mathcal{H}_{t-1}=\{(x_i,r_i)\}_{i=1}^{t-1}\), a stateless system uses
\[
\pi_t = \pi(x_t)
\]
while a stateful system uses
\[
\pi_t = \pi(x_t, \mathcal{H}_{t-1}).
\]
This setup is intended to capture continual learning as online performance improvement due to sequentially accumulated experience rather than due to offline pretrained competence alone [2606.05661].

A central design claim is that every CL-Bench task must satisfy three conditions: headroom, shared latent structure, and a learning mechanism. Headroom means that initial performance must be well below the maximum achievable score. Shared latent structure means that tasks contain reusable regularities such as codebase layout, schema conventions, opponent tendencies, or persistent forecasting factors. A learning mechanism means that earlier instances produce observations informative for later ones, allowing a stateful system to improve while a stateless one cannot [2606.05661].

## 2. Formal task structure and stateful evaluation

CL-Bench defines a hierarchy of benchmark units. A task is a sequence of related instances. An instance is one problem within a task, and reward is defined at this level. A step is one action taken by the agent within an instance. A system is the full evaluated unit, including both the base model and any memory or adaptation machinery. A variant is a modified environment within a task, often used to induce concept drift. A schedule is the ordered sequence of instances a system processes, potentially spanning multiple variants [2606.05661].

This structure is significant because the benchmark is explicitly designed around stateful environments rather than independent examples. The six domains are meant to mimic ongoing professional workflows in which later performance can improve if the system discovers environment-specific regularities. The benchmark further allows concept drift through variant changes, so the evaluation includes both retention and adaptation. This suggests that CL-Bench targets a stability–plasticity trade-off more directly than benchmarks centered only on static or iid task collections.

The benchmark uses structured outputs for automated scoring and, in some domains, persistent terminal or tool-based environments. Full evaluation covers approximately 300+ instances across the six tasks, with task lengths ranging from 12 to 120 instances [2606.05661]. This suggests that CL-Bench is neither a one-shot static benchmark nor a purely synthetic stream generator; it is a collection of curated sequential environments with explicit task horizons.

## 3. Benchmark domains and latent structure

The six benchmark domains are Blind Spectrum Monitoring, Codebase Adaptation, Cohort Studies, Database Exploration, Exploitable Poker, and Sales Prediction [2606.05661]. Each domain is intended to contain a recurring latent structure that can be inferred only through sequential interaction.

Blind Spectrum Monitoring models RF spectrum analysis over a 168 MHz band and 90 scans. The latent structure is the persistent set of channel center frequencies and bandwidths; transmitters may go dormant for long intervals, so later predictions benefit from remembering earlier detections. The schedule has three stages: scans 1–30 use `five_ch_wide`, scans 31–60 use `five_plus_four_mixed`, and scans 61–90 use `full_grid_active` [2606.05661].

Codebase Adaptation presents 19 real Python bugfix issues drawn from two repositories, 9 from `jazzband/tablib` and 10 from `jd/tenacity`, each in a fresh Docker container at the issue’s base commit. The reusable structure is repository-specific: file organization, test invocation patterns, and conventions for patching. The task is blocked by repository, creating two local continual-learning phases [2606.05661].

Cohort Studies casts the system as an epidemiological assistant analyzing five sequential fictional studies about Vardan-A Syndrome. The latent structure is a shared set of factors predictive of survival curves across studies, despite differing observed variables and sampling biases. The schedule contains 20 instances organized into five study blocks—HERALD, MERIDIAN, MOSAIC, FORGE, and CADENCE—each with four instances [2606.05661].

Database Exploration asks 40 natural-language questions over an obfuscated SQLite database built from Amazon product reviews and metadata from Office Products, Electronics, and Musical Instruments. The reusable structure includes hidden schema conventions, unit conventions, and category-specific quirks. After question 20, a simulated schema migration changes the environment, introducing a benchmarked form of drift [2606.05661].

Exploitable Poker consists of 120 hands of heads-up Texas Hold’em against deterministic opponent variants. The latent structure is the opponent’s fixed exploitable strategy, with repeated encounters used to test whether the agent forms and reuses an opponent model. The schedule alternates between `calling_station`, `loose_aggressive`, and `fit_or_fold`, with return stages to test retention and re-adaptation [2606.05661].

Sales Prediction casts the system as a forecasting analyst for a furniture retailer with stores in San Francisco, New York, and Chicago. Each instance corresponds to a year, but only one store’s most recent year of history is visible at a time. The reusable structure includes stable growth rates, price-demand relationships, and product/location clusters. The schedule has 12 instances spanning years 2027–2038 in three stages of four instances each [2606.05661].

The following table summarizes the six domains and their schedules.

| Domain | Instances | Core latent structure |
|---|---:|---|
| Blind Spectrum Monitoring | 90 scans | persistent channel set |
| Codebase Adaptation | 19 issues | repository-specific structure |
| Cohort Studies | 20 instances | shared latent survival factors |
| Database Exploration | 40 questions | hidden schema and unit conventions |
| Exploitable Poker | 120 hands | deterministic opponent policy |
| Sales Prediction | 12 instances | stable forecasting factors |

A plausible implication is that CL-Bench deliberately mixes tasks where continual learning arises from very different mechanisms: environmental persistence, repeated latent-causal inference, repository familiarity, opponent modeling, and long-horizon forecasting. This suggests that the benchmark is not intended to reward a single memory heuristic, but to probe whether a system can extract and reuse structure across qualitatively different sequential settings.

## 4. Expert validation and task construction methodology

CL-Bench emphasizes expert validation as a defining property. Each task is first manually reviewed by two authors and then independently reviewed by 2–3 domain experts along three axes: realism, reusable knowledge, and learning improvement. Experts provided task-specific feedback that was used to refine task design. The paper reports that, on average, 2–3 experts spent 2.14 hours per validation cycle, with times ranging from 1 to 4.5 hours [2606.05661].

This construction process distinguishes CL-Bench from synthetic continual-learning benchmarks and from benchmarks built by simple concatenation of independent tasks. In CL-Bench, the claim is not only that tasks are hard, but that they have been explicitly designed and checked so that sequential learning should matter. That design philosophy resembles broader benchmark efforts that aim to isolate realistic continual-learning factors in other modalities, such as subject-driven domain drift in physiological time series [2207.00010], long-horizon multimodal and textual continual tuning [2505.09952], realistic temporal distribution shift in imagery [2201.06289], or real-life context learning in messy human settings [2604.27043]. CL-Bench differs in centering stateful agent interaction rather than standard supervised incremental prediction, but it shares the benchmark-level objective of making improvement from experience measurable rather than assumed.

## 5. Metrics: reward, gain, normalized gain, and stability–plasticity

CL-Bench uses task-specific reward functions, but its most distinctive metric is gain. If \(r_t^{sf}\) is the reward of a system run statefully on instance \(t\), and \(r_t^{sl}\) is the reward of the same system run statelessly on the same instance, then per-instance gain is
\[
g_t = r_t^{sf} - r_t^{sl}
\]
and aggregate gain is
\[
g = \sum_{t=1}^{N} g_t.
\]
This is meant to isolate how much better the system performs because it had prior experience, rather than because it is intrinsically strong on the task [2606.05661].

Because tasks have different reward scales, CL-Bench introduces normalized gain
\[
\widehat{g} = \frac{\bar{r}^{sf} - \bar{r}^{sl}}{r_{max} - \bar{r}^{sl}}
\]
where \(r_{max}\) is a task-specific reference maximum and \(\bar{r}^{sf}\), \(\bar{r}^{sl}\) are average stateful and stateless rewards [2606.05661]. The benchmark also defines normalized reward
\[
\widehat{r} = \frac{\bar{r}^{sf} - r_{external}}{r_{max} - r_{external}}
\]
where \(r_{external}\) is the stateless ICL reward of GPT-5.4 [2606.05661]. This separates two notions: absolute performance and improvement attributable to state.

The benchmark further decomposes normalized gain into stability and plasticity using variant boundaries. Let
\[
\mathcal{B} = \{t : i_t = 0\}, \qquad \mathcal{W} = \{t : i_t > 0\}
\]
where \(i_t\) is the within-variant index for instance \(t\). Then
\[
\widehat{g}_{\text{stab}} = f_\mathcal{B} \cdot \frac{\bar{r}^{sf}_\mathcal{B} - \bar{r}^{sl}_\mathcal{B}}{r_{max} - \bar{r}^{sl}}
\]
and
\[
\widehat{g}_{\text{plas}} = f_\mathcal{W} \cdot \frac{\bar{r}^{sf}_\mathcal{W} - \bar{r}^{sl}_\mathcal{W}}{r_{max} - \bar{r}^{sl}}
\]
with
\[
\widehat{g} = \widehat{g}_{\text{stab}} + \widehat{g}_{\text{plas}}.
\]
The intended interpretation is that stability measures how well knowledge survives shifts across variants, while plasticity measures how well the system adapts within a variant [2606.05661].

This metric design is a defining methodological contribution. Many continual-learning benchmarks report average accuracy, backward transfer, or forgetting matrices. CL-Bench instead measures per-system improvement over its own stateless baseline. This suggests a different emphasis: not retention on previously labeled tasks, but online learning value above a capability-matched baseline.

## 6. Agent architectures and empirical findings

CL-Bench evaluates several classes of agent architectures, including full-context ICL, ICL Notepad, Mem0, ACE, Claude Code, and Codex, paired with models such as Claude Opus 4.7, Claude Sonnet 4.6, Gemini 3.1 Pro, Gemini 3 Flash, and GPT-5.4 [2606.05661]. Full evaluations are averaged over 5 rollouts per task for most systems, while stateless baselines are evaluated once per task-system pair because order does not matter for stateless runs [2606.05661].

The headline empirical result is that naive full-context ICL is the strongest baseline. The top-ranked system is ICL + Claude Sonnet 4.6 with 22.3% normalized reward and 25.4% normalized gain at a cost of \$30.4, followed by ICL + GPT-5.4 with 20.1% normalized reward and 20.1% normalized gain [2606.05661]. Claude Code + Sonnet 4.6 ranks third with 19.0% normalized reward and 23.9% normalized gain [2606.05661]. Dedicated memory systems do not dominate: Mem0 + GPT-5.4 achieves 15.1% normalized reward and 20.2% normalized gain, while ACE + GPT-5.4 achieves only 4.6% normalized reward and 8.6% normalized gain despite the highest reported cost of \$62.8 [2606.05661].

The paper interprets these results as evidence that current frontier systems show only modest continual improvement and that specialized memory systems do not automatically solve continual learning. Systems often overfit to recent observations, fail to reactivate useful older information, or become rigid under drift [2606.05661]. The benchmark identifies domain-dependent behavior as well: Sales Prediction and Blind Spectrum Monitoring exhibit the clearest learning gains, Database Exploration shows strong gains partly because stateful systems maintain schema knowledge under drift, while Cohort Studies, Codebase Adaptation, and Poker show smaller or more inconsistent gains [2606.05661].

A notable benchmark-level claim is that cost does not reliably buy better continual learning. ICL + Gemini Flash achieves 16.4% gain at \$7.6, while ACE costs \$62.8 but reaches only 8.6% gain [2606.05661]. This suggests that, at least under current architectures, memory-management complexity may add overhead without translating into proportionally better online improvement.

## 6. Relation to broader continual-learning benchmarks and limitations

CL-Bench occupies a distinct place in the benchmark landscape. Traditional continual-learning benchmarks often center on class-, task-, or domain-incremental supervised learning over static datasets, including realistic temporal imagery [2201.06289], multimodal VQA [2211.10567], multimodal task sequences [2206.09059], generative image modeling [2406.04584], long-horizon multimodal and textual tuning [2505.09952], biomedical NLP [2603.16738], or graph-structured continual learning [2211.14568]. CL-Bench differs in measuring continual learning for stateful LLM-based agent systems in interactive environments rather than in standard supervised incremental settings [2606.05661].

This difference creates both strengths and limitations. A major strength is that CL-Bench targets realistic agent deployment scenarios in which experience accumulation should matter. A major limitation is that it does not directly report the standard CL metrics used in benchmark suites for supervised learning, so comparisons across benchmark families are not one-to-one. Another limitation acknowledged in the paper is that the benchmark currently covers only six domains and task horizons of tens of instances rather than the much longer horizons seen in some real deployments [2606.05661]. The initial evaluation also focuses on context-based memory paradigms rather than parametric adaptation methods such as test-time training [2606.05661]. Smaller or open-weight models are not evaluated, and full evaluations are expensive [2606.05661].

The benchmark’s strongest contribution is therefore methodological: it reframes continual-learning evaluation for frontier agent systems around online improvement over a capability-matched stateless baseline. The paper argues that current systems are far better at exhibiting pretrained capability than at learning robustly from ongoing experience [2606.05661]. That conclusion parallels a theme visible in other recent benchmark work: benchmark difficulty often shifts from pure anti-forgetting toward transfer, realistic structure discovery, order robustness, and knowledge accumulation under deployment-like constraints [2207.00010][2201.06289][2505.09952][2603.16738].

In that sense, CL-Bench is best understood not merely as another benchmark collection, but as a proposal for how continual learning should be evaluated in frontier LLM systems: through expert-validated, stateful, environment-specific tasks where improvement from experience can be isolated from raw model strength [2606.05661].

Source: https://www.emergentmind.com/topics/continual-learning-bench-cl-bench