CL-Bench: Continual Learning Benchmark

• Continual Learning Bench (CL-Bench) is a benchmark that evaluates whether LLM-based systems enhance performance through sequential, stateful experience in realistic environments.
• It organizes tasks into sequences of instances with inherent headroom, shared latent structures, and explicit learning mechanisms to gauge online improvement.
• Empirical findings show that while full-context in-context learning often performs strongly, specialized memory systems may not offer proportional advantages relative to their cost.

Searching arXiv for the benchmark and closely related papers to ground the article in current literature. Continual Learning Bench (CL-Bench) is a benchmark for evaluating whether frontier LLM-based systems improve through sequential experience in realistic, stateful environments. It is introduced as “the first difficult, expert-validated benchmark specifically designed to test whether frontier LLM-based systems genuinely improve through sequential experience” and is organized around environments in which tasks share a learnable latent structure that a stateful system can discover online but a stateless one cannot (Asawa et al., 4 Jun 2026). In the benchmark’s framing, continual learning is not merely long-context recall or memory retrieval, but the ability to use prior online experience to improve later performance in the same environment. CL-Bench spans six domains—software engineering, signal processing, disease outbreak forecasting, database querying, strategic game-playing, and demand forecasting—and evaluates both stateless and stateful agent architectures with a metric designed to isolate online learning from baseline model capability (Asawa et al., 4 Jun 2026).

1. Conceptual scope and benchmark motivation

CL-Bench is motivated by a gap between existing evaluations of language-model systems and the practical notion of continual learning required in long-lived deployments. The benchmark argues that static one-shot benchmarks primarily measure capabilities acquired during pretraining, while many long-context or memory evaluations focus on recall fidelity rather than on whether an agent actually improves through repeated interaction in the same environment (Asawa et al., 4 Jun 2026). The benchmark therefore targets a narrower but more deployment-relevant question: whether an LLM-based system performs better on future instances because it learned from prior instances in that same task.

The benchmark formalizes a distinction between stateless and stateful systems. For a sequence $x_1,\dots,x_N$, with history $\mathcal{H}_{t-1}=\{(x_i,r_i)\}_{i=1}^{t-1}$, a stateless system uses

$\pi_t = \pi(x_t)$

while a stateful system uses

$\pi_t = \pi(x_t, \mathcal{H}_{t-1}).$

This setup is intended to capture continual learning as online performance improvement due to sequentially accumulated experience rather than due to offline pretrained competence alone (Asawa et al., 4 Jun 2026).

A central design claim is that every CL-Bench task must satisfy three conditions: headroom, shared latent structure, and a learning mechanism. Headroom means that initial performance must be well below the maximum achievable score. Shared latent structure means that tasks contain reusable regularities such as codebase layout, schema conventions, opponent tendencies, or persistent forecasting factors. A learning mechanism means that earlier instances produce observations informative for later ones, allowing a stateful system to improve while a stateless one cannot (Asawa et al., 4 Jun 2026).

2. Formal task structure and stateful evaluation

CL-Bench defines a hierarchy of benchmark units. A task is a sequence of related instances. An instance is one problem within a task, and reward is defined at this level. A step is one action taken by the agent within an instance. A system is the full evaluated unit, including both the base model and any memory or adaptation machinery. A variant is a modified environment within a task, often used to induce concept drift. A schedule is the ordered sequence of instances a system processes, potentially spanning multiple variants (Asawa et al., 4 Jun 2026).

This structure is significant because the benchmark is explicitly designed around stateful environments rather than independent examples. The six domains are meant to mimic ongoing professional workflows in which later performance can improve if the system discovers environment-specific regularities. The benchmark further allows concept drift through variant changes, so the evaluation includes both retention and adaptation. This suggests that CL-Bench targets a stability–plasticity trade-off more directly than benchmarks centered only on static or iid task collections.

The benchmark uses structured outputs for automated scoring and, in some domains, persistent terminal or tool-based environments. Full evaluation covers approximately 300+ instances across the six tasks, with task lengths ranging from 12 to 120 instances (Asawa et al., 4 Jun 2026). This suggests that CL-Bench is neither a one-shot static benchmark nor a purely synthetic stream generator; it is a collection of curated sequential environments with explicit task horizons.

3. Benchmark domains and latent structure

The six benchmark domains are Blind Spectrum Monitoring, Codebase Adaptation, Cohort Studies, Database Exploration, Exploitable Poker, and Sales Prediction (Asawa et al., 4 Jun 2026). Each domain is intended to contain a recurring latent structure that can be inferred only through sequential interaction.

Blind Spectrum Monitoring models RF spectrum analysis over a 168 MHz band and 90 scans. The latent structure is the persistent set of channel center frequencies and bandwidths; transmitters may go dormant for long intervals, so later predictions benefit from remembering earlier detections. The schedule has three stages: scans 1–30 use five_ch_wide, scans 31–60 use five_plus_four_mixed, and scans 61–90 use full_grid_active (Asawa et al., 4 Jun 2026).

Codebase Adaptation presents 19 real Python bugfix issues drawn from two repositories, 9 from jazzband/tablib and 10 from jd/tenacity, each in a fresh Docker container at the issue’s base commit. The reusable structure is repository-specific: file organization, test invocation patterns, and conventions for patching. The task is blocked by repository, creating two local continual-learning phases (Asawa et al., 4 Jun 2026).

Cohort Studies casts the system as an epidemiological assistant analyzing five sequential fictional studies about Vardan-A Syndrome. The latent structure is a shared set of factors predictive of survival curves across studies, despite differing observed variables and sampling biases. The schedule contains 20 instances organized into five study blocks—HERALD, MERIDIAN, MOSAIC, FORGE, and CADENCE—each with four instances (Asawa et al., 4 Jun 2026).

Database Exploration asks 40 natural-language questions over an obfuscated SQLite database built from Amazon product reviews and metadata from Office Products, Electronics, and Musical Instruments. The reusable structure includes hidden schema conventions, unit conventions, and category-specific quirks. After question 20, a simulated schema migration changes the environment, introducing a benchmarked form of drift (Asawa et al., 4 Jun 2026).

Exploitable Poker consists of 120 hands of heads-up Texas Hold’em against deterministic opponent variants. The latent structure is the opponent’s fixed exploitable strategy, with repeated encounters used to test whether the agent forms and reuses an opponent model. The schedule alternates between calling_station, loose_aggressive, and fit_or_fold, with return stages to test retention and re-adaptation (Asawa et al., 4 Jun 2026).

Sales Prediction casts the system as a forecasting analyst for a furniture retailer with stores in San Francisco, New York, and Chicago. Each instance corresponds to a year, but only one store’s most recent year of history is visible at a time. The reusable structure includes stable growth rates, price-demand relationships, and product/location clusters. The schedule has 12 instances spanning years 2027–2038 in three stages of four instances each (Asawa et al., 4 Jun 2026).

The following table summarizes the six domains and their schedules.

Domain	Instances	Core latent structure
Blind Spectrum Monitoring	90 scans	persistent channel set
Codebase Adaptation	19 issues	repository-specific structure
Cohort Studies	20 instances	shared latent survival factors
Database Exploration	40 questions	hidden schema and unit conventions
Exploitable Poker	120 hands	deterministic opponent policy
Sales Prediction	12 instances	stable forecasting factors

A plausible implication is that CL-Bench deliberately mixes tasks where continual learning arises from very different mechanisms: environmental persistence, repeated latent-causal inference, repository familiarity, opponent modeling, and long-horizon forecasting. This suggests that the benchmark is not intended to reward a single memory heuristic, but to probe whether a system can extract and reuse structure across qualitatively different sequential settings.

4. Expert validation and task construction methodology

CL-Bench emphasizes expert validation as a defining property. Each task is first manually reviewed by two authors and then independently reviewed by 2–3 domain experts along three axes: realism, reusable knowledge, and learning improvement. Experts provided task-specific feedback that was used to refine task design. The paper reports that, on average, 2–3 experts spent 2.14 hours per validation cycle, with times ranging from 1 to 4.5 hours (Asawa et al., 4 Jun 2026).

This construction process distinguishes CL-Bench from synthetic continual-learning benchmarks and from benchmarks built by simple concatenation of independent tasks. In CL-Bench, the claim is not only that tasks are hard, but that they have been explicitly designed and checked so that sequential learning should matter. That design philosophy resembles broader benchmark efforts that aim to isolate realistic continual-learning factors in other modalities, such as subject-driven domain drift in physiological time series (Matteoni et al., 2022), long-horizon multimodal and textual continual tuning (Huai et al., 15 May 2025), realistic temporal distribution shift in imagery (Lin et al., 2022), or real-life context learning in messy human settings (Dou et al., 29 Apr 2026). CL-Bench differs in centering stateful agent interaction rather than standard supervised incremental prediction, but it shares the benchmark-level objective of making improvement from experience measurable rather than assumed.

5. Metrics: reward, gain, normalized gain, and stability–plasticity

CL-Bench uses task-specific reward functions, but its most distinctive metric is gain. If $r_t^{sf}$ is the reward of a system run statefully on instance $t$, and $r_t^{sl}$ is the reward of the same system run statelessly on the same instance, then per-instance gain is

$g_t = r_t^{sf} - r_t^{sl}$

and aggregate gain is

$g = \sum_{t=1}^{N} g_t.$

This is meant to isolate how much better the system performs because it had prior experience, rather than because it is intrinsically strong on the task (Asawa et al., 4 Jun 2026).

Because tasks have different reward scales, CL-Bench introduces normalized gain

$$\widehat{g} = \frac{\bar{r}^{sf} - \bar{r}^{sl}}{r_{max} - \bar{r}^{sl}}$$

where $\mathcal{H}_{t-1}=\{(x_i,r_i)\}_{i=1}^{t-1}$0 is a task-specific reference maximum and $\mathcal{H}_{t-1}=\{(x_i,r_i)\}_{i=1}^{t-1}$1, $\mathcal{H}_{t-1}=\{(x_i,r_i)\}_{i=1}^{t-1}$2 are average stateful and stateless rewards (Asawa et al., 4 Jun 2026). The benchmark also defines normalized reward

$\mathcal{H}_{t-1}=\{(x_i,r_i)\}_{i=1}^{t-1}$3

where $\mathcal{H}_{t-1}=\{(x_i,r_i)\}_{i=1}^{t-1}$4 is the stateless ICL reward of GPT-5.4 (Asawa et al., 4 Jun 2026). This separates two notions: absolute performance and improvement attributable to state.

The benchmark further decomposes normalized gain into stability and plasticity using variant boundaries. Let

$\mathcal{H}_{t-1}=\{(x_i,r_i)\}_{i=1}^{t-1}$5

where $\mathcal{H}_{t-1}=\{(x_i,r_i)\}_{i=1}^{t-1}$6 is the within-variant index for instance $\mathcal{H}_{t-1}=\{(x_i,r_i)\}_{i=1}^{t-1}$7. Then

$\mathcal{H}_{t-1}=\{(x_i,r_i)\}_{i=1}^{t-1}$8

and

$\mathcal{H}_{t-1}=\{(x_i,r_i)\}_{i=1}^{t-1}$9

with

$\pi_t = \pi(x_t)$0

The intended interpretation is that stability measures how well knowledge survives shifts across variants, while plasticity measures how well the system adapts within a variant (Asawa et al., 4 Jun 2026).

This metric design is a defining methodological contribution. Many continual-learning benchmarks report average accuracy, backward transfer, or forgetting matrices. CL-Bench instead measures per-system improvement over its own stateless baseline. This suggests a different emphasis: not retention on previously labeled tasks, but online learning value above a capability-matched baseline.

6. Agent architectures and empirical findings

CL-Bench evaluates several classes of agent architectures, including full-context ICL, ICL Notepad, Mem0, ACE, Claude Code, and Codex, paired with models such as Claude Opus 4.7, Claude Sonnet 4.6, Gemini 3.1 Pro, Gemini 3 Flash, and GPT-5.4 (Asawa et al., 4 Jun 2026). Full evaluations are averaged over 5 rollouts per task for most systems, while stateless baselines are evaluated once per task-system pair because order does not matter for stateless runs (Asawa et al., 4 Jun 2026).

The headline empirical result is that naive full-context ICL is the strongest baseline. The top-ranked system is ICL + Claude Sonnet 4.6 with 22.3% normalized reward and 25.4% normalized gain at a cost of \$\pi_t = \pi(x_t)$$162.8 (<a href="/papers/2606.05661" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Asawa et al., 4 Jun 2026</a>).</p> <p>The paper interprets these results as evidence that current frontier systems show only modest continual improvement and that specialized <a href="https://www.emergentmind.com/topics/memory-systems" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">memory systems</a> do not automatically solve continual learning. Systems often overfit to recent observations, fail to reactivate useful older information, or become rigid under drift (<a href="/papers/2606.05661" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Asawa et al., 4 Jun 2026</a>). The benchmark identifies domain-dependent behavior as well: Sales Prediction and Blind Spectrum Monitoring exhibit the clearest learning gains, Database Exploration shows strong gains partly because stateful systems maintain schema knowledge under drift, while Cohort Studies, Codebase Adaptation, and Poker show smaller or more inconsistent gains (<a href="/papers/2606.05661" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Asawa et al., 4 Jun 2026</a>).</p> <p>A notable benchmark-level claim is that cost does not reliably buy better continual learning. ICL + Gemini Flash achieves 16.4$$\pi_t = \pi(x_t)$262.8 but reaches only 8.6% gain (Asawa et al., 4 Jun 2026). This suggests that, at least under current architectures, memory-management complexity may add overhead without translating into proportionally better online improvement.

6. Relation to broader continual-learning benchmarks and limitations

CL-Bench occupies a distinct place in the benchmark landscape. Traditional continual-learning benchmarks often center on class-, task-, or domain-incremental supervised learning over static datasets, including realistic temporal imagery (Lin et al., 2022), multimodal VQA (Zhang et al., 2022), multimodal task sequences (Srinivasan et al., 2022), generative image modeling (Zhang et al., 2024), long-horizon multimodal and textual tuning (Huai et al., 15 May 2025), biomedical NLP (Zeng et al., 17 Mar 2026), or graph-structured continual learning (Ko et al., 2022). CL-Bench differs in measuring continual learning for stateful LLM-based agent systems in interactive environments rather than in standard supervised incremental settings (Asawa et al., 4 Jun 2026).

This difference creates both strengths and limitations. A major strength is that CL-Bench targets realistic agent deployment scenarios in which experience accumulation should matter. A major limitation is that it does not directly report the standard CL metrics used in benchmark suites for supervised learning, so comparisons across benchmark families are not one-to-one. Another limitation acknowledged in the paper is that the benchmark currently covers only six domains and task horizons of tens of instances rather than the much longer horizons seen in some real deployments (Asawa et al., 4 Jun 2026). The initial evaluation also focuses on context-based memory paradigms rather than parametric adaptation methods such as test-time training (Asawa et al., 4 Jun 2026). Smaller or open-weight models are not evaluated, and full evaluations are expensive (Asawa et al., 4 Jun 2026).

The benchmark’s strongest contribution is therefore methodological: it reframes continual-learning evaluation for frontier agent systems around online improvement over a capability-matched stateless baseline. The paper argues that current systems are far better at exhibiting pretrained capability than at learning robustly from ongoing experience (Asawa et al., 4 Jun 2026). That conclusion parallels a theme visible in other recent benchmark work: benchmark difficulty often shifts from pure anti-forgetting toward transfer, realistic structure discovery, order robustness, and knowledge accumulation under deployment-like constraints (Matteoni et al., 2022, Lin et al., 2022, Huai et al., 15 May 2025, Zeng et al., 17 Mar 2026).

In that sense, CL-Bench is best understood not merely as another benchmark collection, but as a proposal for how continual learning should be evaluated in frontier LLM systems: through expert-validated, stateful, environment-specific tasks where improvement from experience can be isolated from raw model strength (Asawa et al., 4 Jun 2026).