Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments

Abstract: Continual learning, the ability of AI systems to improve through sequential experience, has attracted substantial interest, but no high-quality benchmark exists to evaluate it. We introduce Continual Learning Bench (CL-Bench), the first difficult, expert-validated benchmark designed to measure whether LLM-based systems genuinely improve with experience. CL-Bench spans six diverse domains (software engineering, signal processing, disease outbreak forecasting, database querying, strategic game-playing, and demand forecasting), each validated by domain experts and designed so that tasks share a learnable latent structure (codebase layout, disease outbreak dynamics, opponent strategies) that a stateful system can discover online but a stateless one cannot. We evaluate frontier models across several agent architectures, from naive in-context learning (ICL) to dedicated memory systems, introducing a gain metric to isolate learning from prior capabilities. We find that these systems leave headroom for improved continual learning: agents frequently overfit to immediate observations or fail to reuse knowledge across instances, and dedicated memory systems do not fix this -- in fact, naive ICL outperforms systems dedicated to memory management. CL-Bench is the first benchmark to evaluate continual learning across diverse real-world domains with expert-validated tasks and isolate online learning from underlying model capability, showing a need for better continual learning systems.

• The paper introduces CL-Bench, a benchmark that evaluates LLMs' continual learning ability in realistic, nonstationary tasks by contrasting stateless and stateful performances.
• It employs dual metrics—reward and gain—to quantify both immediate task success and incremental improvements from learning, demonstrating performance across six varied domains.
• Results reveal that simple in-context learning outperforms specialized memory architectures, underscoring significant headroom for more robust, integrated continual learning approaches.

Continual Learning Bench: Rigorous Evaluation of Statefulness in Frontier LLM Systems

Motivation and Benchmark Design

The Continual Learning Bench (CL-Bench) is introduced as a robust, expert-reviewed evaluation suite specifically designed to probe the ability of LLM-based agents to improve performance via genuine state accumulation in real-world interactive environments (2606.05661). This work addresses existing shortcomings where prior continual learning evaluations are confounded by tasks that either inadvertently test static model capability or rely on artificial, skill-taxonomy-constrained domains. CL-Bench instead targets realistic domains with intrinsic, exploitable but nontrivial latent structure and possibilities for concept drift.

CL-Bench consists of six diverse domains:

• Software Engineering (codebase adaptation)
• Database Analytics (schema-discovery and migration)
• Epidemiological Forecasting (across multiple cohorts with shifting covariate structures)
• RF Spectrum Monitoring (persistent and drifted channel usage)
• Strategic Game-Playing (opponent modeling in Poker)
• Sales Forecasting (multi-site, multi-product prediction amidst schema and data drift)

All tasks satisfy three admission criteria:

1. Substantial headroom—stateless performance is well below optimum.
2. Hidden shared latent structure—information non-recoverable from pretraining, discoverable only via online experience.
3. Informative feedback loop—stimuli and reward enable experiential learning.

   Figure 1: The CL-Bench framework illustrated on a Database Exploration task. Agents progressively learn schema structure and must adapt when the underlying environment undergoes concept drift (e.g. schema migration mid-task).

Every task is meticulously validated by multiple domain experts for realism, knowledge transfer, and learnability, minimizing external knowledge confounds or artifactual solution shortcuts.

Figure 2: Task construction and human validation pipeline ensures that all benchmark tasks meet the required criteria, as confirmed by independent domain experts.

Evaluation Protocol and Novel Metrics

CL-Bench deploys a dual metric: reward (absolute task attainment) and gain (incremental improvement attributable to statefulness). Reward is simply the aggregate performance in the task-specific metric across all instances. Gain, in contrast, is the delta between stateful and stateless rollouts per step, directly isolating learning from prior experience versus static baseline ability. Both metrics are normalized across tasks using a principled scheme to enable aggregate system comparisons.

Figure 3: Gain metric isolates performance improvements that result specifically from learning through prior experience, controlling for instance difficulty and model capability.

Systematic Evaluation of Current Agent Architectures

A large-scale evaluation encompasses leading LLMs (Claude Opus 4.7, Claude Sonnet 4.6, Gemini 3 Pro/Flash, GPT-5.4) embedded in architectures including:

• Pure in-context learning (ICL)—full-history or notepad-augmented
• Structured memory systems (Mem0, ACE)
• Specialized agentic shells (Claude Code, Codex)

Performance across these systems highlights several robust, non-intuitive findings:

• Naive full-context ICL reliably outperforms dedicated memory architectures, both in normalized reward (22.3%) and gain (25.4% for Claude Sonnet 4.6 + ICL). Sophisticated memory curation, retrieval, or context compaction does not yield measurable advantage.
• High system cost does not translate to gain: expensive methods like ACE underperform even relative to simpler ICL at a fraction of the resource footprint.
• Task-level learning dynamics diverge: On domains such as Sales Prediction and Blind Spectrum Monitoring, clear online gain is present, while Cohort Studies and certain code-related tasks expose near-zero gain, indicating substantial headroom for algorithmic improvement and a lack of robust cross-task generalization.

  Figure 4: Per-task learning curves for the strongest system. The gap between stateful and stateless curves directly measures learning from experience. Effects vary notably across tasks.

  

  Figure 5: Pareto frontiers contrasting normalized gain and reward against system cost. The majority of efficient points are occupied by ICL variants rather than dedicated memory-augmented agents.

Decomposition: Plasticity vs. Stability in Learning

To further clarify learning behavior, CL-Bench introduces a decomposition of gain into plasticity (within-variant adaptation) and stability (cross-variant retention). The analysis reveals:

• Few systems demonstrate effective long-term stability; most performance gain is attributable to rapid adaptation after feedback rather than persistent memory of latent structure across nonstationary segments.
• Memory curation mechanisms (e.g., ICL Notepad) can marginally improve stability but at the cost of adaptation/plasticity, with no current system achieving simultaneously high marks in both.

  Figure 6: Decomposition of normalized gain illustrates the stability–plasticity trade-off across major agent architectures.

Task-Level Performance Insights

Analysis of per-task system outcomes demonstrates:

• Significant, early-formed learning gaps in Sales Prediction and Blind Spectrum Monitoring, where stateful agents overtly leverage cumulative feedback.
• Minimal improvement in codebase and cohort tasks, where effective integration of disparate experience across nonstationary backgrounds remains unsolved.

Detailed task-level curves—such as those for Database Exploration, Cohort Studies, and Software Engineering—further support the claim that current LLM systems do not generally possess robust, generalizable continual learning capability in a realistic setting.

Implications and Future Directions

CL-Bench provides strong evidence that statefulness and true continual learning in frontier LLMs remain underdeveloped. While context window expansion and naïve accretion are surprisingly effective, advanced memory, retrieval and summary systems do not yet realize the potential of structured, non-parametric, or meta-learned knowledge accumulation. The observed headroom suggests possible gains from:

• Hybrid parametric + non-parametric continual learning solutions
• Robust task-invariant memory/knowledge management architectures
• End-to-end trainable meta-learning and in-the-loop adaptation protocols tuned for long-horizon, non-stationary environments
• Tighter integration between reward-driven adaptation and semantic memory

The community-driven and extensible nature of CL-Bench, anchored by rigorous normative criteria for future task admission, supports its adoption as a canonical benchmark for future developments.

Conclusion

CL-Bench articulates a rigorous new standard for evaluating online learning and stateful behavior in LLM-based agents. In stark contrast to prior benchmarks, its expert-validated, task-agnostic framework definitively separates true continual improvement from base model proficiency or memory recall. Current systems—regardless of architectural sophistication or resource expenditure—fail to capitalize on headroom in statefulness, especially under non-stationary and knowledge-rich conditions. This positions CL-Bench as a crucial platform for tracking genuine algorithmic progress in continual learning for frontier-scale AI systems.