When Does Continual Learning Require Learning
Abstract: As LLMs become increasingly capable, the next question is how can we enable models to continually learn? Today, the field largely frames this as a problem of context management and mitigating forgetting. We argue this framing is incomplete: continual learning is fundamentally about increasing model competence as the world changes. We disentangle this change along two axes -- space, where the model encounters new domains, and time, where the underlying data drifts under a fixed task. This framing lets us study continual learning under realistic conditions: new domains arrive over time, facts drift past their training cutoff, and agentic interactions accumulate state across episodes. To evaluate methods under this setting, we recast widely used LLM benchmarks as sequential problems and introduce a single mechanism-agnostic protocol that compares prompt-based methods (GEPA, ACE), supervised learning (SFT, SDFT), reinforcement learning (GRPO, SDPO), and context compression (Cartridges, In-place TTT). Prompt-based methods fit each new stage quickly but degrade on future tasks. Distillation-based methods accumulate knowledge stably but struggle to update outdated facts. Context compression improves efficiency without substantially improving the ability to learn new tasks. Online reinforcement learning adapts most effectively to knowledge updates but remains sensitive to noisy reward signals. Overall, our results suggest that continual learning is not a single capability: different patterns of environmental change require fundamentally different update behaviors, determining when adaptation must be learned inside model weights and when it can be achieved through external scaffolding. We hope that understanding where each method succeeds and fails will guide the design of stronger continual learning systems.
Summary
- The paper’s main contribution is introducing a unified framework that disentangles spatial and temporal changes in continual learning for LLMs.
- It rigorously evaluates update methods—prompt-based, parametric, and context compression—highlighting tradeoffs between rapid adaptation and long-term stability.
- Empirical results reveal that no single method is universally optimal, underscoring the need for regime-aware, compositional strategies in practical deployments.
Authoritative Summary of "When Does Continual Learning Require Learning" (2607.07847)
Unifying Continual Learning: Framework and Axes of Environmental Change
"When Does Continual Learning Require Learning" (2607.07847) presents a systematic framework for evaluating and understanding continual learning within LLMs. The paper challenges the prevailing focus on context management and catastrophic forgetting, arguing that the central objective of continual learning is the sustained increase of model competence as the environment evolves.
The authors introduce a unified protocol that is agnostic to the update mechanism—prompt-based, parametric, or architectural—enabling controlled comparisons. Central to the framework is the disentanglement of change along two axes: spatial (acquisition of new domains/tasks) and temporal (adaptation as the distribution or knowledge base drifts over time). Temporal changes are further decomposed into slow trends, discrete fact updates, and agentic state accumulation, covering a wide spectrum of realistic deployment scenarios for LLMs.
Figure 1: Continual learning methods and environmental changes are organized along a unified framework, distinguishing update loci (context, parameters, memory/state) and axes of task evolution (domain shift, fact update, temporal drift, agentic state).
Evaluation Protocol: Mechanism-Agnostic Sequential Tasking
The paper operationalizes continual learning evaluation by recasting canonical LLM tasks as multi-stage sequential benchmarks, ensuring each method is evaluated under uniform protocol constraints, such as compute budget and stage delineation. Updates are classified as:
- Prompt-based (GEPA, ACE): Optimization or manual evolution of prompts and playbooks carried across stages, altering context but not core weights.
- Parametric (SFT, SDFT, GRPO, SDPO): Offline supervised or online RL-based weight updates, with some variants incorporating self-distillation or preference-based learning.
- Context Compression (Cartridges, In-place TTT): Augmentation of the model with external key-value adapters or per-input fast-weight updates, decoupled from core parameter modifications.
A core contribution is the adoption of forgetting matrices and well-defined forward/backward transfer metrics to quantitatively surface the stability-acquisition tradeoff across methods.
Empirical Findings Across Axes: Tradeoffs and Method Behaviors
Domain Shift (Space Axis)
Prompt-based methods (GEPA, ACE) demonstrate rapid adaptation to new domains but are highly susceptible to catastrophic forgetting—knowledge of prior tasks is quickly lost when optimizing for subsequent tasks. In contrast, distillation-based approaches (SDFT, SDPO) accumulate knowledge more stably and exhibit less forgetting but at the cost of slower adaptation and less flexibility to absorb new domain-specific structure.
Figure 2: Per-method accuracy trajectories across sequential domain shifts; rapid fit and severe forgetting in prompt-based methods, steadier accumulation in distillation-based methods.
Discrete Knowledge Updates (Fact Change)
In settings requiring targeted updates (e.g., Wikipedia factual drifts), the tension between stability and plasticity is pronounced. Stability-anchored methods (SDFT, SDPO) struggle to incorporate new discrete facts without corrupting existing knowledge, evidencing what the authors term “catastrophic memorizing.” In contrast, reinforcement learning (GRPO) achieves improved adaptation to new knowledge while maintaining performance on stable facts, underscoring the importance of the learning signal in update selectivity.
Figure 3: F1 scores over temporal knowledge slices for each method, illustrating the catastrophic memorizing phenomenon—gain in drift facts at the expense of stable knowledge.
Temporal Drift (Slow, Noisy Trends)
In the context of financial sentiment analysis on annual 10-K filings, distillation-based and context-compression methods (Cartridges, In-place TTT) maintain forward transfer and resist overfitting to transient regime-specific patterns. Prompt and RL methods struggle to sustain future-task generalization, often overfitting to short-term signals with poor extrapolation.
Figure 4: Backward and forward accuracy across fiscal years; distillation and compression methods maintain future generalization while prompt and RL methods degrade.
Prompt evolution analysis further reveals that current prompt optimizers, such as GEPA, trend towards overfitting on shallow heuristics rather than extracting robust, transferable insight.
Figure 5: Prompt evolution on the 10-K sentiment task—optimizers add static, generic heuristics rather than evolving nuanced, time-aware reasoning.
Agentic State Accumulation
In agentic settings where a model’s environment evolves as a function of its own actions (e.g., browser automation tasks), both context (ACE playbook) and supervised fine-tuning (SFT) approaches yield measurable improvements over zero-shot baselines across all evaluated chain lengths, but absolute performance decays as the dependency horizon grows.
Figure 6: End-to-end agentic task success rates as a function of chain length; both ACE and SFT improve over baselines, with continuous decay at longer horizons.
Synthesis: No Single Method is Universally Optimal
The empirical analysis exposes robust tradeoffs:
- Prompt-based strategies facilitate rapid stagewise adaptation but are fundamentally limited in long-term knowledge accumulation due to context bottlenecks and severe forgetting.
- Parametric updates via distillation offer stability and accumulation but hinder the incorporation of discrete, time-sensitive knowledge.
- Context compression techniques primarily benefit efficiency and memory management with minimal impact on learning or retention.
- Reinforcement learning-based methods best support fine-grained factual updates but are fragile in the presence of noisy reward signals.
These findings operationalize the claim that continual learning is not a monolithic capability—adaptation to different environmental regimes demands distinct update mechanisms.
Implications and Future Directions
This work reorients the design and evaluation of continual learning in LLMs away from a singular emphasis on forgetting, towards a regime-aware, protocol-driven taxonomy of methods and behaviors. Practically, the results suggest that deployed LLM systems require compositional, context-sensitive strategies for adaptation: for some shifts, in-context prompt evolution suffices; for others, stable parametric updating is essential for competence accumulation; and for discrete or agent-driven changes, RL-based learning provides superior specificity.
From a theoretical standpoint, the framework facilitates principled investigations into the interplay between plasticity, stability, update locality, and compute constraints. The empirical benchmarks and protocol introduced in this paper also constitute a foundation for benchmarking future LLM continual learning methods in more realistic, open-ended environments, stimulating work on algorithmic evaluation and capability decomposition.
Further research directions include the integration of dynamic regime-detection mechanisms, the development of hybrid update strategies with regime-specific triggers, and extending the framework to larger-scale models and multi-agent contexts.
Conclusion
"When Does Continual Learning Require Learning" contributes a robust analytical and empirical structure for evaluating continual learning in LLMs. It demonstrates that no single update method dominates across the entire spectrum of changing environmental conditions. The nuanced, regime-aware findings underscore the necessity of adaptive, mechanism-diverse systems for real-world LLM deployment, and sets a methodological standard for future continual learning research.
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper asks a simple question: how can we help AI models keep getting better as the world changes? The authors argue that “continual learning” isn’t just about remembering old stuff—it’s about improving skills over time while handling new topics, updated facts, and changing situations.
What questions did the researchers ask?
The team focused on a few everyday-sounding questions:
- When the AI sees totally new topics (like switching from biology to finance), can it learn the new stuff without forgetting the old?
- When facts change over time (like a new team wins the championship), can it update just those facts without messing up everything else?
- When the world shifts slowly (like economic trends), can it absorb useful patterns without chasing random noise?
- When the AI acts in apps (like using Gmail in a browser), can it use experience from earlier steps to do better at later steps?
How did they study it?
They built a fair, shared way to test different “update styles” for improving an AI model over a series of stages—like levels in a game—so each method faces the same tasks in the same order.
To make it realistic, they considered two kinds of change:
- Space (new domains): The AI is asked to handle completely different tasks one after another (e.g., tools → finance → biology).
- Time (changing world): The task stays similar, but the underlying facts or data shift. They split this into:
- Slow drift (trends over years, like finance reports).
- Discrete fact updates (e.g., Wikipedia facts that get edited).
- Agentic accumulation (the AI’s own actions change the environment, like steps in a web app building on each other).
They recast well-known tasks into “sequences” and tried four families of methods on the same base model (Qwen3-8B):
- Prompt-based methods: Change the instructions you give the model (like editing a playbook or a prompt).
- Supervised learning (fine-tuning): Retrain the model’s “brain” using labeled examples.
- Including self-distillation: regularize learning using the model’s previous self as a guide.
- Reinforcement learning: Improve via trial-and-error with rewards for correct behavior.
- Context compression/memory: Keep or compress external notes/adapters without changing the main model.
Think of it like this:
- Prompt-based = editing the model’s “cheat sheet.”
- Fine-tuning = updating the model’s “brain.”
- Distillation = learning while staying close to what you already knew.
- Reinforcement learning = learning from rewards/punishments.
- Context compression = organizing your notebook better, not your brain.
They tested these methods on four kinds of sequential tasks:
- Switching domains (tools → finance → biology).
- Wikipedia facts changing month to month.
- Finance reports (10-Ks) across years with noisy signals.
- Multi-step web actions (like renaming and applying a Gmail label) where each step depends on the previous ones.
What did they find, and why is it important?
Here are the big patterns they saw:
- Prompt-based methods (e.g., GEPA, ACE)
- What happens: They adapt super fast to the current stage, but often forget or harm future performance.
- Why it matters: Great for quick fits, risky over time. Like cramming for a test with hacks that don’t generalize.
- Distillation-based fine-tuning (e.g., SDFT, SDPO)
- What happens: They keep knowledge stable over time and often avoid forgetting, but they’re slow to update when facts change.
- Why it matters: Safe and steady—good for long-term trends, not great for rapid updates.
- Context compression/memory (e.g., Cartridges, In-place test-time training)
- What happens: Better organization and efficiency, but they don’t really learn new skills on their own.
- Why it matters: Useful as support tools, not as the main way to adapt.
- Reinforcement learning (e.g., GRPO)
- What happens: Best at learning when facts genuinely change (like Wikipedia updates), but can fail when rewards are noisy (like in finance predictions).
- Why it matters: Powerful for clear, checkable goals; unstable when feedback is messy.
- Agentic setting (web app chains)
- What happens: Both a good playbook (prompt-based) and supervised fine-tuning let experience compound—models do better across steps than without learning. But success still drops as chains get longer.
- Why it matters: Realistic, because your past actions affect the future. Both “edit the notes” and “update the brain” help here.
Why this matters overall:
- There isn’t one “best” way to continually learn. Different changes in the world need different update strategies.
- Sometimes you must change the model’s weights (its internal knowledge). Other times, you can adapt by changing prompts or organizing context better.
- Picking the right tool for the type of change is crucial.
What’s the impact of this research?
This work gives a practical roadmap for building AI systems that stay useful over time:
- Use fast prompt edits for quick adaptation—but don’t rely on them for long-term stability.
- Use distillation-style fine-tuning to carry knowledge forward when data is noisy or slow-changing.
- Use reinforcement learning when you can verify results clearly and want to adapt to real changes.
- Use context/memory tools to make things efficient and reusable, but pair them with real learning when new skills are needed.
- In interactive systems (like agents), combine strategies: structured prompts plus learned behavior gave the best compounding gains.
In short: Continual learning isn’t one skill—it’s a toolbox. Knowing which tool to use as the world changes makes AI systems more reliable, up-to-date, and useful in real life.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
The paper introduces a useful unifying protocol and reveals important trade-offs, but several aspects remain unexplored or uncertain. The following concrete gaps can guide future work:
- Limited backbone diversity: results are primarily on Qwen3-8B (non-thinking) with some agentic tests on Qwen-32B; it is unclear how conclusions transfer to larger models, reasoning-enabled variants, or different families (e.g., Llama, Mistral, GPT).
- Order sensitivity not assessed: domain experiments use a single task order (ToolUse → FinQA → SciKE-Bio); how outcomes change under task reordering or interleaving remains unknown.
- Short horizons in most settings: domain chain has only 3 stages and TempWiki uses 4 monthly snapshots; scalability to long-horizon continual learning (dozens/hundreds of stages) is not evaluated.
- Compute/latency/storage trade-offs are not measured: while a per-stage budget
Cis held constant, the paper does not report actual compute, wall-clock, memory growth (e.g., playbook and cartridge size), or inference overhead, limiting fair cost–benefit comparisons. - Method-appropriate data formatting may bias comparisons: in TempWiki, weight/RL methods train on 500 (Q/A) drift triples while compression methods ingest full articles. The impact of differing supervision granularity and text volume on outcomes is not quantified.
- No replay/rehearsal baselines: classic continual-learning techniques (e.g., replay buffers, GEM/EWC-style regularizers, parameter isolation, or LoRA-based adapters per task) are not evaluated or adapted for LLMs under the unified protocol.
- Fact-editing baselines absent: targeted knowledge-editing methods (e.g., ROME, MEMIT, SERAC variants) are not compared on the discrete fact-update axis, leaving unclear whether specialized editors outperform RL/SFT in avoiding “catastrophic memorizing.”
- Hybrid strategies underexplored: the potential of combining methods (e.g., prompt evolution + distillation, RL gated by RAG/memory, or compression + PEFT) is not studied despite complementary strengths across regimes.
- Automatic regime detection is missing: there is no mechanism to decide when to write updates into weights vs. scaffolding vs. memory based on observed drift patterns; designing such a meta-controller remains open.
- Reward noise characterization is limited: RL is reported as “sensitive to noisy reward signals,” but the paper lacks systematic ablations varying reward noise, reward shaping, or group-size effects in GRPO/SDPO, especially on 10-K and TempWiki.
- Agentic RL is not tested: the agentic suite evaluates ACE and SFT but not online RL, leaving open whether RL can better exploit trajectory-level credit assignment in stateful environments.
- Context growth and truncation effects are unmeasured: agentic chains keep a single session alive, but the impact of long histories (context window limits, summarization/compression quality) on performance is not quantified.
- Overlapping-structure domains not evaluated: the domain chain deliberately uses dissimilar tasks; forward transfer under partially overlapping skills or shared formats (a common real-world case) is not assessed.
- Calibration and uncertainty are not measured: whether methods maintain or degrade confidence calibration, and whether uncertainty can guide when/how to update, is unexamined.
- Data quality and label noise effects are unclear: noisy labels in 10-K sentiment and possible diff noise in TempWiki are acknowledged but not systematically varied; robustness to noise and outliers is not characterized.
- No standardized competence metrics beyond accuracy/F1: improvements are judged by accuracy and
BWT/FWT, but complementary measures (sample efficiency, time-to-adapt, stability-plasticity ratios, calibration, out-of-distribution generalization) are not reported. - External knowledge integration is narrow: retrieval-augmented methods and persistent external knowledge bases (beyond Cartridges/in-place TTT) are absent, leaving open how RAG or learned memory can handle temporal updates without weight edits.
- Safety/alignment drift not studied: how continual updates affect harmful behavior, objective misgeneralization, or safety constraints over time is not evaluated.
- Catastrophic memorizing diagnosis is coarse: TempWiki uses drift vs. stable F1, but does not localize which beliefs were overwritten, whether errors cluster by relation types, or how edits propagate to related facts.
- Teacher quality and distillation dynamics are underexplored: SDFT/SDPO use previous-stage models as teachers; effects of stronger teachers, teacher ensembles, or curriculum strategies on stability vs. plasticity trade-offs remain open.
- PEFT variants under-tested: beyond full SFT and Cartridges, parameter-efficient tuning (LoRA/adapters) for sequential updates is not benchmarked, leaving unknown their sweet spot between stability, plasticity, and cost.
- Online evaluation protocols missing: updates happen between stages; continual evaluation within streams (interleaved train/test, drifting distributions without clear boundaries) is not examined.
- Continual learning in multimodal settings is absent: the framework and benchmarks are text-only; how conclusions extend to vision, audio, or web-UI grounding remains untested.
- Storage growth and lifecycle management of external artifacts: how playbooks, cartridges, and preference pools scale over long horizons and how to prune/garbage-collect them while preserving performance is not addressed.
- Task- and data-selection policies are static: strategies for selective sampling, prioritizing high-value updates, or detecting when not to update (to avoid overfitting to transient regimes) are not investigated.
- Rollback and auditability of updates: mechanisms to revert harmful updates, track provenance of changes (prompt vs. weight vs. memory), and ensure reproducible behavior are not considered.
- Generalizability to real deployment constraints: privacy, user personalization, non-stationary multi-tenant workloads, and compute-constrained on-device settings are outside the current protocol and left for future work.
Practical Applications
Below are concrete, real-world applications suggested by the paper’s findings and methods. Each item names target sectors, the actionable use case, potential tools/workflows that could emerge, and key assumptions or dependencies.
Immediate Applications
- Continual-learning evaluation and governance harness
- Sectors: Software/AI platforms, Enterprise MLOps, Academia, Policy
- Use case: Adopt the paper’s mechanism-agnostic protocol (sequential stages, forgetting matrices with FWT/BWT) to audit and A/B test prompt, weight, RL, and compression updates before deployment.
- Tools/workflows: “CL dashboard” that renders FWT/BWT heatmaps per release; per-stage compute budgets; sequentialized versions of in-house benchmarks; gates that block rollout if backward transfer drops below thresholds.
- Assumptions/dependencies: Access to representative sequential datasets; reproducible training seeds; logging of update operators; manageable compute budgets.
- Regime-aware update playbook for enterprise AI systems
- Sectors: Software, Enterprise AI/LLMOps
- Use case: Operationalize a decision tree that maps change types to update methods:
- Discrete verifiable fact updates → online RL (GRPO-style)
- Slow, noisy drift → self-distillation (SDFT) or Cartridges
- New domains (format/style changes) → SDFT/SDPO (accumulate)
- Short-lived tasks/agentic recipes → prompt/playbook methods (ACE/GEPA)
- Tools/workflows: Update routing service that tags incoming changes as “domain shift,” “slow trend,” “fact update,” or “agentic state,” then triggers the appropriate operator with a fixed per-stage budget.
- Assumptions/dependencies: Basic regime detection (rule-based or simple heuristics); access to each update operator; monitoring to roll back if forward transfer collapses.
- Fact-refresh pipelines with verifiable rewards
- Sectors: Knowledge bases, Search, News/Media, E-commerce catalogs, Healthcare guidelines (structured changes)
- Use case: Build ingestion jobs that detect diffs (e.g., Wikipedia infobox changes, product prices, drug label updates) and apply online RL (GRPO) to update beliefs while preserving stable knowledge.
- Tools/workflows: Diff scrapers; verifiers that can label correctness (e.g., exact-match against updated triples); small-batch RL with tight KL constraints; tests on stable-fact probes to prevent “catastrophic memorizing.”
- Assumptions/dependencies: Availability of clean, verifiable rewards; guardrails for noisy or adversarial diffs; automated rollback if stable-fact F1 drops.
- Temporal-drift resilient modeling via self-distillation
- Sectors: Finance (10-K analysis, risk monitoring), Compliance, Operations analytics
- Use case: For weak, noisy, slowly evolving signals (e.g., year-over-year documents), favor SDFT or Cartridges to accumulate transferable structure and avoid overfitting to the latest slice.
- Tools/workflows: “Previous checkpoint as teacher” training loops; rolling-window retraining; forward-transfer scorecards that track future-year accuracy; light Cartridge adapters for rapid refresh.
- Assumptions/dependencies: Labeled or weakly labeled streams; stable teacher quality; compute to run periodic distillation; drift detection for retrain cadence.
- Agentic web/RPA improvements with playbooks and distilled traces
- Sectors: RPA, Enterprise productivity, E-commerce ops, Customer support tooling
- Use case: Increase multi-step web task success by maintaining ACE playbooks (procedural guidance) and/or fine-tuning with successful teacher traces; persist session state to compound experience across steps.
- Tools/workflows: Headless browser harness with per-step verifiers; playbook editor (add/keep/drop); trace distillation jobs for smaller agents; per-chain budgets and decay-aware planning.
- Assumptions/dependencies: Programmatic verifiers; reliable browser automation; session persistence; teacher agents or human-verified traces.
- Context compression for long-doc efficiency
- Sectors: Finance (filings), Legal (briefs), Healthcare (EHR summaries), Education (course packs)
- Use case: Use Cartridges to store reusable long-context representations to cut inference cost and latencies on long documents without expecting large new capability gains.
- Tools/workflows: Cartridge training from teacher traces; registry of document-level cartridges; usage policies that swap in cartridges at retrieval time.
- Assumptions/dependencies: High-quality teacher outputs; stable long-context APIs; evaluation to ensure no loss on key tasks.
- CL-aware model governance and compliance
- Sectors: Policy, Regulated industries (Finance, Healthcare, GovTech)
- Use case: Track and report update effects using FWT/BWT and per-stage matrices; require different evidence standards for prompt vs weight vs RL updates.
- Tools/workflows: Update provenance logs (operator, data slice, budget); “forgetting audits” before go-live; alarms for negative backward transfer.
- Assumptions/dependencies: Organizational buy-in; standardized reporting formats; ability to snapshot/compare models across stages.
- Academic benchmark standardization
- Sectors: Academia, Open-source community
- Use case: Release sequentialized versions of popular benchmarks and evaluate methods on the common protocol for fair comparisons across update mechanisms.
- Tools/workflows: Open-source repos implementing the protocol; baseline scripts for GEPA/ACE/SFT/SDFT/GRPO/SDPO/Cartridges/TTT; leaderboards reporting FWT/BWT.
- Assumptions/dependencies: Stable data licenses; compute grants; community adoption.
- Personal productivity agents with safe “experience compounding”
- Sectors: Daily life, SMBs
- Use case: Email/calendar/file-management assistants that keep a playbook of successful strategies (ACE) and optionally fine-tune on confirmed-good traces to maintain performance across longer task chains.
- Tools/workflows: Opt-in trace logging; human verification UI; simple rollback to base model when errors persist; per-user playbook management.
- Assumptions/dependencies: Privacy controls; user consent; on-device or secure storage for traces.
- Risk controls for RL in noisy environments
- Sectors: Software, Finance, Ops automation
- Use case: Introduce reward-quality gates and fallback policies (e.g., revert to SDFT) when reward signals are weak; constrain RL with KL penalties and small compute budgets.
- Tools/workflows: Reward diagnostics; confidence-weighted updates; canary evaluation on stable probes; automated rollback.
- Assumptions/dependencies: Telemetry on reward noise; implementation of conservative constraints; human oversight for critical updates.
Long-Term Applications
- Auto-orchestration of update operators (meta-continual learner)
- Sectors: Software, AI platforms
- Use case: A controller that diagnoses change regime in real time and selects between prompting, self-distillation, RL, cartridges, or no-op; learns when to commit to weights versus external scaffolding.
- Tools/products: “CL orchestrator” microservice integrated into LLMOps; meta-learning policies trained on historical sequences; cost–risk trade-off optimizer.
- Assumptions/dependencies: Robust regime detection; diverse training corpora; reliable reward estimation; strong model observability.
- Fine-grained, locality-preserving weight editing for factual changes
- Sectors: Knowledge bases, Search, Enterprise knowledge management
- Use case: Methods that rewrite specific beliefs while provably preserving stable knowledge, avoiding “catastrophic memorizing.”
- Tools/products: Fact-localized weight-editing algorithms, knowledge-graph alignment and verification; differential tests on stable-fact suites.
- Assumptions/dependencies: Better causal localization of facts in weights; scalable verification; compatibility with PEFT/LoRA-like adapters.
- Robust RL under noisy or weak rewards
- Sectors: Finance, Operations, Robotics, Education
- Use case: RL variants that denoise reward signals (e.g., uncertainty-aware advantages, learned reward models) to safely adapt under drift without reinforcing spurious patterns.
- Tools/products: Noise-robust advantage estimators; preference-model ensembles; off-policy safety layers; integrated confidence calibration.
- Assumptions/dependencies: High-quality preference data; off-policy evaluation; computational budget for ensembles.
- Regulatory certification suites for continual learners
- Sectors: Policy, Regulated industries
- Use case: Standardized sequential tests and documentation requirements (FWT/BWT, provenance, rollback plans) for certifying LLM updates in sensitive domains.
- Tools/products: Compliance testbeds that synthesize domain shift, temporal drift, and fact changes; audit APIs; third-party verification services.
- Assumptions/dependencies: Regulatory consensus; public–private collaboration; secure audit mechanisms.
- Agentic systems with state-aware planning and memory
- Sectors: RPA, Robotics, Enterprise workflows
- Use case: Agents that explicitly model and track environment state across episodes, plan over longer horizons, and compound experience without collapse at long chain lengths.
- Tools/products: Persistent state stores; cross-episode memory architectures; chain-length-aware curricula; verifier-driven feedback loops.
- Assumptions/dependencies: Reliable state introspection; long-horizon credit assignment; scalable memory without context collapse.
- Sector-specific continual learners
- Healthcare: Guideline and formulary updates (fact updates via robust RL) + longitudinal patient trend modeling (self-distillation for slow drift).
- Energy: Demand forecasting under slow trends (SDFT) with discrete operator rule changes (localized updates).
- Cybersecurity: Threat intelligence ingestion with verifiable indicators (RL for facts) plus baseline classifiers preserved via distillation.
- Legal/Policy: Statute and case-law trackers that integrate discrete changes (weight edits/RL) and preserve doctrine structures (distillation).
- Tools/products: Domain-tuned orchestrators; evidence-linked update logs; hybrid RAG+CL stacks with cartridges for long references.
- Assumptions/dependencies: Access to structured change feeds; sector-specific verifiers; strict privacy and compliance frameworks.
- Education: curriculum- and semester-aware tutors
- Sectors: EdTech
- Use case: Tutors that preserve stable core knowledge while adapting to new curricula and textbooks each term, maintaining forward transfer to future modules.
- Tools/products: Semester cartridges; self-distillation across course iterations; per-module FWT/BWT dashboards for instructors.
- Assumptions/dependencies: Labeled curricular changes; student privacy; reliable assessment signals.
- Foundation-model lifecycle management with cartridges and registries
- Sectors: AI platforms, Enterprise IT
- Use case: “Cartridge registry” for reusable long-context representations (e.g., product catalogs, policy manuals), with dependency tracking and hot-swapping at inference.
- Tools/products: Versioned cartridge store; compatibility tests; dependency graphs; rollout/rollback pipelines.
- Assumptions/dependencies: Standardized cartridge formats; teacher availability for distillation; CI/CD for inference graphs.
- Auditable update provenance ledger
- Sectors: Policy, Enterprise, Open-source
- Use case: A ledger that records every prompt, weight, RL, or memory update, linked to its data slice and evaluation outcomes; supports incident forensics and compliance.
- Tools/products: Immutable logs, SBOM-like “Model BOM” for updates, diff viewers for prompts/playbooks, integrated FWT/BWT snapshots.
- Assumptions/dependencies: Secure storage; organizational policy; adoption of common schemas.
- Multi-agent knowledge sharing via playbooks and distilled skills
- Sectors: Enterprise automation, Customer support, DevOps
- Use case: Teams of agents exchange ACE playbooks and distilled adapters to propagate best practices without brittle prompt sprawl.
- Tools/products: Playbook marketplaces; adapter-sharing protocols; collective evaluation harnesses for shared procedures.
- Assumptions/dependencies: Quality control for shared assets; compatibility across model families; governance over propagation.
These applications rely on the paper’s central insight: continual learning is not a single capability. Choosing the right update mechanism depends on whether the environment presents domain shifts, slow temporal drift, discrete fact changes, or agent-induced state changes. Systems that operationalize this insight can deploy more reliable, efficient, and auditable AI in practice.
Glossary
- ACE (Agentic Context Engineering): A prompt-based method that maintains and evolves a structured “playbook” to guide model behavior across tasks. "ACE \citep{zhang2025agentic} maintains a markdown playbook edited incrementally through explicit add/keep/drop operations, which the authors show resists brevity-bias and context collapse."
- activation editing: Modifying model activations using learned vectors to adapt behavior without full retraining. "Activation editing \cite{ilharco2022editing} uses task vectors applied to all parameters."
- adapter-based tuning: A parameter-efficient fine-tuning approach that adds small adapter modules to a frozen backbone. "More commonly, work studies parameter-efficient methods, such as LoRA \cite{hu2022lora} and adapter-based tuning \cite{lin2025continual,lopez2017gradient}, which restrict updates to small subsets of parameters."
- agentic accumulation: A temporal regime where the environment’s state changes as a direct consequence of the agent’s own prior actions. "A third sub-regime is agentic accumulation: the task family itself stays fixed (e.g., actions inside a web app), but the environmentâs state drifts as a direct consequence of the modelâs own prior actions, so that what one step leaves behind becomes part of the task the model faces at the next step."
- agentic state: The evolving environment state shaped by an agent’s actions, which in turn influences subsequent tasks. "We identify four axes along which tasks evolve: domain shift, fact update, temporal drift, and agentic state."
- BWT (Backward Transfer): A metric that quantifies how training on later stages affects performance on earlier tasks (negative indicates forgetting). "BWT \citep{lopez2017gradient} measures whether later stages improve or damage earlier ones (negative BWT is catastrophic forgetting)."
- Cartridges: A context-compression technique that learns small, stage-specific components while freezing the backbone. "Cartridges \citep{eyuboglu2025cartridges} freezes the backbone and learns a small per-stage component, distilled from a teacher trace and stitched in at inference."
- catastrophic forgetting: The loss of previously learned capabilities when learning new tasks sequentially. "Traditionally, continual learning has been defined as mitigating catastrophic forgetting \cite{mccloskey1989catastrophic}:"
- catastrophic memorizing: Failing to update only the necessary knowledge during factual changes, leading to corrupted stable knowledge. "The failure to do so is what we call 'catastrophic memorizing'."
- context compression: Techniques that compress or structure context to make large inputs efficient without necessarily improving learning new tasks. "Context compression improves efficiency without substantially improving the ability to learn new tasks."
- continual learning: The process of improving a model’s competence over time as its environment or tasks change. "We define continual learning as the problem of increasing competence as the world changes."
- domain shift: Changes in the input distribution requiring new skills without forgetting old ones. "We identify four axes along which tasks evolve: domain shift, fact update, temporal drift, and agentic state."
- forgetting matrix: An evaluation matrix measuring performance on all tasks after each training stage to assess forgetting and transfer. "The task chain produces a forgetting matrix that answers \"after seeing tasks $1...i$, how well does the model still do task ?\""
- forward-KL loss: A divergence-based objective that encourages the student model to match a teacher’s distribution (forward direction). "SDFT \citep{shenfeld2026self} is self-distillation under a forward-KL loss against soft targets from a teacher"
- FWT (Forward Transfer): A metric for how earlier training stages improve performance on future, unseen stages. "FWT measures whether prior stages prepare the model for stages it has not yet trained on, and is the sharper signal: it tests acquisition of structure that transfers, rather than fitting the current stage at the cost of the rest."
- frame problem: The challenge of determining which beliefs should change when the environment updates. "This second sub-regime is the LLM analogue of the frame problem \cite{mccarthy1981some}: when the world updates, a learner must decide which of its beliefs are now outdated and which can remain fixed."
- GEPA: A prompt-evolution method that iteratively mutates and selects prompts based on validation performance. "GEPA \citep{agrawal2025gepa} optimizes the system prompt through a reflect-and-mutate loop scored on a validation minibatch; the best-scoring mutations propagate forward."
- GRPO: A reinforcement learning method using grouped rollouts and relative advantages to update the policy under a KL constraint. "GRPO \citep{shao2024grpo} samples rollouts in groups, computes within-group advantages from a verifiable reward, and updates the policy under a KL constraint."
- headless Chromium: A browser environment without a graphical interface used for automated agent interaction and evaluation. "on top of a browser-use agent driving a headless Chromium against the live app."
- In-place TTT (In-Place Test-Time Training): A method that performs input-conditioned weight updates during inference and resets between inputs. "In-place TTT \citep{feng2026place} performs an input-conditioned weight update at inference using a self-supervised auxiliary loss, reset between inputs."
- KL constraint: A regularization that limits policy updates by bounding the Kullback–Leibler divergence from a reference. "GRPO \citep{shao2024grpo} samples rollouts in groups, computes within-group advantages from a verifiable reward, and updates the policy under a KL constraint."
- LoRA (Low-Rank Adaptation): A parameter-efficient fine-tuning technique that injects low-rank updates into weight matrices. "More commonly, work studies parameter-efficient methods, such as LoRA \cite{hu2022lora} and adapter-based tuning \cite{lin2025continual,lopez2017gradient}, which restrict updates to small subsets of parameters."
- non-parametric updates: Adaptation strategies that modify prompts, context, or external memory rather than changing model weights. "Non-parametric updates. At deployment, we can adapt LLMs without weight changes by changing the inference-time environment around a fixed model."
- on-policy self-distillation (OPSD): A distillation approach where the model learns from preferences or outputs generated by its own current policy. "OPSD~\cite{zhao2026self, lu2025onpolicydistillation}, SDFT~\cite{shenfeld2026self} and SDPO \cite{sdpo} learn from self-distillation"
- parameter-efficient methods: Fine-tuning techniques that update only small subsets of parameters to reduce compute and memory. "More commonly, work studies parameter-efficient methods, such as LoRA \cite{hu2022lora} and adapter-based tuning \cite{lin2025continual,lopez2017gradient}, which restrict updates to small subsets of parameters."
- preference signal: The supervisory signal produced from chosen vs. rejected outputs guiding DPO-style updates. "so the preference signal is implicitly relative to the model's accumulated prior rather than to a fresh reference."
- retrieval-augmented generation (RAG): Enhancing generation by retrieving and conditioning on external documents. "as a memory problem solved with retrieval-augmented generation \cite{lewis2020retrieval}"
- self-distillation: Training a model using signals derived from itself (e.g., as a teacher), often to stabilize learning. "OPSD~\cite{zhao2026self, lu2025onpolicydistillation}, SDFT~\cite{shenfeld2026self} and SDPO \cite{sdpo} learn from self-distillation"
- sparse memory finetuning: A fine-tuning approach that targets sparsely accessed memory slots to localize updates. "sparse memory finetuning \cite{lin2025continual} localizes the next-token objective to sparsely accessed memory slots"
- test-time training: Adapting model parameters during inference based on input-specific signals. "test-time training \cite{feng2026place} performs input-conditioned weight updates during inference."
- temporal drift: Changes over time in the underlying data distribution while the task remains fixed. "We identify four axes along which tasks evolve: domain shift, fact update, temporal drift, and agentic state."
- verifiable reward: A reward signal computed via programmatic or ground-truth verification rather than heuristic judgments. "GRPO \citep{shao2024grpo} samples rollouts in groups, computes within-group advantages from a verifiable reward, and updates the policy under a KL constraint."