Learning, Fast and Slow: Towards LLMs That Adapt Continually

Abstract: LLMs are trained for downstream tasks by updating their parameters (e.g., via RL). However, updating parameters forces them to absorb task-specific information, which can result in catastrophic forgetting and loss of plasticity. In contrast, in-context learning with fixed LLM parameters can cheaply and rapidly adapt to task-specific requirements (e.g., prompt optimization), but cannot by itself typically match the performance gains available through updating LLM parameters. There is no good reason for restricting learning to being in-context or in-weights. Moreover, humans also likely learn at different time scales (e.g., System 1 vs 2). To this end, we introduce a fast-slow learning framework for LLMs, with model parameters as "slow" weights and optimized context as "fast" weights. These fast "weights" can learn from textual feedback to absorb the task-specific information, while allowing slow weights to stay closer to the base model and persist general reasoning behaviors. Fast-Slow Training (FST) is up to 3x more sample-efficient than only slow learning (RL) across reasoning tasks, while consistently reaching a higher performance asymptote. Moreover, FST-trained models remain closer to the base LLM (up to 70% less KL divergence), resulting in less catastrophic forgetting than RL-training. This reduced drift also preserves plasticity: after training on one task, FST trained models adapt more effectively to a subsequent task than parameter-only trained models. In continual learning scenarios, where task domains change on the fly, FST continues to acquire each new task while parameter-only RL stalls.

• The paper introduces FST, a dual-channel framework interleaving slow weight updates and fast prompt optimization for LLM adaptation.
• It demonstrates improved sample efficiency, achieving 3x fewer RL steps and higher peak accuracies across tasks like CodeIO and Math.
• FST preserves model plasticity and mitigates catastrophic forgetting, enabling robust continual learning and rapid adaptation to new domains.

Fast-Slow Training for LLM Adaptation: A Dual-Channel Approach to Continual and Efficient Learning

Introduction

This paper introduces a unifying fast-slow adaptation framework, Fast-Slow Training (FST), for LLMs that combines the benefits of slow parametric updates (i.e., model weight finetuning via RL or SFT) and fast, dynamically optimized textual context (prompts and instructions). FST explicitly models adaptation as a dual-channel process: slow weights ($\theta$) persist broad capabilities, updated infrequently and expensively, while fast weights ($\Phi$)—a population of optimized textual contexts—encode rapidly shifting or task-specific features, updated cheaply and exploited immediately.

The rationale for FST stems from the limitations of pure RL/SFT post-training, which pushes all improvements into static parameters. Such monolithic adaptation leads to catastrophic forgetting, collapse of output entropy, impaired capacity for future adaptation (plasticity loss), and over-specialization to recent reward signals. Pure prompt optimization, although data-efficient and robust to forgetting, is fundamentally bottlenecked in representational capacity and cannot reach the asymptotic performance achievable with direct parameter learning.

By interleaving RL updates to model weights with continual prompt evolution (via reflective optimization using GEPA), FST achieves both fast, task-level adaptation and robust, general post-training, outperforming single-channel methods both in sample efficiency and final performance.

Figure 1: FST interleaves slow RL updates of parameters with fast, reflective evolutionary optimization of a population of prompts (contexts), distributing adaptation across two distinct time scales and representational substrates.

Fast-Slow Training: Methodology

FST delineates LLM learning as concurrent optimization in two spaces:

• Slow weights ($\theta$): Model parameters are updated via RL with verifiable reward (RLVR), employing a high-fidelity reward signal computed by rule-based verifiers or automatic evaluators dominant in code, math, and reasoning tasks.
• Fast weights ($\Phi$): A population of textual prompts optimized via GEPA (a Pareto-optimal, reflective evolutionary prompt search), guided by feedback and natural language critiques extracted from rollouts.

At each training cycle, FST:

1. Uses the current policy to generate rollouts under the current prompt pool.
2. Elicits critiques and candidate prompt mutations using a frozen, strong LLM as a reflective agent.
3. Maintains a diverse pool of prompt candidates (Pareto frontier) that maximize reward on an anchor batch.
4. Interleaves cycles of RL-VR optimization of weights, during which mini-groups sample from the prompt pool, ensuring parameter learning is always conditioned on a diverse contextual distribution.

This design allows each "channel" to specialize: persistent model weights consolidate durable, broadly applicable skills and compositional reasoning, while the fast channel scavenges rapidly from the reward stream, absorbs transient feedback, and handles task/domain drift without requiring persistent parameter change.

Empirical Advantages of Fast-Slow Training

Data Efficiency and Performance Asymptote

FST demonstrates strong improvements in data efficiency, requiring 3x fewer RL steps to match the peak performance of parameter-only RL on CodeIO and HoVer-hard, and 1.4x fewer steps on Math (Polaris). Moreover, FST consistently converges to a higher performance ceiling (asymptote) than RL-only methods across diverse domains:

• On CodeIO: Asymptotic accuracy of 47.4% (FST) vs. 43.0% (RL).
• On Math (Polaris): 49.2% vs. 46.4%.
• On HoVer-hard: 25.0% vs. 17.3%.

  Figure 2: FST achieves higher validation rewards with substantially fewer training steps compared to RL across code, math, and reasoning tasks.

  

  Figure 3: FST asymptotes (green) are uniformly higher than RL (blue) across tasks, as determined by sigmoid fits to validation-reward curves.

The faster adaptation is explained by the fast textual prompts’ ability to extract reward-driven structure much earlier (by step 50 in the synthetic star-graph search task, compared with RL’s 300-step delay), leveraging detailed feedback and flexible context injection.

Figure 4: FST escapes the zero-reward regime in synthetic star-graph search an order of magnitude faster than RL-only baseline.

Policy Drift, Plasticity, and Continual Learning

At any matched reward, FST-trained policies move substantially less from the base initialization in terms of KL divergence (up to 70% less KL), indicating that part of the required adaptation signal is offloaded to the fast context channel rather than encoded in weights. This conservative movement preserves the generality and entropy of the base model, reduces risk of overfitting, and critically, maintains the model’s ability to adapt to future tasks.

Figure 5: At matched reward, FST (green) achieves lower policy KL from the base than RL (blue), reflecting reduced parameter drift.

Explicit plasticity probes show that after training on task X (Math or Physics), continuing RL on task Y (HoVer-hard) from FST checkpoints enables rapid adaptation, while RL-trained weights completely stall or collapse, validating that catastrophic specialization is avoidable via dual-channel adaptation.

Figure 6: Continued RL from FST checkpoints maintains learnability on new tasks, while RL-only initializations often collapse plasticity.

In true continual learning benchmarks where task distributions change on the fly, only FST maintains the ability to acquire new domains, rapidly climbing to domain-peak accuracy at every shift, while RL-only models suffer dramatic stagnation after the first domain shift.

Figure 7: In continual learning over three domains, FST (solid) adapts seamlessly across shifts, whereas RL (dashed) stalls after the first transition.

Ablations and Component Analysis

Ablations reveal that:

• The fast channel (prompt pool) is solely responsible for early reward uptake.
• However, the highest performance ceiling (both data-efficient and asymptotic) is achieved only when both fast and slow channels are jointly optimized. Prompt-only and distillation-from-prompts methods plateau well below FST.
• Maintaining a diverse population of prompts (vs. a single best prompt) aids both generalization and exploration, mediated via multi-modal advantage computation during RL.

  Figure 8: FST lifts validation accuracy on HoVer above both RL-only and prompt-only baselines; distillation from fast weights alone cannot close the gap to FST.

Theoretical and Practical Implications

The empirical results indicate that constraining post-training adaptation to parameter updates, as standard RL and SFT do, is suboptimal for both sample efficiency and robustness. The explicit separation of adaptation time scales in FST leverages the complementary strengths of persistent and ephemeral memory. This has multiple implications:

• Catastrophic forgetting and plasticity loss, long demonstrated in RL and continual learning literature, are mitigated by dual-channel adaptation. This is supported by substantial literature on capacity loss (Lyle et al., 2022), plasticity [Dohare2024, (Tang et al., 31 May 2025)], and forgetting (Luo et al., 2023).
• Data efficiency improvements are grounded in the fast channel’s access to rich text-based feedback and its ability to explore the reward landscape via context diversity, moving beyond the severe information bottleneck of sparse binary RL rewards.
• The fast–slow decomposition is architecture and optimizer-agnostic, and could integrate with other prompt-optimization, SFT, or RL variants as advances emerge.
• The dual-channel approach aligns with cognitive theories (complementary learning systems) and echoes prior work on fast-weight architectures [hinton1987using, schmidhuber1992learning, ba2016usingfastweightsattend], extending these ideas to the rich structure of LLM/ICL architectures.

Future developments should consider:

• Scaling and adapting FST to larger base models and more complex, multi-modal data streams.
• Integrating more advanced context optimizers and scalable, cost-efficient RL surrogates.
• Exploiting fast channel diversity for improved exploration, generalization, and robustness to distributional shift.

Conclusion

FST reframes LLM post-training as a dual-channel, interleaved optimization problem rather than a sequence of parameter learning followed by prompt tuning. By consistently outperforming parameter-only and context-only methods on data efficiency, asymptotic accuracy, preservation of plasticity, and continual learning, FST establishes the necessity of co-evolving fast (textual/contextual) and slow (parametric) adaptation mechanisms for scalable and robust LLM deployment. This paradigm provides a theoretically sound and empirically validated approach to both efficient domain adaptation and mitigation of catastrophic forgetting, with high relevance for future LLM-based continual learning and life-long adaptation systems.

---

References:

• "Learning, Fast and Slow: Towards LLMs That Adapt Continually" (2605.12484)
• Cited papers in the arXiv reference list, e.g., "Understanding and Preventing Capacity Loss in Reinforcement Learning" (Lyle et al., 2022), "Loss of plasticity in deep continual learning" [Dohare2024], "Mitigating Plasticity Loss in Continual Reinforcement Learning" (Tang et al., 31 May 2025), "GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning" (Agrawal et al., 25 Jul 2025), "Scaling Laws for Forgetting When Fine-Tuning LLMs" (Kalajdzievski, 2024), and foundational works on fast weights and CLS.