Self-Distillation Enables Continual Learning

Abstract: Continual learning, enabling models to acquire new skills and knowledge without degrading existing capabilities, remains a fundamental challenge for foundation models. While on-policy reinforcement learning can reduce forgetting, it requires explicit reward functions that are often unavailable. Learning from expert demonstrations, the primary alternative, is dominated by supervised fine-tuning (SFT), which is inherently off-policy. We introduce Self-Distillation Fine-Tuning (SDFT), a simple method that enables on-policy learning directly from demonstrations. SDFT leverages in-context learning by using a demonstration-conditioned model as its own teacher, generating on-policy training signals that preserve prior capabilities while acquiring new skills. Across skill learning and knowledge acquisition tasks, SDFT consistently outperforms SFT, achieving higher new-task accuracy while substantially reducing catastrophic forgetting. In sequential learning experiments, SDFT enables a single model to accumulate multiple skills over time without performance regression, establishing on-policy distillation as a practical path to continual learning from demonstrations.

• The paper presents SDFT, a method that uses a model’s in-context learning to generate on-policy training signals via reverse KL divergence minimization.
• It bridges supervised imitation and on-policy reinforcement learning, boosting new-task accuracy (up to 100%) while reducing catastrophic forgetting.
• Empirical results show SDFT’s superior performance and Pareto efficiency in sequential learning and skill retention across various benchmarks.

Self-Distillation Fine-Tuning for Continual Learning in Foundation Models

Introduction

Continual learning presents a critical challenge for foundation models: enabling the acquisition of new skills and knowledge without degradation of existing capabilities. The prevalent supervised fine-tuning (SFT) approach, which trains models to imitate expert demonstrations, is fundamentally off-policy and thus subject to severe catastrophic forgetting of prior abilities. The paper "Self-Distillation Enables Continual Learning" (2601.19897) introduces Self-Distillation Fine-Tuning (SDFT), a straightforward yet effective method that leverages a model's in-context learning to generate on-policy training signals from demonstrations. By using the same model in both student and teacher roles, with the teacher conditioned on expert demonstrations, SDFT directly minimizes the reverse KL divergence between the student and teacher distributions. The result is significantly improved new-task accuracy and substantially reduced forgetting in sequential learning frameworks.

The SDFT Algorithm: On-Policy Learning from Demonstrations

The SDFT procedure exploits the in-context learning capabilities of large pretrained models. For each training query $x$, the model is evaluated in two configurations:

• Student Policy: Conditioned only on $x$, representing the current behavior.
• Teacher Policy: Conditioned on $(x, c)$, where $c$ is an expert demonstration, thus eliciting behavior reflecting the demonstration's intent.

The optimization targets the reverse KL divergence between these two output distributions:

$$\mathcal{L}(\theta) = D_{KL}\left(\pi_\theta(\cdot|x) \parallel \pi(\cdot|x, c)\right)$$

Such distillation is performed on student rollouts, i.e., on-policy, preventing the drift into unseen state distributions that plague off-policy imitation learning.

Figure 1: SDFT turns expert demonstrations into on-policy signals by leveraging in-context learning; the teacher policy remains close to the base while enabling high new-task accuracy.

This mechanism has a formal equivalence to an implicit inverse reinforcement learning: the reward is defined by the log-probability difference between the demonstration-aware teacher and the current student, rendering SDFT an efficient on-policy algorithm for demonstration-driven skill transfer.

Empirical Validation of the In-Context Assumption

A key theoretical prerequisite for the efficacy of SDFT is that the demonstration-conditioned model approximates the optimal next policy and stays close to the base distribution. Empirical evaluation in the ToolAlpaca benchmark demonstrates that:

• Conditioning on demonstrations raises accuracy from 42% (base) to 100% (teacher).
• The KL divergence between teacher and base policy is half that of standard SFT, maintaining critical proximity.

Figure 2: Conditioning the model on demonstrations yields outputs substantially closer to the base policy compared to SFT.

Performance Trade-Offs: Accumulating Skills Without Forgetting

Benchmarks in skill learning (Science Q&A, Tool Use, Medical Reasoning) and knowledge acquisition (new Wikipedia facts) show that SDFT models consistently occupy the Pareto frontier—achieving superior new-task accuracy and simultaneously retaining prior capabilities. Catastrophic forgetting is dramatically mitigated relative to SFT and other baselines.

Figure 3: SDFT achieves superior Pareto efficiency across tasks, balancing new skill acquisition and retention of prior capabilities.

Sequential training across multiple tasks further confirms the stability: SDFT enables continual accumulation of skills, unlike SFT which displays regression once new training phases begin.

Figure 2: SDFT retains performance across tasks in a sequential setting, while SFT incurs sharp drops due to forgetting.

Model Scaling and Skill Generalization

The effectiveness of SDFT scales with model size, benefiting from stronger in-context reasoning. Larger models show widening performance gaps over SFT on complex tasks. Additionally, improvements measured by pass@$k$ exhibit genuine skill acquisition rather than mere entropy reduction, suggesting the robustness of SDFT's gains.

Figure 4: SDFT leverages increased in-context learning in larger models and demonstrates uniform improvements in pass@$k$ metrics.

Ablations: Context, Teacher Definition, and Reasoning Behavior

Critical analysis reveals that conditioning the teacher on both demonstration text and answer maximizes knowledge transfer efficacy, outperforming text-only or answer-only conditioning.

Figure 5: Full demonstration context is essential for effective knowledge injection in SDFT.

Maintaining an exponential moving average (EMA) of the student parameters as the teacher delivers stable and performant learning dynamics, contrasting with instability when using the current student directly or lack of adaptability when using a frozen base.

Figure 6: EMA teacher yields stable training; other teacher choices introduce instability or underperformance.

SDFT also enables the adaptation of reasoning models in settings devoid of explicit reasoning supervision: training on answer-only data preserves reasoning depth and accuracy, while SFT leads to reasoning collapse.

Theoretical and Practical Implications

• On-policy Distillation as a Bridge: SDFT bridges the gap between supervised imitation and on-policy RL, enabling learning from demonstrations without explicit rewards.
• Scalability and Efficiency: SDFT provides per-token supervision and requires a single on-policy rollout per prompt, offering computational advantages over group-based RL estimators.
• Limitations: SDFT depends on strong in-context learning abilities and struggles in shifting model behavior when underlying reasoning styles differ sharply from demonstrations.
• Scope for Future Work: Integration of reward-based signals, enhancement of robustness to spurious teacher artifacts, and adaptation to settings with sub-optimal or noisy demonstrations.

Conclusion

Self-Distillation Fine-Tuning represents a significant advancement in continual learning for foundation models, offering a practical mechanism for the on-policy incorporation of expert demonstrations. SDFT enables the stable acquisition of new tasks and knowledge without sacrificing prior capabilities and generalization, outperforming standard fine-tuning and other baselines. Its reliance on in-context learning links SDFT efficacy to model scale and reasoning ability, suggesting growing advantages with further model development. The method's integration into continual learning paradigms opens new avenues for the adaptive evolution of AI systems.