Self-Distillation-Induced Degradation

• Self-distillation-induced degradation is a phenomenon where iterative self-training leads to over-regularization, reduced uncertainty expression, and diminished performance on novel inputs.
• Empirical evidence in LLMs and diffusion models shows that excessive self-distillation compresses reasoning traces and creates train–test mismatches, degrading model effectiveness.
• Mitigation strategies such as adjusting teacher context, using mode-seeking objectives, and carefully monitoring distillation rounds are essential to balance performance gains and degradation.

Self-distillation-induced degradation refers to the empirically and theoretically characterized phenomenon where applying self-distillation—having a model learn from its own predictions or trajectories—ultimately leads to degraded performance. This negative outcome can manifest as reduced out-of-distribution (OOD) generalization, diminished reasoning capability, or suboptimal optimization dynamics, despite initial rounds of self-distillation often yielding regularization benefits and performance gains. This effect has been observed in diverse domains, including LLMs, diffusion-based LLMs, and kernelized function estimation.

1. Formalization and Mechanisms of Self-Distillation-Induced Degradation

Self-distillation operates by iteratively training a model (the student) to match the output distribution or trajectories of either itself or a previously trained instance (the teacher), typically under a richer or more informative conditioning context. The canonical objective minimizes the KL divergence between the student’s predictive distribution and the teacher’s, potentially under different informational contexts: $$\mathcal{L}_{\rm SD}(θ) = \sum_{t=1}^T \mathrm{KL}\!\Bigl( π_θ(·\mid x,y_{<t}) \,\Big\|\, \text{stopgrad}\,\bigl(π_θ(·\mid x,c,y_{<t})\bigr) \Bigr)\,,$$ where $c$ represents additional context (such as a full solution). In the context of diffusion LMs, trajectory-level self-distillation aligns the joint $(x_0, x_t)$ pairs by minimizing forward-KL or using discriminative, reverse-KL objectives.

Degradation arises when this process over-regularizes the student, either by systematically suppressing uncertainty-verbalization (as in LLMs), introducing a train–test mismatch in state distributions (as in few-step diffusion models), or amplifying spectral regularization (as in kernelized learners). In each context, the self-distillation process initially improves generalization by smoothing over noise or irrelevant modes, but persistent or overly aggressive distillation causally leads to loss of useful signal, under-fitting, or collapse of expressivity (Kim et al., 25 Mar 2026, Zhang et al., 12 Feb 2026, Mobahi et al., 2020).

2. Empirical Manifestations in LLMs and Diffusion LMs

LLMs and Reasoning

In mathematical reasoning tasks, LLMs subjected to self-distillation, especially with highly informative teachers, experience reduced chain-of-thought length and suppressed use of epistemic markers (tokens such as “perhaps” or “maybe” indicating uncertainty). Quantitative experiments show:

• In-domain gains: Self-distillation with fully-specified context ($c=s$) produces precise, concise outputs, yielding high in-domain accuracy and concise reasoning traces (up to 8-fold reduction in length).
• Out-of-domain failure: OOD accuracy on benchmarks such as AIME24 and AMC23 is reduced by as much as 40% and 15%, respectively, when epistemic verbalization is suppressed.
• Suppression of uncertainty: Explicit removal of “> ” segments in teacher traces ($c=s_{\setminus\text{think}}$) mitigates but does not eliminate degradation (Kim et al., 25 Mar 2026).

  Setting	Avg. Score	Avg. Length	Epistemic Token Count
  Unguided	0.30	13054	182.5
  Full Solution	0.98	1873	8.8
  No <think>	0.78	12036	159.8
  Regeneration	0.95	2808	24.1

  Table: Impact of context richness on score, chain-of-thought length, and epistemic marker usage (Kim et al., 25 Mar 2026).

  Diffusion LLMs

  For fast few-step generation, trajectory self-distillation in diffusion LMs leads to accelerated inference but at the cost of quality—if naïve forward-KL distillation or excessive compression is used:

  • Mode-covering distillation: Forward-KL forces the student to cover all modes, oversmoothing and degrading quality, especially in coarse inference states.

  • Train–test mismatch: Under few-step schedules, the intermediate state distributions seen by the student diverge from those at test time, worsening performance.
  • Factorization error: Skipping denoising steps increases inter-token dependencies, which are not modeled by a student trained on random masks.

  Empirically, naïve self-distillation leads to catastrophic drops in accuracy (e.g., 22% on MATH500 with standard trajectory distillation) that are only rescued using mode-seeking objectives (e.g., Direct Discriminative Optimization, DDO) and path consistency, as in T3D (Zhang et al., 12 Feb 2026).

  3. Theoretical Analysis and Information-Theoretic Perspective

  Entropy Compression and Suppression of Exploration

  Self-distillation, by enforcing agreement with a low-entropy teacher, systematically reduces the student model’s output entropy $H(y|x)$. While this eliminates spurious variation, it also removes the “exploration” steps that allow a model to hedge or reconsider. In LLMs, these steps (epistemic markers) are beneficial for adapting to OOD or compositional reasoning. The average OOD accuracy correlates positively with mean chain-of-thought length: $$\text{Accuracy}_\text{OOD} \approx a + b\;\mathbb{E}[L(y)]\quad (b>0)$$ Shortening traces below a model-specific threshold sharply reduces generalization (Kim et al., 25 Mar 2026).

  Progressive Regularization in Hilbert Spaces

  In kernelized regression, each round of self-distillation amplifies $\ell_2$ regularization, with the net effect that high-frequency (small-eigenvalue) basis components are pruned exponentially faster than low-frequency components. This “power iteration” of the ridge regularizer initially improves bias-variance tradeoff but, after a finite number of rounds,

  $$y^{(t)}=\prod_{i=0}^{t-1}A_i\,y^{(0)},\quad A_s=G(G + c_sI)^{-1}$$

  eventually leads to under-fitting and collapse of usable signal (Mobahi et al., 2020).

  4. Domain-Specific Factors and Task Coverage

  The impact of self-distillation-induced degradation is modulated by:

  • Task diversity: Low-diversity (in-domain) settings tolerate aggressive self-distillation, as the confident teacher style is sufficient.
  • Generalization demands: As the diversity of the target distribution increases, the lack of uncertainty expression inhibits the model’s ability to adapt to novel or rare solution paths.
  • Data and schedule alignment: In diffusion LMs, improper alignment between the student’s and teacher’s trajectories (especially under few-step constraints) exacerbates factorization error, further compounding degradation (Zhang et al., 12 Feb 2026).

  5. Mitigation Strategies and Practical Recommendations

  Empirically validated mitigations to counter self-distillation-induced degradation include:

  • Reducing context informativeness: Using partially rather than fully informative teacher context, e.g., supplying solutions stripped of epistemic markers to preserve some uncertainty (Kim et al., 25 Mar 2026).
  • Epistemic-aware objectives: Enforcing similarity between the epistemic token distribution $P_E$ of the student and that of the unguided model via an added KL-divergence penalty.
  • Task-coverage scheduling: Starting from narrow coverage and gradually increasing diversity while relaxing distillation strength.
  • Mode-seeking distillation: In diffusion LMs, replacing forward-KL (mode-covering) with DDO (mode-seeking, reverse-KL) and employing path-consistency weighting to correct for block-wise error propagation (Zhang et al., 12 Feb 2026).
  • Limiting rounds and monitoring fit: In kernelized settings, stopping self-distillation iterations as soon as the norm of the label vector approaches the error tolerance, thereby preventing collapse (Mobahi et al., 2020).

  6. Empirical and Theoretical Insights

  Multiple lines of evidence support that self-distillation acts as a spectrum-dependent regularizer with domain- and task-dependent optimality. Initial self-distillation rounds suppress overfitting and improve generalization; excessive rounds, or overly strong context compression, eliminate valuable exploratory behavior, suppress uncertainty cues, and degrade OOD performance. For rigorous application, monitoring error curves, label norms, or chain-of-thought statistics is essential to avoid tipping from variance reduction into under-fitting.

  7. Limitations and Open Directions

  Remaining limitations include the inherent tradeoff between inference efficiency (shorter traces, fewer diffusion steps) and robustness (retention of uncertainty and adaptability), as well as the imprecise alignment of on-policy distributions in compressed or hybrid schedules. Stronger adaptive, epistemic-aware losses and better alignment strategies—potentially combined with architectural or reinforcement-based innovations—present plausible avenues for further reducing self-distillation-induced degradation (Kim et al., 25 Mar 2026, Zhang et al., 12 Feb 2026). Theoretical understanding in infinite-dimensional spaces and under non-Kernelized architectures is also incomplete, motivating future research.