When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation

Abstract: Self on-policy distillation trains a student policy against a teacher derived from its own parameter history, yet the teacher's update schedule -- which governs the \emph{temporal coupling} between teacher and student -- has not been systematically studied as a stability variable. Through a controlled schedule sweep on Qwen3-8B, we establish that \emph{isolation periods}, defined as complete teacher freezing between updates, are the key structural property enabling stable learning, not teacher age. To characterize these underlying training dynamics, we introduce a diagnostic framework of temporal KL structure, refresh shock, and length-tail risk. This framework further uncovers \emph{state-oblivious collapse}: optimal short-horizon fixed schedules catastrophically fail under long-horizon training because a clock-driven refresh can copy a transiently drifting student into the teacher in a single, irreversible step. This failure mode is invisible under short-horizon evaluation and mechanistically distinct from EMA's chronic contamination. To address this, we propose \emph{Consolidation-Gated Teacher Refresh} (CGTR), which preserves isolation periods while gating each refresh on joint evidence of reward improvement and length-tail safety, ensuring every teacher movement responds to genuine student consolidation rather than a clock signal. With a single shared parameter set and no per-dataset retuning, CGTR achieves \textbf{zero collapse} and the best final score on all four tasks (Chemistry, Biology, Physics, ToolUse), self-regulating its refresh frequency to each task's learning dynamics.

• The paper introduces the CGTR method, which adaptively refreshes the teacher based on measured consolidation signals.
• It demonstrates that isolation periods and state-aware gating prevent collapse in long-horizon self on-policy distillation.
• Empirical results show CGTR outperforms fixed schedules and EMA, achieving high accuracy across multiple benchmarks.

Temporal Coupling and Training Stability in Self On-Policy Distillation

Introduction

Self on-policy distillation is an increasingly prominent paradigm for post-training LLM improvement, wherein the student policy learns both from its own environment feedback (e.g., reward signals) and dense teacher supervision extracted from its own parameter history. The central question addressed is how the scheduling of teacher updates—defining the temporal coupling between teacher and student—structurally shapes learning stability, particularly in long-horizon optimization regimes. This work systematically characterizes the interplay between teacher update policy, parameter independence, and behavioral stability, ultimately introducing a new method, Consolidation-Gated Teacher Refresh (CGTR), which adaptively controls teacher updates as a function of consolidation evidence, and demonstrating its superiority over fixed-interval and EMA baselines across diverse tasks and datasets (2606.03532).

Failure Modes of Teacher Update Policies

The canonical mechanisms for teacher update—tight hard refresh (small $M$), EMA, and sparse hard refresh (large $M$)—embody distinct structural properties and corresponding failure modes:

• Tight coupling ($M{=}1,2$): Overly frequent refresh collapses the temporal independence between teacher and student, driving the KL regularizer to zero and eliminating the corrective signal (reference collapse). Training degenerates, as evidenced by rapid sequence length explosion and entropy collapse.
• EMA (any $\alpha$): Although per-step contamination is bounded, cumulative drift accumulates in the teacher over long horizons (steady-state contamination), with the teacher becoming a blurred reflection of the student's historical trajectory.
• Sparse hard refresh (large $M$): Isolation periods between updates enforce parameter independence, which was shown to be critical for stable learning and high signal-to-noise ratio in short-horizon training. However, when refresh schedules are oblivious to student state, a single ill-timed refresh during transient drift can irreversibly contaminate the teacher with suboptimal behavior (state-oblivious collapse), leading to catastrophic degradation in long-horizon regimes.

Empirically, in the Qwen3-8B / SciKnowEval Chemistry setup, $M{=}50$ yields superior mean@16 test scores during the initial 100 steps—with SNR $0.497$ and entropy mean $0.499$—but suffers abrupt collapse after step 189 in 300-step experiments. The lag-aligned EMA baseline ($\alpha{=}0.04$) achieves slightly lower performance and experiences entropy drift but avoids hard collapse.

Figure 1: Long-horizon performance comparison (Chemistry, 300 steps). $M{=}50$ suffers catastrophic collapse at step 189, EMA stabilizes but with high entropy drift, and CGTR achieves the best final score with zero collapse.

Diagnostic Framework: KL Structure, Refresh Shock, and Length-Tail Risk

The paper introduces a diagnostic framework for analyzing self-distillation stability:

• Temporal KL structure: The critical indicator for a productive teacher update is a transient KL spike (refresh shock), signifying meaningful correction. Persistent flat KL after a refresh is a marker of reference collapse.
• Length-tail expansion: Growth in the tail of the sequence length distribution is a leading indicator of incipient generation collapse, detectable far earlier than entropy or mean reward degradation.
• Collapse detection criteria: Unconstrained training is recognized by runaway sequence length ($M$08k tokens) and test accuracy plunging below 0.01.

  Figure 2: Illustration of corrective (left, large $M$1) and contaminating (right, low $M$2/EMA) teacher refresh dynamics. Only structurally independent schedules induce meaningful refresh shock.

  

  Figure 3: For $M$3, the seqlen tail begins expanding at step 50, foreshadowing collapse detected at step 86. Stable schedules ($M$4) keep the tail bounded.

CGTR: A State-Aware Adaptive Teacher Refresh Mechanism

To mitigate state-oblivious collapse while preserving the benefits of isolation, the CGTR mechanism is proposed. CGTR introduces a three-gate system:

1. Isolation Gate: Enforces a minimum freeze period ($M$5 steps), establishing hard parameter independence.
2. Reward Gate: Requires the rolling mean reward in the candidate window to exceed the prior peak by a threshold $M$6, providing a consolidation ratchet.
3. Length-Tail Gate: Prohibits refresh if the sequence length tail ratio exceeds a threshold $M$7, precluding reward-illusory drift.

A refresh executes only if all gates are satisfied. This enables the refresh schedule to adapt to the learning dynamics of each task, self-regulating both the frequency and timing of teacher updates without dataset-specific retuning.

Figure 4: Comparison of teacher update strategies. Only CGTR matches isolation periods with explicit reward/length-tail gating, eliminating state-oblivious collapse evidencing fault-tolerance across training regimes.

Empirical Results

CGTR achieves strong uniformity across all benchmarks:

• Zero collapse rate and best final accuracy on Chemistry, Biology, Physics, and ToolUse, without any per-task hyperparameter tuning.
• Adaptive behavior: Number of teacher refreshes self-tunes to task learning profile (e.g., 3 for Chemistry over 300 steps, 2 for Biology).
• Ablations confirm that each gate is critical. Absence of the reward gate, in particular, induces collapse due to refreshes during drift not intercepted by the length-tail criterion.

In long-horizon Chemistry, CGTR achieves a test mean@16 score of $M$8 with only three carefully-timed teacher updates, compared to $M$9 (collapse) for fixed $M{=}1,2$0 and $M{=}1,2$1 (entropy drift, plateau) for EMA-0.04.

Figure 5: Long-horizon results (Chemistry, 300 steps, no auxiliary recovery). CGTR's self-gated refreshes maintain stable performance and low entropy, in contrast to catastrophic collapse with fixed interval and unbounded entropy growth in EMA.

Theoretical and Practical Implications

The empirical and theoretical findings fundamentally revise the understanding of structural requirements for stable self on-policy distillation. The key insights are:

• Isolation periods, not teacher age or lag time, are the determinative factor for maintaining a non-self-confirming distillation objective.
• Fixed clock-based schedules are inadequate for long-horizon regimes due to state-oblivious collapse, a behaviorally distinct failure from EMA's chronic contamination.
• State-aware, consolidation-gated refresh is both necessary and sufficient for robust, task-agnostic self-distillation.

The method obviates the need for dataset-specific tuning and provides strong out-of-domain performance guarantees under a single parameterization.

Future Directions

Potential future research trajectories include scaling CGTR to larger model families, extending the analysis to more complex task types (especially with denser reward structures), and exploring interaction effects between learned reward models and the consolidation gating dynamics. Additionally, theoretical extensions may analyze the optimality of multi-signal gating versus reinforcement-learned teacher schedules.

Conclusion

This work establishes that structural properties—specifically, isolation periods and state-aware update gating—are necessary for stable self on-policy distillation, and that naive clock-driven teacher scheduling fundamentally fails in long-horizon regimes. The CGTR framework, by coupling isolation with reward and behavioral state checks, delivers robust, collapse-free performance and provides a principled template for future self-optimization techniques in LLM distillation.