Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation

Abstract: On-policy distillation (OPD) has emerged as an efficient post-training paradigm for LLMs. However, existing studies largely attribute this advantage to denser and more stable supervision, while the parameter-level mechanisms underlying OPD's efficiency remain poorly understood. In this work, we argue that OPD's efficiency stems from a form of ``foresight'': it establishes a stable update trajectory toward the final model early in training. This foresight manifests in two aspects. First, at the \textbf{Module-Allocation Level}, OPD identifies regions with low marginal utility and concentrates updates on modules that are more critical to reasoning. Second, at the \textbf{Update-Direction Level}, OPD exhibits stronger low-rank concentration, with its dominant subspaces aligning closely with the final update subspace early in training. Building on these findings, we propose \textbf{EffOPD}, a plug-and-play acceleration method that speeds up OPD by adaptively selecting an extrapolation step size and moving along the current update direction. EffOPD requires no additional trainable modules or complex hyperparameter tuning, and achieves an average training acceleration of $3\times$ while maintaining comparable final performance. Overall, our findings provide a parameter-dynamics perspective for understanding the efficiency of OPD and offer practical insights for designing more efficient post-training methods for LLMs.

• The paper identifies that OPD's main contribution is its foresight mechanism, which aligns early parameter updates with the final solution to accelerate LLM training.
• It shows that OPD avoids functional redundancy by concentrating updates on critical modules, achieving higher reasoning accuracy with more compact changes.
• EffOPD, an adaptive acceleration framework, extrapolates along stabilized, low-rank parameter directions to reduce convergence time significantly.

Parameter-Dynamics Perspective on the Efficiency of On-Policy Distillation in LLMs

Introduction

On-Policy Distillation (OPD) has become a preferred paradigm for efficient post-training of LLMs, consistently demonstrating superior convergence rates and performance compared to reinforcement learning (RL)–based methods for reasoning and alignment tasks. Despite widespread empirical adoption, the underlying mechanistic factors that explain the pronounced efficiency of OPD remain opaque. Prior work suggests that OPD’s advantage stems mainly from denser, more stable supervision, but such global, optimization-level explanations are insufficient to capture the details of parameter-space evolution. This paper provides a granular, parameter-dynamics analysis, positing that the core efficiency of OPD originates from a foresight mechanism wherein the optimization trajectory aligns early and persistently with the direction of the final solution. The implications span both practical acceleration strategies and a richer understanding of post-training dynamics for LLMs.

Figure 1: Illustration of the foresight mechanism in OPD. Compared with RL, OPD identifies critical modules and aligns with the final optimization direction early in training, concentrating effective updates while reducing redundancy. Based on this, EffOPD extrapolates along the predicted direction to accelerate training.

Functional Redundancy Avoidance in OPD

A central finding is that OPD avoids functional redundancy at the module level. While RL-based approaches tend to distribute parameter updates across the architecture, including layers with negligible marginal utility, OPD concentrates the update budget on modules that are empirically indispensable for reasoning, particularly intermediate-layer MLP blocks. This concentration is not merely a consequence of reduced update norms; when matching update magnitudes, OPD produces stronger reasoning gains than RL, indicating a structurally more efficient allocation of parameter changes.

Figure 2: Comparison of parameter update efficiency between RL and OPD. (a) For matched update norms at the final checkpoint, OPD achieves higher reasoning accuracy. (b) Throughout training, OPD consistently reaches equivalent accuracy with smaller parameter movements.

This efficiency is evident throughout the training trajectory. Fine-grained sliding-window interventions reveal that RL accumulates large updates in layers peripheral to reasoning quality, driving up parameter movement without proportional performance gains. In contrast, OPD suppresses unnecessary updates in these regions and achieves comparable or superior outcomes with more compact changes. Empirical quantification across embedding, attention, and MLP components confirms that the main reasoning signals are highly localized, and OPD's update pattern closely matches this localization while mitigating superfluous adjustments.

Figure 3: Functional contributions and update distributions across architectural components. Middle-layer MLP updates exert dominant influence on downstream reasoning accuracy; OPD suppresses redundant peripheral changes more aggressively than RL.

Geometric Structure: Early Low-Rank Lock-in

OPD does not merely concentrate updates at the module-level, but also exhibits strong geometric structure in parameter space. Singular value decomposition (SVD) of parameter updates demonstrates that OPD’s update matrices are characterized by higher spectral norm, higher spectral-to-Frobenius norm ratio, lower effective rank, and greater concentration in the leading $k\%$ subspace compared to RL. This denotes a pronounced low-rank structure: most parameter movement during OPD is aligned with a narrow subset of highly effective directions.

Figure 4: Low-rank subspace analysis. (a) OPD’s updates in the principal subspace yield stronger performance per unit norm; (b) RL expends greater norm on tail directions, attaining marginal performance gain.

Controlled experiments confirm that over $95\%$ of total performance improvement is recoverable from just the top $10\%$ of singular components of the update matrix, with OPD outperforming RL for any fixed norm budget. RL directs substantive movement into subspaces that fail to contribute significantly to reasoning performance, indicating inefficient parameter search or credit assignment. The tail spectrum of RL updates has disproportionately large norm but low functional utility.

Crucially, OPD establishes this low-rank, functionally aligned subspace early in training. Temporal analysis via t-SNE of top-$k$ principal directions and cosine similarity measures shows that OPD rapidly locks into globally effective directions. Subsequent optimization consists mainly of norm amplification along these axes with little further subspace exploration, whereas RL proceeds with more erratic, less stable trajectories. Scaling the update magnitude of early OPD checkpoints along their established directions yields up to $80\%$ of final reasoning performance—an explicit demonstration of early lock-in.

Figure 5: Subspace evolution and weight scaling analysis. (a) t-SNE reveals OPD's smoother, more stable top-1 subspace trajectories; (b) cosine similarity metrics show early OPD updates align closely with the final solution; (c) scaling update norms in early checkpoints recovers most final accuracy.

EffOPD: Acceleration via Directional Extrapolation

Building on the observation that OPD’s update directions stabilize and align with the final solution early, the authors propose EffOPD—a plug-and-play acceleration framework. EffOPD triggers at exponentially spaced checkpoints, estimating the current update direction and extrapolating the parameters along this direction with adaptively tuned step sizes, validated by lightweight held-out batches. This enables efficient exploitation of the early-locked directionality, avoiding the computationally wasteful, incremental steps of vanilla OPD and mitigating overfitting/semantic drift arising from excessive post-convergence updates.

Empirical results across code generation and mathematical reasoning tasks (“Eurus-RL-Code”, “DeepMath-103K”) and model sizes from 1.5B to 32B parameters establish that EffOPD obtains equivalent or better final accuracy in less than one-third the wall-clock time, with convergence observable in as few as 10 training steps for substantial scales. The adaptive validation set, even with small sample sizes, is sufficient; EffOPD is robust to learning rate settings and exhibits superior early performance relative to static extrapolation baselines.

Figure 6: Performance comparison of different distillation methods on code and math datasets; EffOPD outpaces vanilla OPD and static extrapolation methods on both convergence speed and final accuracy.

Figure 7: EffOPD ablation studies on learning rate, validation set difficulty, and runtime; performance and efficiency advantages are robust to configuration choices.

Practical and Theoretical Implications

The parameter-dynamics discoveries in this work decisively shift interpretability of OPD from macroscopic notions of “dense supervision” to precise, quantitative statements about update compactness, functional localization, and early subspace lock-in. Practically, these results furnish both diagnostic tools (spectral/trajectory analysis of parameter updates) and actionable acceleration strategies for LLM post-training, offering compatibility with existing frameworks and orthogonality to other optimization improvements.

Theoretically, the analysis reveals that OPD’s advantage is a consequence of the joint effect of dense, teacher-driven supervision and the emergent low-rank parameter geometry, resulting in a "foresight" inductive bias. RL, encumbered by sparse rewards and high-variance credit assignment, not only learns more slowly but also diffuses parameter changes across less efficacious directions, carrying higher aggregate update norm for equivalent performance. This insight points to future algorithm designs that could further couple module-wise importance estimation and subspace analysis for adaptive, data-efficient post-training, especially in settings of distributional shift or complex, multi-modal tasks.

Conclusion

This work articulates and empirically validates two essential properties underpinning OPD efficiency in LLMs: Functional Redundancy Avoidance and Early Low-Rank Lock-in. By providing a parameter-dynamics foundation for the well-known empirical advantages of OPD, the investigation sets up a new agenda for post-training research, focusing on explicit geometric and functional trajectory analysis. The EffOPD framework operationalizes this understanding, delivering practical acceleration with minimal overhead and strong empirical results. Future directions include extending the analysis to multimodal and multi-turn agent architectures and leveraging parameter-dynamics metrics for online monitoring and adaptive control during distillation and alignment processes.

---

Reference:

“Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation” (2605.11739)