On-Policy Delta Distillation

Abstract: On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model. Although on-policy distillation has been studied and applied across various settings, its fundamental design remains underexplored. In this paper, we introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution. The delta signal is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. It therefore captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Using extensive empirical evidence, we show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD$<sup>2$). Experiments across mathematics, science, and code-reasoning benchmarks demonstrate that OPD$<sup>2$ consistently outperforms conventional on-policy distillation, enabling reasoning LLMs to achieve strong performance with only a short post-training period. Code will be available at https://github.com/naver-ai/opd2

• The paper introduces OPD², which replaces the teacher–student log-probability gap with a teacher–base-model delta signal to isolate reasoning improvements and consistently outperform standard OPD and ExOPD across 14 math, code, and science benchmarks.
• OPD² combines advantage centering with a sign-consistency gate that stabilizes training, prevents collapse toward one-hot token distributions, and improves performance across Qwen3 and Gemma4 models in both thinking and non-thinking modes.
• The method delivers substantial gains, including a 54.6 versus 51.4 math average for a Qwen3-1.7B student and a 69.2 versus 51.7 AIME24 score for Gemma4, but requires access to the teacher’s base checkpoint and adds up to 28% training overhead.

On-policy distillation (OPD) has emerged as a competitive alternative to reinforcement learning for post-training reasoning LLMs, replacing sparse, reward-model-driven feedback with dense token-level supervision from a teacher. This paper by Heo et al. from NAVER AI Lab revisits the fundamental design of the OPD reward itself. The authors observe that conventional OPD rewards — the log-probability difference between teacher and student on student-sampled tokens — entangle two sources of knowledge: the general next-token preferences acquired during pre-training and the reasoning capability acquired during post-training. Their proposal, On-Policy Delta Distillation (OPD$^2$), replaces the standard reward with a delta signal: the log-probability difference between the reasoning-tuned teacher and its own base (pre-instruction-tuning) model. Because both endpoints of this difference share the same pre-training prior, the signal isolates precisely what reasoning tuning added (2607.15161).

Motivation and analysis of the delta signal

The paper's central claim is that distilling the reasoning-tuned teacher's full output distribution transfers not only reasoning knowledge but also stylistic and natural-language preferences that are irrelevant to reasoning. Three analyses support this. First, word-cloud visualizations over 10k math questions (Qwen3-1.7B student, Qwen3-4B-Thinking-2507 teacher) show that the delta signal preferentially enhances logical connectives such as hence, however, and instead, while suppressing exploratory and verification vocabulary (see, try, verify) that appears in the base model's preference profile. Second, token-level visualizations on synthetically corrupted reasoning traces show that OPD can assign positive rewards to tokens in demonstrably incorrect reasoning — because the student's negative signal magnitude exceeds the teacher's — whereas the delta signal remains reliably negative there, since the reasoning-tuned teacher is more sensitive than its base to incorrect tokens. Third, statistical analysis across Math, Code, and Science domains (10k questions each, 72.4M/53.3M/36.7M generated tokens) shows that switching from OPD to delta systematically enhances logical-connective tokens and suppresses hedging expressions such as perhaps.

These analyses carry a practical implication: the delta signal correlates more tightly with reasoning correctness than the teacher–student difference, which is important because OPD provides no external verifier.

Method: centering and joint conditioning

Two design decisions convert the raw delta reward into a stable training objective. The first is centering: following the policy-gradient principle that action-independent bias is uninformative, the authors subtract the expected reward under the student's sampling distribution (computed over top-$k=1024$ tokens for memory efficiency), yielding advantages $A_t^{\mathrm{OPD}}$ and $A_t^{\Delta}$. The second addresses a convergence pathology specific to the delta signal. Since $R_t^\Delta$ does not depend on the student at all, naive maximization would drive the student toward a one-hot distribution on the maximum-reward token. To prevent this, the paper introduces a joint condition that gates the delta advantage:

$$A_t^{D^2} = \begin{cases} A_t^{\Delta} & \text{if } A_t^{\Delta}\, A_t^{\mathrm{OPD}} > 0 \ 0 & \text{otherwise} \end{cases}$$

Updates are restricted to sign-consistent directions shared by the trace signal and the standard distillation signal, with $A_t^{\Delta}$ controlling gradient magnitude. When the student matches the teacher ($\pi_\theta = \pi^{*}$), the advantage vanishes, restoring a well-defined convergence point. Training follows an RL-style REINFORCE gradient over student rollouts, implemented on TRL's GRPOTrainer with single completions per question and group normalization disabled.

Experimental results

The evaluation framework is deliberately broad: students of Qwen3-1.7B/4B/8B (both non-thinking and thinking modes) and Gemma4-E4B-it, teachers drawn from the same families (up to Qwen3-30B-A3B and Gemma-4-31B), training on a balanced 1:1:1 mix of OpenMathReasoning, OpenScienceReasoning-2, and OpenCodeReasoning questions (100k questions, fewer than one epoch), and evaluation on 14 benchmarks across Math (AIME24/25, AMC23, HMMT25, MATH500, OlympiadBench, ReasoningGym Math), Code (CodeContests, CodeForces, LiveCodeBench v5, RG Algorithm), and Science (GPQA, SuperGPQA, SciBench), all pass@1 averaged over repetitions. All main-table results are reported at the final training step rather than the best checkpoint.

The headline findings are consistent across every configuration:

Setting	Baseline avg	+ ExOPD	+ OPD$^2$
Qwen3-1.7B non-think Math	34.8	51.4	54.6
Qwen3-8B non-think Math	46.9	67.8	71.6
Qwen3-8B think Math	73.7	73.6	75.9
Qwen3-1.7B think Code	29.3	37.1	40.4
Gemma4-E4B think Math	60.6	65.3	67.8

In non-thinking mode, where Qwen3 models are comparatively weak, OPD$^2$ delivers large gains — e.g., Qwen3-4B reaches 70.3 average Math score versus 66.4 for ExOPD, meaning the 4B model trained with OPD$k=1024$0 surpasses the 8B model trained with either baseline. In thinking mode, where baselines are already strong, standard OPD frequently degrades performance and ExOPD yields only marginal or inconsistent improvements, whereas OPD$k=1024$1 still improves all three model sizes on Math (+3.5/+1.5/+2.2 points) and achieves the best averages on Code and Science; notably it lifts Qwen3-8B HMMT25 from 44.3 to 52.3. On Gemma4, OPD causes severe degradation (Code average drops from 55.2 to 36.9), ExOPD partially recovers, and OPD$k=1024$2 retains substantially more capability (49.5) while achieving the best Math result, including AIME24 improving from 51.7 to 69.2. Training curves show that OPD and ExOPD peak early then plateau or decline, while OPD$k=1024$3 maintains a persistent advantage throughout.

An ablation isolates the source of these gains: removing the delta signal (reverting to $k=1024$4) causes the largest degradation across all domains and modes, confirming it as the primary driver; removing the agreement condition or centering has smaller, less consistent effects. The computational overhead is modest — an extra teacher-base forward pass adds roughly 24–28% wall-clock time for Qwen3 and 8% for Gemma4 relative to OPD, comparable to ExOPD, and the implementation is admittedly unoptimized for the reward computation.

Limitations and open questions

Several caveats bear directly on the results. The method requires access to the teacher's base checkpoint, which restricts applicability to open-weight or internally controlled model families; closed teachers without released base models cannot be used. The convergence-point argument rests on the joint condition being sufficient to prevent instability, but the theoretical analysis of this gating is informal, and the ablation shows the condition's contribution is small and inconsistent across settings, leaving its necessity somewhat unresolved. The evaluation covers only 100 optimization steps on strong instruct-tuned students with same-family teachers; behavior under longer training, weaker or cross-family teachers, or larger scale is untested. Finally, the interpretation of the delta signal as "the learning trace of reasoning tuning" is supported empirically through lexical statistics rather than mechanistically, so the precise reason it outperforms direct imitation remains an open question.

Conclusion

This paper makes a targeted but well-supported revision to on-policy distillation: replace the teacher–student probability gap with the teacher–base gap as the primary reward, stabilized by advantage centering and a sign-consistency condition against the original distillation signal. Across three reasoning domains, fourteen benchmarks, four model sizes, two generation modes, and two model families, OPD$k=1024$5 consistently outperforms both vanilla OPD and the extrapolation-based ExOPD, with the ablation attributing the gains primarily to the delta signal itself. The work demonstrates that the design of the distillation reward — long treated as fixed — is a productive axis for improving post-training efficiency, at the cost of requiring teacher-base checkpoints whose availability constrains deployment.