Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

Abstract: On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base \texttt{Qwen3-1.7B}, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263\% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.

• The report challenges the dominant belief that on-policy distillation (OPD) improves student models through a reliable knowledge-transfer from teacher models by demonstrating significant performance improvements even with highly unreliable teacher signals.
• The study introduces On-Policy Self-Adaptation (OPSA), a method that adapts a learning strategy based on token-level entropy without needing a teacher model to achieve significant improvements in mathematical reasoning tasks.
• OPSA demonstrates substantial gains in performance metrics (e.g., 263.5% relative increase in AIME24 Avg@32 for the Qwen3-1.7B model) and is computationally cheaper and less entropy intensive than alternative methods, resulting in improvements on average sampled accuracy and robustness.

The paper examines a central but underexplored premise of on-policy distillation (OPD): whether performance improvements arise from genuine transfer of teacher behavior or from a simpler transformation induced by the optimization procedure itself. Its empirical thesis is deliberately contrarian: OPD can improve a student despite highly unreliable teacher advantages, and much of its benefit can be reproduced by suppressing low-probability student-generated tokens without any teacher, reward, labels, or reference answers. From this analysis, the paper introduces On-Policy Self-Adaptation (OPSA), a token-level, supervision-free training method that assigns entropy-adaptive negative advantages to low-probability tokens. The study reports substantial gains on mathematical reasoning while arguing that the mechanism is not indiscriminate entropy reduction: OPSA sharpens predictions at low-entropy positions but preserves branching diversity at high-entropy reasoning forks (2608.31046).

Motivation and analytical framework

RLVR methods such as GRPO provide response-level advantages derived from verifiable outcomes. Although these methods can improve reasoning, their supervision is sparse over long trajectories and can vanish when sampled responses share the same outcome. OPD addresses this granularity problem by evaluating student-generated trajectories with a stronger teacher and assigning token-level advantages based on the reverse-KL objective. For a student-sampled token, the advantage is determined by the difference between teacher and student log probabilities. Positive values increase the probability of the sampled token; negative values suppress it.

This construction is computationally and conceptually nontrivial. The teacher evaluates prefixes generated by the student, although those prefixes may be substantially off-policy relative to the teacher. The paper therefore investigates whether teacher-provided token signals remain aligned with verifiable correctness in precisely the distribution on which OPD trains. The analysis uses Qwen3-1.7B as the student and Qwen3-4B, Qwen3-30B-A3B, and Qwen3-235B-A22B as teachers, with answer-token correctness determined by a verifier on DAPO-17k questions.

The paper’s key methodological choice is to define supervision noise at verifiable answer tokens rather than attempting to assign correctness to every intermediate reasoning token. A teacher signal is labeled noisy when its sign disagrees with the final verifiable outcome: a negative advantage on a correct answer or a positive advantage on an incorrect answer. This definition is narrower than trajectory-level noise metrics and does not establish that intermediate-token advantages are incorrect. It nevertheless provides a direct test of whether OPD’s terminal supervision is directionally compatible with outcome correctness.

Teacher supervision is noisy and increasingly insensitive to correctness

The measured noise is substantial for every teacher scale. With the 4B teacher, 20.4% of correct trajectories receive negative advantages on their answer tokens, while 40.8% of incorrect trajectories receive positive advantages. The aggregate noise rate is therefore 30.6%. It rises to 34.7% for the 30B-A3B teacher and 50.6% for the 235B-A22B teacher.

The behavior of the largest teacher is particularly important. It assigns negative advantages to 97.8% of answer tokens in correct trajectories and 96.6% in incorrect trajectories. Thus, for this student–teacher pairing, the teacher signal is almost entirely negative and barely discriminates between correctness classes. The paper attributes this trend to increasing distributional mismatch: as teacher capability and scale increase, student-generated sequences become more off-policy under the teacher distribution. This interpretation is plausible, although the experiments do not isolate distributional mismatch from other scale-dependent effects such as calibration, instruction tuning, or differences in decoding behavior.

Figure 1: Teacher-advantage noise increases with teacher scale, while OPD remains comparatively insensitive to filtering noisy trajectories.

The more surprising result comes from the filtering experiment. Standard OPD trains on all trajectories; two controls train exclusively on trajectories containing noisy answer-token signals or exclusively on trajectories without them. All three variants reach comparable performance after similar numbers of gradient steps. In particular, training only on trajectories classified as noisy still produces improvement at a rate comparable to standard OPD.

This result directly weakens the interpretation that OPD’s gains primarily reflect reliable teacher-to-student knowledge transfer. It does not prove that teacher information is never used: the noise criterion is evaluated only at answer tokens, while the optimization acts over the entire sequence, and the teacher may provide useful structural information at unmeasured intermediate positions. Nevertheless, the result demonstrates that OPD can improve under supervision whose terminal-token direction is frequently inconsistent with verifiable correctness.

The effective signal is concentrated in low-probability tokens

The paper next decomposes OPD’s effect by token probability and advantage magnitude. The relevant logit gradients vanish in two regimes: when the advantage is close to zero and when the sampled token already has probability close to one. These observations imply that nominally dense token-level supervision may be highly sparse in effective gradient capacity.

For the Qwen3-1.7B student and Qwen3-4B teacher, 29.2% of tokens have exactly zero advantage and 51.7% have advantage magnitude below $10^{-4}$. Near-zero advantages are concentrated among tokens with high student log probability. When the student is already confident in a sampled token, the teacher often assigns a similar probability conditioned on the same prefix, making the OPD advantage negligible.

Figure 2: Near-zero OPD advantages concentrate on high-log-probability tokens, and training exclusively on those tokens produces little improvement.

The controlled token-selection experiment supports this gradient-level account. Training on progressively larger fractions of the student’s highest-log-probability tokens produces little noticeable improvement on AIME24. Replacing the original OPD advantages with random values in $[-1,1]$ does not materially change this result. The implication is strong: the high-probability tokens that appear to receive teacher supervision contribute little to policy improvement under the tested optimization regime. The result is also a caution against equating token-level density with effective supervision density.

By contrast, restricting OPD to the 20% of tokens with the lowest student log probabilities achieves performance comparable to full-token OPD. This establishes that the bulk of the useful update is concentrated in a small subset of student-sampled tokens. The analysis does not show that all low-probability tokens are harmful or that high-probability tokens are universally irrelevant; rather, it identifies a regime in which their OPD gradients are empirically weak.

Negative advantages reproduce much of OPD’s improvement

The authors then remove the teacher’s token-specific advantage values. They compare standard OPD with teacher-free training that assigns either a fixed negative advantage of $-0.5$ or a fixed positive advantage of $+0.2$ to the lowest-log-probability 20% of tokens.

Figure 3: Fixed negative advantages improve the student, whereas fixed positive advantages induce rapid policy collapse.

The negative-advantage variant steadily improves AIME24 Avg@4 and produces response-length growth similar to standard OPD, eventually stabilizing near 12,000 tokens. Standard OPD restricted to the lowest-log-probability tokens also performs comparably to full-token OPD. These findings imply that the direction of the signal—suppression rather than reinforcement—is more important than the teacher’s fine-grained preference estimate in this setting.

The positive-advantage control has the opposite behavior. Within approximately 40 training steps, response length falls toward zero, the gradient norm explodes, and the model degenerates into largely random or garbled outputs. This is a particularly informative negative control: the observed benefit is not explained merely by applying arbitrary token-level updates. It depends on selectively assigning negative credit to unlikely sampled tokens.

Figure 4: Entropy-correlated negative advantages provide a more effective and stable update than fixed or inversely correlated signals.

The paper varies the magnitude of negative advantages as a function of token entropy. A positive correlation between entropy and negative-signal magnitude reaches 50.0% AIME24 Avg@4, compared with 35.13% for standard OPD. In contrast, assigning stronger negative signals to lower-entropy positions produces instability between training steps 350 and 450, persistently larger gradient norms, and slightly worse final performance than the fixed-negative baseline.

The result separates two notions often conflated in RL training: low probability and high uncertainty. A sampled token can have low probability because it is a tail event in a sharply peaked distribution, or because probability mass is distributed across many plausible alternatives. The paper argues that the latter context is especially useful for learning, motivating an entropy-aware update rather than ranking tokens by log probability alone.

OPSA: entropy-adaptive self-improvement

OPSA operationalizes the preceding findings through three rules. It selects the lowest-log-probability 20% of sampled token positions, assigns them negative advantages, and scales the magnitude of those advantages positively with token entropy. The method requires only the student’s own rollout probabilities and entropies. It does not use a teacher, verifiable rewards, ground-truth answers, reference answers, or auxiliary hints.

At high-entropy positions, suppressing a sampled low-probability token reallocates probability mass among a relatively broad set of plausible head tokens. At low-entropy positions, the same operation suppresses unlikely tail events while leaving already-confident predictions largely unchanged. The resulting update is therefore asymmetric across the sequence: it reduces implausible deviations in confident contexts while maintaining alternatives at uncertain contexts.

Figure 5: OPSA suppresses sampled tail tokens, redistributes mass among plausible head tokens at high-entropy positions, and preserves high-confidence predictions.

This mechanism is central to the paper’s interpretation of why OPSA improves reasoning. The method is not intended to maximize global entropy or to preserve a uniform distribution. Instead, it allocates uncertainty selectively. At low-entropy positions, concentration improves precision. At high-entropy fork positions, probability mass is redistributed among competing head tokens rather than collapsed onto a single mode. The method consequently combines local sharpening with local exploration.

The authors further associate high-entropy positions with reflective tokens such as “wait,” “but,” “however,” and “check.” OPSA increases the frequency of such tokens and produces longer trajectories. AIME24 Avg@4 increases positively with response length in the reported experiments, suggesting that the additional tokens correspond to productive reflection rather than merely verbosity.

Figure 6: OPSA increases response length and reflective-token frequency, with higher AIME24 Avg@4 associated with longer reasoning trajectories.

A fork-masking ablation strengthens this causal interpretation. When positions whose top-five candidate tokens include predefined reflective words are excluded from the OPSA objective, both response-length growth and accuracy gains largely disappear; response length collapses near training step 300. This result indicates that the high-entropy fork positions are not incidental correlates of improvement. However, the masking criterion is lexically defined and therefore does not cover all forms of reflection or branching. The experiment supports the proposed mechanism within the operational definition used by the authors, rather than establishing that every OPSA gain arises from explicit reflective language.

Empirical performance

The main experiments train Qwen3-1.7B, Qwen3-4B, and Qwen3.5-9B on DAPO-17k questions without labels. Evaluation uses AIME24, AIME25, HMMT25, MBPP+, and GPQA-Diamond, with 32 sampled responses per problem. The strongest gains occur on mathematical reasoning, especially for smaller models.

Model	Benchmark	Base Avg@32	OPSA Avg@32	Absolute gain
Qwen3-1.7B	AIME24	13.44	48.85	+35.41
Qwen3-1.7B	AIME25	9.69	35.31	+25.62
Qwen3-1.7B	HMMT25	5.73	23.33	+17.60
Qwen3-4B	AIME24	23.33	62.08	+38.75
Qwen3.5-9B	AIME24	76.35	87.81	+11.46
Qwen3.5-9B	HMMT25	44.48	67.40	+22.92

For Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263.5% relative increase. Relative gains are 264.4% on AIME25 and 307.2% on HMMT25. Pass@32 rises from 40.00% to 80.00% on AIME24, from 30.00% to 66.67% on AIME25, and from 23.33% to 50.00% on HMMT25. Thus, the method improves both average sampled accuracy and the probability that at least one of 32 samples is correct.

The gains persist at larger model scale but diminish on already strong models. Qwen3.5-9B improves by 11.46 AIME24 points and 22.92 HMMT25 points, while Pass@32 increases by 3.34 and 6.66 points, respectively. This scaling pattern is consistent with the method’s proposed operating regime: models with greater remaining uncertainty and weaker post-training have more probability mass available for useful redistribution.

OPSA also outperforms the comparison methods on Qwen3-1.7B. Averaged over AIME24, AIME25, and HMMT25, it reaches 35.83 Avg@32 and 65.56 Pass@32, compared with 24.79 and 54.44 for GRPO, 22.15 and 54.44 for OPD, and 23.58 and 56.67 for OPSD. Relative to the best baseline, OPSA gains 11.04 points in Avg@32 and 8.89 points in Pass@32.

The out-of-domain results are more modest but directionally consistent. On Qwen3-1.7B, MBPP+ increases from 58.24 to 59.44 and GPQA-Diamond from 27.92 to 32.40. These gains suggest that OPSA does not exclusively memorize the mathematical training distribution, although they are small compared with the in-domain improvements and do not establish broad task-agnostic transfer.

Diversity, entropy, and computational cost

A potential concern is that repeated suppression of low-probability tokens may cause mode collapse. The paper evaluates diversity using pairwise Jaccard distance over token-level 4-grams from 32 samples per AIME24 problem. OPSA initially changes the diversity profile, but as response length increases, its distance from the base model narrows and approaches zero. The authors interpret this as evidence that OPSA preserves long-form response diversity despite lowering aggregate entropy.

Figure 7: OPSA maintains long-form response diversity by preserving alternative branches at high-entropy reasoning forks.

The entropy analysis is especially relevant. OPSA exhibits substantially lower training entropy than GRPO and NSR while achieving higher Pass@32 on all three mathematical benchmarks. Consequently, aggregate entropy is not a sufficient proxy for exploration. Under OPSA, exploration is concentrated at selected high-entropy forks, whereas low-entropy contexts become more deterministic. This explains how a lower global entropy can coexist with preserved or improved multi-sample coverage.

The authors also control for response length. OPSA produces approximately 23,205 inference tokens for Qwen3-1.7B, compared with 19,108 for GRPO and 15,286 for OPD. Token-budget-matched controls increase GRPO and OPD response lengths to approximately 23,261 and 23,472 tokens, respectively, but their AIME24 Avg@32 scores remain 32.81 and 31.67, versus 48.85 for OPSA. The improvement therefore cannot be attributed solely to generating more tokens.

Figure 8: OPSA achieves higher Pass@32 while maintaining lower aggregate entropy than comparison methods.

OPSA is also computationally cheaper during training because it avoids teacher inference and does not require multi-response rollout groups for reward normalization. On Qwen3-1.7B, the reported step time is 46.3 seconds for OPSA, compared with 61.2 seconds for OPD and 186.2 seconds for GRPO. This comparison should be interpreted as an implementation-level measurement under the stated hardware and software configurations, not as a universal complexity result. At inference time, OPSA responses are longer: 23,205 tokens on average compared with 4,457 for the base model, and inference time rises from 1.78 to 6.58 seconds. The method therefore exchanges additional test-time computation for higher sampled accuracy.

Finally, an OPSA-trained Qwen3-4B checkpoint used to initialize GRPO improves by approximately nine validation Avg@4 points over 40 additional training steps without apparent collapse.

Figure 9: An OPSA checkpoint provides a stable initialization for subsequent GRPO training.

This cold-start experiment suggests that OPSA is not merely an alternative endpoint to reward-based RL; it can alter the optimization regime sufficiently to provide a useful initialization. The result is limited to one model family and one continuation setup, so it does not establish general complementarity with all RL algorithms.

Limitations and open questions

The experiments cover models up to 9B parameters and do not test large dense models or mixture-of-experts architectures. Whether the fixed 20% selection rule and entropy scaling remain effective at larger scale is unresolved. The method may also have limited impact on heavily post-trained models whose distributions are already sharply concentrated and have little residual entropy to redistribute.

OPSA primarily rearranges probability mass already present in the student policy. It therefore may not expand the underlying exploration frontier when correct reasoning paths have negligible probability. This limitation is reflected in the relatively modest improvements in thinking-mode Pass@32 compared with the larger non-thinking-mode gains. The method’s dependence on existing candidate branches distinguishes it from reward-driven approaches that can, in principle, reinforce a newly discovered successful trajectory.

The central mechanistic conclusion also remains partly empirical. The filtering and fixed-negative-advantage experiments show that teacher supervision is unnecessary for reproducing much of OPD’s gain under the tested conditions, but they do not fully characterize the contribution of teacher information in other OPD estimators, teacher–student calibrations, sequence lengths, or decoding regimes. In particular, the study focuses on the K1 reverse-KL estimator and should not be generalized automatically to all forms of OPD. The authors appropriately identify a need for theoretical analysis of why this estimator produces similar response-length dynamics under teacher-based and teacher-free negative updates.

Conclusion

The paper presents evidence that OPD’s observed improvements can be substantially decoupled from accurate teacher imitation. Teacher advantages on student-generated trajectories are noisy, become more correctness-insensitive with increasing teacher scale, and can be filtered or replaced without eliminating student improvement. The effective update is concentrated on low-log-probability tokens, and fixed negative advantages reproduce much of the behavior attributed to distillation.

OPSA turns this diagnosis into a practical algorithm: it suppresses low-probability sampled tokens, scales suppression with entropy, sharpens confident positions, and preserves alternatives at high-entropy reasoning forks. On Qwen3 models, this produces large gains in mathematical Avg@32 and Pass@32, modest out-of-domain improvements, lower training cost than OPD and GRPO, and no measured long-form diversity collapse. The principal open question is whether these results reflect a general property of reverse-KL on-policy optimization or a regime-specific interaction among Qwen models, non-thinking rollouts, entropy structure, and reasoning benchmarks.