Rethinking Privileged Information in On-Policy Self-Distillation

Abstract: On-policy self-distillation (OPSD) trains a student on its own responses using token-level supervision from the same model conditioned on privileged reference information. We investigate whether performance gains from OPSD show that the student learned the information in the reference or instead reflect recovery of reasoning behavior already present in the base model. We perform OPSD experiments on science and mathematics datasets using Qwen3 models ranging from 1.7B to 8B. Our analysis framework separates the supervision induced by the reference from the supervision provided by the teacher without the reference and measures how each aligns with changes in the student's predictions. The correct reference does not provide a consistent performance benefit across teacher generation modes, model sizes, and training datasets. Students can improve without the correct reference, and a solution from another problem can outperform the correct solution on several mathematical reasoning benchmarks. The student's predictions align more strongly with the base model's thinking behavior than with the supervision induced by the reference, but controls constructed from other problems reproduce much of both alignments. Moreover, stronger alignment attributable to the correct reference does not reliably coincide with a greater performance benefit from the reference. Performance gains and distributional alignment alone therefore cannot determine how privileged reference information contributes to student learning in OPSD.

• The paper systematically tests on-policy self-distillation across Qwen3 models from 1.7B to 8B on math and science tasks, finding that correct reference solutions provide no consistent performance advantage.
• Mismatched references frequently outperform correct ones, including a 67.6 versus 59.5 average on NuminaMath benchmarks at 8B, showing that reference benefits depend on teacher mode, model scale, and dataset.
• Distributional analyses show student updates align strongly with recovery of latent base-model reasoning behavior, while control contexts reproduce most apparent reference alignment and fail to establish genuine information transfer.

On-policy self-distillation (OPSD) trains a student model on its own sampled trajectories while a frozen teacher—instantiated from the same base model—provides token-level supervision conditioned on privileged information, typically a reference solution. The method's central premise is that the reference makes the teacher's supervision more informative and that this information transfers to the student. This paper subjects that premise to systematic scrutiny across Qwen3 models at 8B, 4B, and 1.7B scales, trained on NuminaMath and MegaScience subsets (2608.18271). Its central finding is negative: neither benchmark performance nor distributional alignment measurements can establish what role privileged reference information actually plays in student learning.

Background and motivation

OPSD generalizes on-policy distillation (OPD), which addresses the exposure bias of sequence-level distillation by evaluating the teacher at states visited by the student's current policy and minimizing token-level divergence between their next-token distributions (Agarwal et al., 2023). In OPSD, the same model serves as both teacher and student under different contexts: the student conditions on the problem alone, while the teacher also receives a reference solution unavailable at inference time. Prior work already indicates complications—excessive reference exposure can degrade reasoning behavior (Kaur et al., 6 Jul 2026), rich teacher context can suppress uncertainty and impair out-of-distribution generalization (Kim et al., 25 Mar 2026), and problem-specific privileged content may not transfer to a student lacking it at inference (Zhu et al., 11 May 2026). Concurrent work reports that mismatched references can preserve or improve OPSD performance (Ichihara et al., 10 Aug 2026) and that likelihood changes induced by privileged context do not automatically constitute useful token credit (Nguyen et al., 10 Aug 2026).

The paper asks three questions: (i) whether the correct reference is necessary for improvement, (ii) whether the reference must correspond to the problem being solved, and (iii) whether gains reflect learning from the reference or recovery of reasoning behavior latent in the base model.

Analysis framework

The methodological contribution decomposes total teacher supervision into two directions in log-probability space. For each position $t$ of a fixed student-generated response, the total difference between the teacher-with-reference distribution $T$ and the student distribution $S_c$ splits as

$$\log T - \log S_c = \underbrace{(\log T - \log N)}_{D_{\mathrm{ref}}} + \underbrace{(\log N - \log S_c)}_{D_{\mathrm{rec}}},$$

where $N$ is the reference-free teacher distribution. $D_{\mathrm{ref}}$ isolates the supervision induced by the reference; $D_{\mathrm{rec}}$, the "recovery" direction, captures the remaining gap between the reference-free teacher and the initial student. A third direction, $D_{\mathrm{think}}$, measures the shift induced by enabling thinking mode without a reference. All metrics are computed on responses generated by the initial student and held fixed across checkpoints, so only the student distribution varies during training.

Crucially, every comparison is repeated against controls constructed by replacing the correct teacher context with context from another problem. If alignment between the student's change $\Delta S_c$ and a teacher direction survives this substitution, it cannot be attributed to the correct problem-specific reference. The paper measures projection coefficients, cosine similarities (the primary directional measure, since magnitudes vary under control substitution), the fraction of $\lVert \Delta S_c\rVert^2$ captured by the span of teacher directions ($T$0), and KL divergence from the reference-free teacher. Uncertainty comes from paired bootstrap over problems, verified across three different control pairings.

Performance results

The headline performance findings contradict the premise that correct reference information drives OPSD gains:

• Adding the correct reference to a thinking teacher provides no consistent advantage at 8B. On NuminaMath, thinking-teacher students with and without the reference average 59.4 versus 60.3; on MegaScience, 60.7 versus 60.8.
• A solution from another problem can outperform the correct one. Under a non-thinking teacher at 8B, the mismatched reference yields 67.6 average on NuminaMath benchmarks versus 59.5 for the correct reference, with paired 95% intervals excluding zero on AIME 2025 (+12.5) and HMMT 2025 (+11.7). The mismatched reference wins all three competition mathematics benchmarks at 4B and 1.7B as well.
• The benefit of the reference depends jointly on teacher mode, scale, and dataset. At 4B on NuminaMath, the correct reference improves competition-math scores under a thinking teacher; at 1.7B on MegaScience, removing the reference improves all three science benchmarks (49.3 vs. 44.8 average).

These results imply that the reference functions less as transferable problem-specific knowledge than as a generic conditioning signal whose value is contingent on configuration—an implication consistent with the concurrent finding of Ichihara et al. (Ichihara et al., 10 Aug 2026), but established here across three model scales and two domains.

Distributional analysis

The distributional experiments address whether the student learns from the reference or recovers base-model behavior. At training step 0, the reference accounts for a substantial share of total teacher supervision (projection coefficient $T$1 of 0.46 on NuminaMath, 0.39 on MegaScience), confirming that the reference does alter the teacher's predictions even where it confers no performance benefit. Yet three observations complicate any inference from this alteration to learning:

First, the student's change aligns more strongly with the base model's thinking behavior than with the reference-induced supervision. Under a thinking teacher, cosine similarity with the recovery direction exceeds similarity with the reference direction on both datasets. Because a non-thinking teacher shares the student's mode, this pattern suggests movement toward behavior associated with the base model's thinking regime rather than acquisition of reference-specific content.

Second, controls reproduce much of the apparent alignment. Replacing the correct reference with one from another problem yields nearly identical trajectories for both evaluated-problem and control directions. At step 150 under a non-thinking teacher, the difference between cosine similarities with $T$2 and its control is actually negative (−0.23 ± 0.03 on NuminaMath; −0.38 ± 0.04 on MegaScience). At checkpoint 300 under a thinking teacher, additional alignment attributable to the correct reference is modest (cosine differences of +0.05 and +0.10), and $T$3 values for control directions match or exceed those for evaluated-problem directions (0.94–0.98 vs. 0.88–0.97). A high fraction of student change lying in the teacher-direction span therefore does not indicate dependence on the correct context.

Third, alignment does not predict performance benefit. Across scales, the largest reference-vs.-control cosine difference occurs at 1.7B on MegaScience (+0.40), yet the no-reference student outperforms the with-reference student there on all three science benchmarks. Conversely, larger cosine gaps at 4B and 1.7B on NuminaMath coincide with reference benefits. Additional analyses show that the sign of raw cosine similarity flips across scales without determining relative-to-control alignment, that KL divergence from the reference-free teacher grows steadily while overall change magnitude plateaus, and that entropy collapses identically regardless of reference presence under a non-thinking teacher.

Limitations

The authors are explicit about scope constraints. Experiments use only Qwen3-family models; generalization to other families is untested. Training covers mathematical and scientific reasoning only, leaving coding and other domains unexamined. Each condition is trained once, so run-to-run variance is not measured—a material concern given that several reported performance differences are small. Checkpoint selection uses two held-in selection benchmarks before final evaluation, introducing a mild selection bias favoring the compared methods equally but not zero. The scale-ordering observation (increasing reference-control cosine difference at smaller sizes) is explicitly descriptive rather than causal.

Conclusion

This paper demonstrates that the standard evidence used to justify privileged-information distillation is insufficient. Correct reference solutions provide inconsistent, configuration-dependent benefits; mismatched solutions frequently perform better; and controls built from unrelated problems reproduce most measured alignment between student updates and both reference-induced and thinking-mode supervision. Stronger reference-attributable alignment does not track stronger performance benefit. The open question left by the work is what measurement—if any—can isolate genuine information transfer in self-distillation settings where teacher and student share parameters and generation machinery, since both performance deltas and distributional projections conflate reference-specific learning with recovery of behaviors latent in the base model.