Contrastive On-Policy Distillation

Abstract: On-policy Distillation (OPD) supervises a student model on trajectories sampled from its own policy by minimizing the divergence between the output distributions of the teacher and student at each token position, thereby providing dense token-level supervision. Although existing OPD methods have demonstrated strong performance in improving the reasoning ability of student models, their objectives fundamentally rely on token-level distribution matching. Consequently, they lack an explicit signal for comparing a token's relative compatibility across reasoning modes and thus do not directly model preferences between these modes. To address this limitation, we propose COPD, a contrastive OPD framework. Specifically, for each token generated by the student model, a frozen teacher model scores the same student state under two contrasting instructions that elicit light and heavy reasoning. The difference between the resulting log probabilities serves as a token-level advantage signal to guide the OPD update. Rather than merely imitating a single teacher distribution, COPD directly encourages the student model to learn more concise and efficient reasoning strategies. We conduct experiments on nine multimodal benchmarks covering both reasoning and understanding tasks. The results show that COPD substantially reduces reasoning length without compromising model performance and consistently improves efficiency across different tasks and model scales. Furthermore, the contrastive formulation can be seamlessly integrated into the On-policy Self-distillation (OPSD) framework, where self-contrastive supervision is constructed without an additional teacher model, thereby enabling the model to distill itself toward lightweight reasoning.

• The paper introduces COPD, a contrastive on-policy distillation method that compares teacher likelihoods under light- and heavy-thinking prompts to create token-level advantages without length penalties or answer labels.
• The method improves both accuracy and efficiency across nine multimodal benchmarks, including a 2.1 percentage-point accuracy gain, 57.0% shorter responses, and a 141.2% Acc@1K increase over ExOPD in one setting.
• The paper shows that the contrastive reference is essential: light-thinking likelihood alone lengthens outputs, while COPD also supports self-distillation, smaller datasets, and substantial training-cost reductions of up to 60%.

Motivation and problem statement

On-policy distillation (OPD) supervises a student model on trajectories sampled from its own policy, with a frozen teacher scoring each token at student-visited prefixes. This yields dense token-level supervision while avoiding the exposure mismatch of offline distillation. The authors identify a structural limitation: at any given prefix, standard OPD matches the student to a single teacher distribution, which measures token plausibility under one context but provides no relative information about whether that token advances concise versus exhaustive reasoning. Existing reasoning-compression methods—concise-trace selection, preference pairs, token budgets, length penalties—inject compression signals through external objectives or curated targets, whereas OPD-based methods inherit this single-context ambiguity.

The paper proposes Contrastive On-Policy Distillation (COPD), which replaces single-distribution imitation with a paired conditional comparison. For every token generated by the student, the frozen teacher scores the identical (input, prefix, token) triple under two instructions eliciting light-thinking (LT) and heavy-thinking (HT) reasoning. The log-likelihood difference serves as a detached, clipped token-level advantage in a PPO-style policy-gradient update:

$$A_t = \log \pi_\phi(y_t \mid x^{LT}, y_{<t}) - \log \pi_\phi(y_t \mid x^{HT}, y_{<t})$$

with $$\tilde A_t = \operatorname{stopgrad}(\operatorname{clip}(A_t, -10, 10))$$. Notably, the signal contains no explicit length penalty; shortening emerges because tokens that advance, summarize, or directly answer receive positive advantage relative to tokens compatible with continued deep deliberation. Training uses ViRL-16K (a filtered subset of ViRL-39K retaining only samples with intermediate pass rates), implemented in verl with FSDP and vLLM rollouts, batch size 192, three epochs, and no accuracy rewards, KL rewards, or formatting constraints.

Main results

Evaluation covers nine multimodal benchmarks—four reasoning (MathVista, MathVerse, WeMath, LogicVista) and five understanding (HalluBench, MMMU, MMStar, RWQA, MMGist)—across four teacher–student configurations built on Qwen3-VL-2B and Qwen3.5-2B students. Three metrics are reported: accuracy, average response length, and Acc@1K (accuracy per thousand output tokens).

The headline finding is that COPD dominates the accuracy–length frontier rather than trading cost for accuracy. In all four settings it achieves both the highest average accuracy and the shortest responses. Against ExOPD in the Qwen3-VL-8B→2B setting, COPD improves average accuracy by 2.1 pp while cutting response length by 57.0%, more than doubling Acc@1K (+141.2%). This pattern holds across configurations, with COPD consistently more than doubling Acc@1K relative to ExOPD.

Setting	Method	AVG Accuracy / Tokens / Acc@1K
Qwen3-VL-8B→2B	ExOPD	55.2 / 2.07K / 26.7
Qwen3-VL-8B→2B	COPD	57.3 / 0.89K / 64.4
Qwen3-VL-32B→2B	ExOPD	56.9 / 1.92K / 29.6
Qwen3-VL-32B→2B	COPD	57.1 / 0.79K / 72.3
Qwen3.5-4B→2B	ExOPD	57.6 / 3.66K / 20.3
Qwen3.5-4B→2B	COPD	60.1 / 1.92K / 46.4
Qwen3.5-35A3B→2B	ExOPD	56.9 / 3.77K / 18.7
Qwen3.5-35A3B→2B	COPD	59.3 / 2.06K / 44.3

Compression is broad but not uniform. Relative to the base model, COPD shortens outputs on all nine benchmarks and improves accuracy on eight, with the largest gains on WeMath (+11.3 pp at 60.7% fewer tokens) and LogicVista (+11.4 pp at 59.1% fewer tokens). RWQA is the exception: accuracy drops 0.9 pp as already-short outputs shrink by 76.9%, exposing a boundary condition—tasks with minimal initial responses leave little compressible margin.

A further result contradicts a simple teacher-scaling heuristic. Doubling down on teacher size does not monotonically improve outcomes: moving from the 8B to the 32B Qwen3-VL teacher slightly reduces accuracy (−0.2 pp) while improving efficiency, and scaling Qwen3.5 from 4B to 35A3B reduces accuracy by 0.8 pp and lengthens responses by 7.3%. The authors attribute the recurring benefit to the LT–HT relative preference itself rather than teacher capacity, consistent with prior analysis that distillation requires compatible thinking patterns between teacher and student (Li et al., 14 Apr 2026).

Ablations

Scale generalization. With the 35A3B teacher fixed, growing the student from 4B to 9B preserves the effect (41.6% and 36.8% length reductions, >58% Acc@1K gains). Conversely, replacing the 35A3B teacher with the larger 122A10B teacher for a fixed 9B student adds 0.8 pp accuracy but lengthens responses by 24.1% and lowers Acc@1K by 18.6%, reinforcing that teacher scale shifts the operating point rather than improving efficiency.

Data generalization. Training on Geo3K (only 2K samples) for three epochs still improves average accuracy by 1.5 pp and cuts length by 22.4%. Under a matched iteration budget (Geo3K for 24 epochs vs. ViRL-16K for 3 epochs), ViRL-16K retains an edge of 0.9 pp accuracy, 12.7% shorter responses, and 16.2% higher Acc@1K—broader coverage helps, but repeated training on small data captures much of the benefit.

Prompt robustness. Simple, complex, and default LT/HT templates yield similar results (accuracy 56.0–57.3%, lengths 0.82–0.92K), indicating the gains stem from the contrastive construction rather than specific wording.

Self-distillation (COPSD). Replacing the external teacher with a frozen snapshot of the initial student preserves the effect across scales: the 2B model gains 3.5 pp accuracy with 63.8% shorter responses (+194.4% Acc@1K); the 122A10B model gains 1.7 pp with 17.6% shorter responses. The authors interpret this as policy reselection rather than knowledge acquisition—concise correct paths already exist in the model's distribution (Wu et al., 11 Feb 2025), and long traces often contain redundant self-verification after the answer is reached.

The contrast is essential. A decisive ablation replaces $A_t$ with the light-thinking likelihood alone ($A_t = \ell_t^{LT}$). This yields essentially no accuracy gain (52.9% → 53.1%) and actually lengthens outputs (2.10K → 2.26K). Only the light-minus-heavy difference produces the intended behavior (57.3% accuracy, 0.89K tokens). This confirms the paper's central claim: the heavy-thinking score functions as a necessary reference that converts absolute plausibility into a directional preference, without which light-context likelihood alone assigns high scores to broadly plausible—including redundant—tokens.

Training efficiency

COPD requires 60 GPU hours in the 8B→2B setting, versus 132 for OPD and 150 for ExOPD—a 54.5–60.0% reduction—while achieving the best accuracy and shortest outputs. Response-length trajectories show monotone shortening below 1K tokens under COPD, whereas OPD and ExOPD remain flat at high lengths, supporting the interpretation that the contrastive advantage suppresses redundant continuations without an explicit penalty. A case study on MathVerse illustrates the mechanism: the distilled model produces 44.2% of the base model's tokens by eliminating redundant checks while correctly identifying key geometric relations.

Limitations and open questions

The paper concedes several boundaries. Compression is not uniformly beneficial: RWQA loses accuracy when near-minimal outputs are compressed further, suggesting the method needs compressible redundancy to work with. Teacher scaling behaves non-monotonically, so practitioners cannot assume a larger teacher improves the trade-off. The self-distillation result, while strong, rests on the assumption that the useful signal derives from context-induced relative preference rather than new knowledge—an interpretation supported indirectly by cited analyses rather than proven mechanistically within the paper. Finally, the authors explicitly note that extending the paired-context contrast beyond reasoning modes—to safety alignment, style control, or grounded generation—and handling interactions among multiple such preferences remain untested open questions.

Conclusion

COPD reframes on-policy distillation supervision from single-distribution matching to state-matched contrast between reasoning-mode contexts. The token-level LT–HT advantage supplies dense compression preferences without target lengths, length penalties, or answer labels, yielding simultaneous accuracy and efficiency gains across nine benchmarks, multiple teacher–student configurations, and a teacher-free self-distillation variant. The ablation isolating the heavy-thinking reference as indispensable substantiates the paper's core argument: efficient distillation can rely on distinguishing useful reasoning progress from redundant continuation, rather than enforcing brevity externally.