RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation

Abstract: On-policy self-distillation (OPSD) provides dense, token-level supervision for reasoning models by aligning a model's own distribution with the distribution it produces under privileged context, typically a verified solution. However, we show that the learning signal drawn from this distributional gap concentrates on style tokens rather than task-bearing ones, as the hinted model tends to produce more direct, shorter outputs. We term this pathology \emph{privilege-induced style drift}, which destabilizes training or causes response length to shrink. To address this, we propose \textbf{RLCSD} (Reinforcement Learning with Contrastive on-policy Self-Distillation), which mitigates this drift by contrasting the teacher-student gap under a correct hint against that under a wrong hint, suppressing the style shift that conditioning on a hint tends to induce regardless of correctness, and yielding a signal that is more concentrated on task-bearing tokens. Experiments on Qwen3 (1.7B/4B/8B) and Olmo-3-7B-Think across mathematical and logical reasoning show that RLCSD consistently outperforms GRPO and prior OPSD methods. We further show that the contrastive principle is general: it plugs into existing OPSD methods to improve them, and its underlying insight extends to the broader cross-model on-policy distillation setting.

• The paper introduces a contrastive self-distillation method that replaces one-sided hints with symmetric supervision to mitigate privilege-induced style drift.
• It employs K-marginalized negative hints and verifier-anchored modulation to ensure robust credit assignment and stable training.
• Empirical results show significant improvements in reasoning tasks, with enhanced out-of-distribution performance and alignment on correctness-bearing tokens.

Reinforcement Learning with Contrastive On-Policy Self-Distillation: Methods, Insights, and Implications

Motivation and Background

On-policy self-distillation (OPSD) provides dense, token-level supervision for reasoning models by aligning a model's policy with the distribution it produces when conditioned on privileged context, such as a verified solution. While OPSD offers a practical alternative to traditional on-policy distillation (OPD)—which requires white-box access to an external teacher model, shared vocabularies, and incurs substantial compute overhead—existing OPSD approaches exhibit a significant pathology: privilege-induced style drift. Here, the supervised signal becomes dominated by stylistic tokens (e.g., connectives, formatting, discourse markers), rather than correctness-bearing tokens relevant to the task, destabilizing training and leading to issues such as response length shrinkage or entropy explosion.

Figure 1: Top-10 tokens by $|e_c|$ and $|e_{\mathrm{ctr}}|$ (left: style-dominated vs content-dominated signal), response length collapse (middle), and benchmark performance across baselines (right).

To address these limitations, RLCSD (Reinforcement Learning with Contrastive On-Policy Self-Distillation) introduces a core methodological innovation: replacing the conventional one-sided, privileged hint signal with a contrastive signal obtained by conditioning on both correct and incorrect reference solutions, thus directly mitigating style drift at the token-level.

Methodological Advances

Privilege-Induced Style Drift Diagnosis

Empirical vocabulary partitioning on math reasoning tasks demonstrates that, under existing OPSD methods, $|e_c|$ (the privileged teacher-student log-prob gap) is overwhelmingly concentrated on stylistic tokens (mean: $0.263$) as opposed to task-relevant tokens (mean: $0.083$), leading to undesirable optimization behavior.

Contrastive Self-Distillation Objective

RLCSD introduces a symmetric token-level supervision signal:

$$e_{\text{ctr},t} = [\log \pi_T(y_t|x, y^*_c, y_{<t}) - \log \pi_S(y_t|x, y_{<t})] - [\log \pi_T(y_t|x, y^*_w, y_{<t}) - \log \pi_S(y_t|x, y_{<t})]$$

Here, $y^*_c$ and $y^*_w$ are correct and incorrect sampled rollouts, respectively. Both are wrapped in identical prompt templates, promoting shared stylistic context and isolating correctness-related information. Empirical analysis shows that this subtraction sharply reduces the style/task token gap, increasing the alignment of learning with the solution's substantive correctness.

Figure 2: Overview of RLCSD three-stage pipeline: (1) sampling and verifying rollouts, (2) symmetric prompt construction for correct and incorrect hints, and (3) contrastive token-level signal computation and advantage modulation.

Key Refinements:

• K-marginalized negative hints: Averaging the negative (incorrect) hint log-probabilities over $K$ sampled hints increases robustness to error-type heterogeneity among incorrect solutions.
• Exclusion of self-hints: Preventing a rollout from being used as its own hint reduces spurious over-confidence in the signal.
• Verifier-Anchored Modulation: The contrastive signal modulates, rather than replaces, the outcome-level advantage from GRPO, ensuring optimization directionality remains anchored in verifiable correctness.
• Two-Path Loss Aggregation: Independent normalization of modulated (meaningfully contrastive) tokens preserves their contribution and prevents dilution by the majority of unmodulated (background) tokens.

Figure 3: Entropy evolution during training; RLCSD avoids both entropy explosion (instability) and length shrinkage witnessed in prior OPSD approaches.

Empirical Results and Analysis

Main Benchmarks

RLCSD achieves consistent improvements over GRPO and all prior OPSD baselines (OPSD, SDPO, SRPO, RLSD) across Qwen3 (1.7B/4B/8B) and Olmo-3-7B models for both mathematical and logical reasoning (including OOD splits). Gains are particularly pronounced on out-of-distribution generalizations, e.g., +21.0 points on 11-role Knight-and-Knaves for Qwen3-8B.

• Stability:
  • RLCSD maintains stable entropy and response length throughout training.
  • Competing methods either exhibit entropy/length instability or degenerate into short-answer/trivial rater-matching.
• Fine-grained Credit Assignment:
  • By contrastively isolating correctness-bearing tokens, RLCSD meaningfully assigns credit to intermediate reasoning steps even in long-horizon settings.
  • 

Figure 4: Training entropy for math reasoning; only RLCSD and GRPO avoid unbounded growth and collapse.

Figure 5: Rollout-level mean of $e_{\text{ctr},t}$ across hinting strategies, confirming effectiveness of K-marginalization in stabilizing signal directionality.

Generality of the Contrastive Principle

Ablation experiments demonstrate that contrastive self-distillation is method-agnostic—plugging the contrastive construction into other OPSD variants (e.g., OPSD, RLSD) directly improves their performance and training robustness. The benefit is observed regardless of whether hints are drawn from dataset CoTs or self-sampled rollouts.

• Contrast mitigates both entropy explosion and premature length shrinkage failure modes.

Ablations: Design Choices

• Removal of K-marginalized negatives or exclusion of self-hints degrades performance on both reasoning and logical testbeds.
• Removing verifier-anchoring (relying solely on the contrastive signal for per-token advantage) leads to severe decrease in performance, indicating the fundamental importance of maintaining outcome-level verifier guidance.
• Global loss averaging (instead of two-path) heavily dilutes the effect of the contrastive signal.

  Figure 6: Distribution of teacher token probabilities under different hint sources, confirming the necessity of excluding self-rollouts from the hint pool to avoid overconfidence.

Broader Insights for Cross-Model On-Policy Distillation

Analysis extends to OPD with external teachers (cross-model). When the teacher and student styles are mismatched, the reverse-KL signal can become dominated by surface form tokens, resulting in stylistic mimicry rather than true capability transfer. The task/style decomposition applies regardless of whether privileged hints are used. Per-token KL distribution offers a practical diagnostic for teacher selection in OPD, and suggests that the contrastive principle could motivate improved objectives for model-to-model distillation.

Implications and Future Directions

Practical Implications:

• RLCSD is a drop-in replacement for previous RLVR+OPSD strategies in the post-training pipelines of reasoning-intensive LLMs, offering better generalization, greater stability, and finer credit assignment for complex reasoning chains.
• The contrastive method is computationally efficient; the teacher-side forward pass overhead remains small compared to overall rollout generation.

Theoretical Perspective:

• RLCSD sharpens the distinction between stylistic register and true reasoning capability in token-level learning signals.
• The construction of contrastive supervision by symmetric cancellation may provide a general motif for future work on both self-distillation and cross-model distillation in RLHF and RLVR settings, where confounding stylistic effects are present.

Outlook:

• Future research might generalize contrastive distillation beyond simple correct/incorrect splitting, e.g., via hierarchical or soft-grading of intermediate reasoning steps.
• Developing diagnostics and automated teacher/hint selection protocols based on per-token KL/task decomposition could further enhance transfer fidelity in both self and cross-model distillation setups.

Conclusion

RLCSD demonstrates that dense token-level supervision derived from contrastive self-distillation—anchored with outcome-level verifier signal—overcomes the stability and credit assignment bottlenecks in OPSD. It systematically mitigates privilege-induced style drift, shifts learning toward correctness-bearing tokens, and yields substantial improvements in both in-domain and out-of-distribution generalization. The generality of the contrastive principle—valid across various OPSD instantiations—suggests a broader methodological avenue for future advances in distillation and post-training for large reasoning models.