Self-Distilled RLVR

Abstract: On-policy distillation (OPD) has become a popular training paradigm in the LLM community. This paradigm selects a larger model as the teacher to provide dense, fine-grained signals for each sampled trajectory, in contrast to reinforcement learning with verifiable rewards (RLVR), which only obtains sparse signals from verifiable outcomes in the environment. Recently, the community has explored on-policy self-distillation (OPSD), where the same model serves as both teacher and student, with the teacher receiving additional privileged information such as reference answers to enable self-evolution. This paper demonstrates that learning signals solely derived from the privileged teacher result in severe information leakage and unstable long-term training. Accordingly, we identify the optimal niche for self-distillation and propose \textbf{RLSD} (\textbf{RL}VR with \textbf{S}elf-\textbf{D}istillation). Specifically, we leverage self-distillation to obtain token-level policy differences for determining fine-grained update magnitudes, while continuing to use RLVR to derive reliable update directions from environmental feedback (e.g., response correctness). This enables RLSD to simultaneously harness the strengths of both RLVR and OPSD, achieving a higher convergence ceiling and superior training stability.

• The paper introduces RLSD, a technique that decouples token credit direction from magnitude to address instability and leakage in on-policy self-distillation.
• It leverages privileged teacher signals solely for credit magnitude estimation while using external rewards for update direction, ensuring training stability and improved accuracy.
• Empirical results on multimodal reasoning benchmarks show RLSD outperforms baselines with a notable accuracy gain and robust token-level credit assignment.

Self-Distilled RLVR: A Unified Framework for Stable and Fine-Grained Credit Assignment in Policy Optimization

Introduction

This work presents a formal diagnosis of information leakage and instability in on-policy self-distillation (OPSD) for reasoning LLMs, and introduces Reinforcement Learning with Self-Distillation (RLSD): a method that structurally resolves the deficiencies of OPSD by decoupling the direction and magnitude of token-level credit during policy optimization. Specifically, RLSD retains the external verifier reward as the sole arbiter of update directions while exploiting the dense, privileged signal available from self-distillation exclusively for intra-trajectory credit magnitude modulation. The theoretical framework formalizes the mutual information-induced irreducible gap in OPSD and supports these findings with ablation and diagnostic experiments. RLSD is empirically validated on the Qwen3-VL-8B-Instruct model and multiple multimodal reasoning benchmarks.

The Structural Failure of On-Policy Self-Distillation

OPSD operates by letting the same model act as both teacher (with privileged information, e.g., a reference answer) and student (without). Unlike conventional On-Policy Distillation (OPD) which uses an external teacher observing only the student-accessible context, the teacher in OPSD leverages additional privileged signals, making the information flow asymmetric.

A critical empirical finding is that OPSD-trained models exhibit progressive privileged information leakage—explicitly referencing unseen data at test time—as well as deteriorating performance after initial rapid gains. Figure 1 confirms this by quantifying leakage frequency, the stagnation of KL divergence between student and teacher, and associated validation accuracy degradation.

Figure 1: Leakage, KL divergence, and validation performance of OPSD and its ablated variants, demonstrating growing leakage and performance collapse with training.

The underlying structural issue is formalized as an irreducible mutual information gap in the OPSD loss, $I(Y_t; R \mid X, Y_{<t})$, reflecting the part of the teacher's token-level distribution that is fundamentally unmatchable for the student due to lack of access to $r$. This residual term contaminates gradient estimates with $r$-specific deviations, driving the model to encode spurious $x \to r$ mappings.

Crucially, this pathology arises in all distribution-matching configurations under information asymmetry, as proven in the paper. Attempts to compress the target signal—e.g., Teacher's Top-1, Student's Top-1—only modulate the bandwidth of leakage, not its irreducibility, which is quantifiable and directly observed in diagnostics.

RLSD: Decoupled Direction and Magnitude Credit Assignment

RLSD is proposed to harness the dense supervision available in OPSD but while entirely eliminating leakage. The core idea is to anchor update directions exclusively to reliable environment reward via standard RLVR (e.g., GRPO), while using the privileged teacher signal strictly for assessing relative per-token magnitude of credit.

The RLSD mechanism is as follows:

1. For each on-policy trajectory, compute both student and privileged (teacher) conditional log-probabilities per token.
2. The difference, $\Delta_t = \log P_T(y_t) - \log P_S(y_t)$, is used as a stop-gradient signal to modulate step sizes, not directions.
3. Per-token weights, $w_t = \exp(\text{sign}(A)\cdot\Delta_t)$, are applied, with $A$ being the environment-anchored sequence-level advantage. Clipping is used for trust-region stabilization, bounding any single token's influence.
4. The overall gradient update thus uses the same policy gradient structure as GRPO for direction, but credits tokens proportionally to their information gain under privileged context.

A diagrammatic overview of RLSD’s workflow appears below.

Figure 2: An overview of the RLSD method, showing the computation pipeline for integrating self-distillation in policy optimization.

This design is grounded in a Bayesian interpretation. The per-token evidence ratio $P_T(y_t)/P_S(y_t)$ exactly quantifies, under mild conditions, the Bayesian update in belief that the privileged reference is correct upon generating $y_t$, ensuring that credit is logically attributed to tokens advancing or impeding correct reasoning.

Empirical Results and Analysis

RLSD attains strong improvements over both RLVR (GRPO) and all self-distillation baselines on five challenging multimodal reasoning datasets using Qwen3-VL-8B-Instruct. In Table 1, RLSD achieves the best average accuracy (56.18%), outperforming the base LLM by 4.69% and GRPO by 2.32%. The largest relative gains manifest on datasets requiring fine-grained reasoning, such as MathVista and MathVision.

The learning curve comparison in Figure 3 demonstrates the superior training stability and convergence ceiling of RLSD.

Figure 3: RLSD achieves both the stable training dynamics of GRPO and a higher convergence ceiling, while maintaining faster initial convergence than OPD-based baselines.

Moreover, RLSD avoids the rapid entropy collapse seen in vanilla GRPO, instead maintaining higher exploration by focusing credit on decisively informative tokens rather than uniformly suppressing alternatives (as shown in Figure 4).

Figure 4: Training dynamics on multimodal reasoning, highlighting RLSD's faster ascent and robust entropy control compared to OPSD and GRPO.

A token-level credit analysis (Figure 5) corroborates that RLSD attributes maximal credit to tokens directly controlling correct logical transitions and maximal blame to tokens responsible for reasoning errors or misinterpretations.

Figure 5: RLSD’s token-level credit heatmaps reveal that decisive reasoning steps receive the strongest positive or negative credit, matching true logical attributions.

Theoretical Guarantees and Practical Implications

RLSD is proven immune to information leakage both at the gradient and support level: the stop-gradient mechanism ensures that privileged information modulates only the scalar magnitude, not direction, of parameter updates. Consequently, RLSD resolves the trilemma that constrains all distribution-matching self-distillation methods, being the only known framework (up to our knowledge) to simultaneously achieve:

• On-policy training,
• High credit assignment granularity,
• Structural leakage immunity,
• Sustoained improvement without objective drift or reward model dependence,
• Compatibility as a drop-in extension of standard RLVR.

The theoretical analysis further generalizes to show that attempts to “fix” OPSD by periodically freezing or synchronizing parameters only trade off between deadlocked capacity or instability, never eliminating leakage unless distribution matching is abandoned.

Implications for Future AI Systems

RLSD offers an efficient and principled resolution for fine-grained credit assignment in RLVR without incurring the computational cost of maintaining a large teacher or a step-level reward model. The self-distillation module provides actionable guidance for future research in scaling stable and data-efficient RLVR, particularly in domains where post-training relies on verifiable outcomes rather than dense human preferences.

It suggests a structural template for separating direction and magnitude signals in other domains where partial privileged context may be available (e.g., demonstration-guided reasoning compression, multimodal environments with latent variables, or curriculum learning pipelines), generalizing beyond multimodal LLMs.

Conclusion

Self-distilled RLVR (RLSD) solves the key instability and leakage issues of on-policy self-distillation for LLMs reasoning by rigorously decoupling direction (reward-aligned) and magnitude (privileged discrepancy) in token-credit assignment. The result is a stable, efficient, and theoretically grounded framework that achieves both reliability and granularity in policy optimization, as confirmed by strong empirical performance across multiple challenging reasoning settings.

For further details and proofs, see "Self-Distilled RLVR" (2604.03128).