Self-Distilled Policy Gradient

Abstract: On-policy self-distillation, where a LLM conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse-reward reinforcement learning. Actually, it can be instantiated as an auxiliary full-vocabulary student-to-teacher reverse Kullback-Leibler divergence loss. We therefore propose SDPG, a self-distilled policy-gradient framework that combines group-relative verifier advantages with normalized standard deviation, exact full-vocabulary on-policy self-distillation, as well as reference-policy KL regularization. Empirically, SDPG improves stability and performance over RLVR and self-distillation baselines. The code is available at https://github.com/lauyikfung/SDPG.

• The paper introduces the Self-Distilled Policy Gradient (SDPG) which fuses RLVR with full-vocabulary self-distillation to provide dense token-level rewards.
• It employs on-policy privileged distillation with adaptive gating and KL regularization to stabilize training and enhance sample efficiency on complex reasoning tasks.
• Empirical results on mathematical reasoning benchmarks demonstrate that SDPG significantly outperforms standard RLVR methods while mitigating instability and entropy collapse.

Self-Distilled Policy Gradient: Densifying RLVR Supervision via On-Policy Full-Vocabulary Self-Distillation

Introduction

Sparse reward assignment is a persistent limitation in reinforcement learning for LLMs, particularly for complex reasoning tasks such as mathematics and code generation. Although Reinforcement Learning with Verifiable Rewards (RLVR) and its current standard, Group Relative Policy Optimization (GRPO), provide sample-efficient policy optimization by exploiting outcome-based, rule-verifiable rewards, they suffer from sequence-level credit assignment that obscures token-level correctness and induces variance early in training. Recent advances in on-policy distillation—especially methods leveraging context-privileged self-supervision—offer dense, per-token signals but require careful integration with RLVR paradigms to avoid instability and mode collapse.

The Self-Distilled Policy Gradient (SDPG) framework proposes a theoretically principled and empirically robust integration of exact full-vocabulary self-distillation within RLVR, combining verifier-anchored policy gradient updates, privileged on-policy distillation losses, and explicit KL regularization to a reference policy. SDPG reinterprets self-distillation as a local policy-gradient step with a log-ratio advantage, regularized and stabilized through positive-advantage gating and adaptive distillation scheduling. This approach addresses sparse-reward and instability issues while decoupling dense privileged knowledge transfer from policy exploration.

SDPG: Theoretical Motivation and Objective

Limitations in RLVR and Motivation for Dense Supervision

GRPO and similar RLVR algorithms optimize LLMs solely on binary scalar outcomes $R(x, y)$ produced by automatic verifiers, applying a normalized reward advantage uniformly across all tokens. This token-homogeneous assignment impedes fine-grained credit propagation, limiting sample efficiency and robustness, especially for long, multi-step reasoning sequences. Extensions to step-level credit assignment remain limited by annotation costs or heuristic credit heuristics.

On-policy self-distillation methods solve credit sparsity by conditioning model outputs on privileged context (reference answers, auxiliary solutions, etc.), using the model itself as both student (deployable, unprivileged) and teacher (privileged). These approaches densify supervision through KL divergence objectives between the two distributions, increasing gradient informativeness but risking instability if not coupled with external reward signals or robust regularization.

Formulation of SDPG

SDPG fuses outcome-based RLVR objectives with a full-vocabulary reverse-KL on-policy privileged distillation term (OPD) and a reference-policy KL regularizer. Concretely, for each sampled prefix $(x, y_{<t})$, the privileged teacher $q_t$ is $\pi_\theta(\cdot~|~c, x, y_{<t})$ and the deployable student $p_t$ is $\pi_\theta(\cdot~|~x, y_{<t})$. The SDPG loss is:

$$\mathcal{L}_{\mathrm{SDPG}} = \mathcal{L}_\mathrm{out} + \beta(k)\,\mathcal{L}^+_\mathrm{OPD} + \alpha\,\mathcal{L}_\mathcal{K}(\pi_\theta, \pi_\mathrm{ref})$$

where:

• $\mathcal{L}_\mathrm{out}$: REINFORCE-style reward-based policy gradient with group-relative normalized advantages, without PPO-style clipping in the strict on-policy setting,
• $\mathcal{L}^+_\mathrm{OPD}$: Gated full-vocabulary reverse-KL student-to-teacher loss, conditioned on positive outcome advantage,
• $\mathcal{L}_\mathcal{K}$: Unnormalized KL regularization to a fixed reference policy (either forward or reverse KL variant),
• $(x, y_{<t})$0: Step-dependent coefficient with warmup/decay to modulate distillation strength,
• $(x, y_{<t})$1: Weight for reference policy regularization.

The policy-gradient interpretation of the privileged distillation follows from a gradient identity: the student-side gradient of the full-vocabulary reverse-KL is equivalent to a local policy gradient with advantage given by the centered log teacher/student likelihood ratio.

SDPG Algorithmic Design

On-Policy Self-Distillation Implementation

The privileged OPD signal is computed exactly over the full vocabulary at each token position in sampled trajectories, ensuring dense and accurate per-token supervision congruent with the student’s current state distribution. Positive-advantage gating restricts OPD supervision only to successful (positive advantage) trajectories, mitigating the risk of distilling privileged signals on unverified or incorrect rollouts.

$(x, y_{<t})$2

where $(x, y_{<t})$3 if the group-relative advantage $(x, y_{<t})$4, $(x, y_{<t})$5 otherwise.

Reference Policy Anchor

To prevent overfitting the privileged context and support controlled policy drift, SDPG includes a rollout-based unnormalized KL to a reference policy $(x, y_{<t})$6. Both unnormalized forward and reverse KL variants are supported, and corresponding surrogate loss formulations are derived for practical training.

Adaptive Stabilizers

SDPG employs a step-scheduled distillation coefficient $(x, y_{<t})$7 that grows during an initial warm-up (to avoid dominating policy updates when the privileged teacher is noisy) and decays after main training, acknowledging the irreducibility of mutual information between privileged and unprivileged distributions.

(Figure 1)

Figure 1: Training dynamics and benchmark performance on Qwen3-4B trained with baseline algorithms and SDPG variants. Top: convergence on AIME24, AIME25, AMC23; Bottom: reward, actor entropy, response length: SDPG stabilizes actor entropy and shortens verbose generations compared to RLSD and GRPO.

Empirical Results

Experimental Setup

Experiments center on the Qwen3-4B and Qwen3-1.7B LLM bases, fine-tuned on complex mathematical tasks (AIME24, AIME25, AMC23) using the DAPO-Math dataset. Privileged context is generated by Gemini 2.5 Pro, and standard RLVR baselines (GRPO, RLSD), as well as pure self-distillation (OPCD, OPSD), are included for comparison. All models are trained with AdamW, global batch size of 128, and maximum context/response lengths of 2,048/4,096 tokens.

Main Results

Both SDPG-URKL and SDPG-UFKL outperform GRPO and RLSD on all considered benchmarks, with SDPG-UFKL reaching peak pass@1 accuracy across five of six main evaluation settings. Training curves demonstrate that SDPG methods achieve superior convergence rates and early plateau on reward metrics.

SDPG-UFKL, in particular, maintains higher actor entropy throughout training, while RLSD exhibits early entropy collapse—a hallmark of mode collapse in pure self-distillation lacking outcome constraints. SDPG controls verbosity, achieving stable, intermediate response lengths supporting multi-step reasoning but without the inefficiency of GRPO-generated outputs.

Figure 2: Ablation study on SDPG-4B showing effects of removing either policy KL ($(x, y_{<t})$8) or OPD ($(x, y_{<t})$9) terms; both components are necessary for best accuracy, stability, and non-pathological entropy/length.

Ablations

Disabling the OPD term ($q_t$0) slows early convergence and degrades final accuracy, especially on difficult benchmarks, confirming the pivotal role of privileged distillation for dense credit assignment. Removing policy KL regularization ($q_t$1) destabilizes response lengths and induces entropy growth, highlighting the necessity of an anchor for coherent exploration. The combination of OPD and KL regularization is essential for stable, performant models.

Results transfer to smaller-scale models (Qwen3-1.7B): SDPG variants consistently outperform RLSD, GRPO, and OPCD, with SDPG-UFKL mitigates instability and entropy collapse observed in pure self-distillation.

(Figure 3)

Figure 3: Training and evaluation curves for Qwen3-1.7B: SDPG variants robustly avoid instability and outperform OPCD and RLSD baselines, confirming scale-transferable improvements.

Practical and Theoretical Implications

SDPG proposes a blueprint for equipping RLVR-led LLM training with full-vocabulary, privileged, on-policy distillation, giving rise to hybrid exploration-exploitation dynamics robust to reward sparsity and unstable credit propagation. The policy-gradient interpretation of OPD anchors self-distillation in established RL theory, while adaptive gating and coefficient scheduling pragmatically address noise and overfitting. Importantly, SDPG demonstrates that privileged context can be leveraged stably when modulated by verifier feedback and reference anchors, with applications in automated mathematical reasoning and any domain where dense ground truth is obtainable or synthesizable.

Beyond LLMs, SDPG’s integration strategy for privileged information and full-vocabulary self-distillation is extensible to other sequential decision-making settings, especially where credit sparsity, reward misalignment, or exploration inefficiency impede progress. Future directions include optimizing curriculum learning schedules, investigating richer privileged contexts, and scaling SDPG to even larger or more diverse foundation models.

Conclusion

SDPG advances state-of-the-art RLVR for LLM reasoning by integrating exact full-vocabulary on-policy self-distillation with verifier-grounded policy gradients and explicit policy anchoring. This design yields dense, stable, and performant supervision overcoming the limitations of both pure RLVR and naive self-distillation. Theoretical insights and empirical results validate SDPG as a preferred paradigm for stable, sample-efficient RL fine-tuning in LLM reasoning.