Self-Distilled Policy Gradient: Dense Supervision for Smarter Language Models
This presentation explores Self-Distilled Policy Gradient (SDPG), a breakthrough training method that overcomes sparse reward limitations in reinforcement learning for large language models. By fusing outcome-based policy optimization with full-vocabulary self-distillation and adaptive regularization, SDPG delivers dense per-token supervision that accelerates convergence, maintains training stability, and achieves state-of-the-art performance on complex mathematical reasoning benchmarks.Script
Training language models to solve math problems hits a wall: they only learn whether the final answer was right or wrong, with no feedback on which reasoning steps were correct. The authors of this paper asked whether a model could teach itself more effectively by learning from a privileged version of itself that has access to reference solutions.
Current reinforcement learning methods for language models rely on binary success signals at the sequence level. This sparse supervision leaves the model guessing which tokens contributed to success and which led it astray, especially in long multi-step reasoning chains where early mistakes compound.
SDPG combines three components into one training objective. First, outcome-based policy gradients anchor learning to verifier rewards. Second, full-vocabulary self-distillation provides dense per-token guidance by having the model learn from itself when given privileged context. Third, reference policy regularization prevents the model from drifting too far and maintains stable exploration.
On mathematical reasoning benchmarks including AIME and AMC, SDPG variants outperform both standard policy optimization and naive self-distillation. The method converges faster, reaches higher accuracy, and crucially avoids the entropy collapse that plagues pure distillation approaches. Response lengths stabilize at efficient levels rather than growing verbose or collapsing entirely.
Ablation studies reveal that both components are essential. Removing privileged distillation slows convergence and reduces final accuracy, confirming its role in credit assignment. Removing the reference policy anchor destabilizes training, causing uncontrolled response length growth and entropy drift. Only the full system achieves stable, performant learning.
SDPG demonstrates that language models can effectively learn from privileged versions of themselves when that self-teaching is grounded by outcome verification and policy anchors. This fusion of sparse rewards with dense self-supervision opens new directions for training more capable reasoning models. Explore the full technical details and create your own video explanations at EmergentMind.com.
