Reinforcement Learning via Self-Distillation

Abstract: LLMs are increasingly post-trained with reinforcement learning in verifiable domains such as code and math. Yet, current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck. Many verifiable environments actually provide rich textual feedback, such as runtime errors or judge evaluations, that explain why an attempt failed. We formalize this setting as reinforcement learning with rich feedback and introduce Self-Distillation Policy Optimization (SDPO), which converts tokenized feedback into a dense learning signal without any external teacher or explicit reward model. SDPO treats the current model conditioned on feedback as a self-teacher and distills its feedback-informed next-token predictions back into the policy. In this way, SDPO leverages the model's ability to retrospectively identify its own mistakes in-context. Across scientific reasoning, tool use, and competitive programming on LiveCodeBench v6, SDPO improves sample efficiency and final accuracy over strong RLVR baselines. Notably, SDPO also outperforms baselines in standard RLVR environments that only return scalar feedback by using successful rollouts as implicit feedback for failed attempts. Finally, applying SDPO to individual questions at test time accelerates discovery on difficult binary-reward tasks, achieving the same discovery probability as best-of-k sampling or multi-turn conversations with 3x fewer attempts.

• The paper introduces SDPO, a self-distillation algorithm that converts detailed tokenized feedback into dense, per-token credit assignments.
• It refines reinforcement learning by replacing scalar rewards with rich, context-specific signals, thereby accelerating convergence and enhancing task accuracy.
• Empirical results show SDPO outperforms GRPO on programming and reasoning tasks, achieving higher accuracy, faster improvements, and concise model outputs.

Reinforcement Learning via Self-Distillation: A Technical Synthesis

Motivation and Paradigm Shift

Traditional reinforcement learning (RL) for LLMs in verifiable domains, such as code and mathematical reasoning, has relied predominantly on scalar outcome rewards—formally, the Reinforcement Learning with Verifiable Rewards (RLVR) setting. While straightforward to implement, this approach exhibits a fundamental limitation: a severe credit assignment bottleneck. Scalar rewards, often binary in real-world tasks, fail to convey the detailed structure of intermediate states or errors encountered during model interaction, resulting in sparse, high-variance learning signals and impeding policy improvement.

However, many real-world environments provide substantially more informative signals. Verifiable settings such as programming contests or math solvers produce rich, tokenized feedback—runtime errors, test case failures, or detailed judge evaluations—encapsulating not just the final outcome, but granular diagnostic information about where, why, and how an attempt failed. The authors formalize this expanded protocol as Reinforcement Learning with Rich Feedback (RLRF), illustrated in (Figure 1).

Figure 1: RLVR offers only scalar reward per attempt, whereas RLRF exposes tokenized feedback, encapsulating both outcome and detailed state information for dense credit assignment.

Self-Distillation Policy Optimization (SDPO): Algorithmic Framework

SDPO is introduced as an on-policy RL algorithm that employs self-distillation for credit assignment, leveraging the model's own in-context retrospection capabilities. Rather than seeking external strong teachers or relying on explicit reward modeling, SDPO repurposes the policy as its own feedback-conditioned teacher. SDPO's procedure for a given question $x$ is:

1. Sample Original Rollouts: Answers $y$ are sampled from the current policy $\pi_\theta$.
2. Feedback Acquisition: Rich environment feedback $f$ (e.g., runtime errors, test summaries) is obtained.
3. Self-Teacher Construction: The model is re-prompted (in context) with both $x$ and $f$, forming $\pi_\theta(\cdot \mid x, f)$.
4. Distillation Loss: The student policy is trained via token-level KL divergence, aligning its next-token distribution to the self-teacher's predictions, freezing gradients for the teacher to ensure genuine retrospection.

This process assigns dense, logit-level advantages per token, efficiently identifying and correcting mistakes without requiring additional sampling or explicit external teaching. An illustrative example of logit-level credit assignment is shown in (Figure 2).

Figure 2: Dense per-token credit assignment via self-teaching—tokens where the self-teacher disagrees indicate precise locations of errors.

Theoretical Foundations and Gradient Estimation

The SDPO objective can be interpreted as maximum-entropy RL with a reward signal constructed implicitly via the self-teacher's probabilistic assessments. Let $y$ be a generated sequence, and $f$ feedback:

$$\mathcal{L}_{\mathrm{SDPO}}(\theta) = \sum_t \mathrm{KL}(\pi_\theta(\cdot \mid x, y_{<t}) \| \mathrm{stopgrad}(\pi_{\theta}(\cdot \mid x, f, y_{<t})))$$

The gradient is a logit-level policy gradient with per-token advantages defined as:

$$A_{i,t}^{\mathrm{SDPO}}(\hat{y}_{i,t}) = \log \frac{\pi_{\theta}(\hat{y}_{i,t} \mid x, f_i, y_{i,<t})}{\pi_{\theta}(\hat{y}_{i,t} \mid x, y_{i,<t})}$$

This contrasts sharply with RLVR/GRPO, where the advantage is a sequence-level scalar, constant for all tokens.

Empirical Evaluations

RLVR Environments (Without Rich Feedback)

SDPO was benchmarked against an extensively tuned GRPO baseline on tasks such as scientific question-answering and tool-API calling. Even in absence of structured environment feedback, SDPO reinterprets successful batch samples as "feedback" to failed attempts. SDPO consistently yielded higher final accuracy, faster policy improvement per wall-clock hour, and significantly conciser reasoning (up to 7× reduction in response length) compared to GRPO.

Figure 3: SDPO's logit-level self-teaching supports sparse, effective credit assignment—identifying and modifying mistakes with minimal verbosity.

RLRF Environments (With Rich Feedback)

On competitive programming tasks (LiveCodeBench v6) where LeetCode-style execution feedback is available, SDPO showed further quantitative improvements. SDPO attained higher final validation accuracy (48.8%) versus GRPO (41.2%) and required 4× fewer generations to match GRPO’s peak performance. SDPO improvements became more pronounced with larger model scales (Figure 4).

Figure 4: SDPO compresses interaction history via self-distillation—enabling efficient learning from rich context.

Test-Time Self-Distillation

SDPO was applied at test-time to hard, sparse-reward tasks, where base model pass@$k$ rates were below 0.03. In this regime, SDPO achieved the same solution discovery probability as best-of-$k$ sampling and multi-turn in-context feedback—with 3× fewer attempts. Not notably, SDPO solved instances that remained unsolved by baseline methods at high generation budgets, underscoring the practical value of iterative self-distillation.

Ablations and Analysis

Dense Credit Assignment: Ablations reveal that the granularity of credit assignment is pivotal—SDPO’s logit-level distillation outperformed token or sequence-level variants, but even sequence-level SDPO (using rich feedback) surpassed GRPO.

Scaling Behaviour: The marginal benefit of SDPO over GRPO is tightly correlated to the in-context learning abilities of the underlying model—benefits are emergent with increased model scale.

Catastrophic Forgetting: Compared to both on-policy RL and off-policy self-distillation baselines, SDPO avoided degradation of previously learned capabilities, achieving an optimal performance–forgetting tradeoff.

Hybridization: Combining GRPO and SDPO advantages benefits weaker models (where self-teacher retrospection is less reliable) but is counterproductive for stronger models, consistent with the hypothesis that dense feedback is more valuable where the policy has adequate representational capacity.

Implications and Points of Departure

SDPO provides a technically elegant, minimal modification to existing RLVR pipelines, capitalizing on feedback-rich environments without reliance on external reward models or teachers. It closes the credit assignment gap by converting rich tokenized feedback into numerous fine-grained learning signals, efficiently leveraging the inherent capacity of LLMs for in-context diagnosis and self-correction.

Practically, SDPO enables:

• Improved sample efficiency, accelerated convergence, and higher final task accuracy across reasoning and programming tasks.
• Concise, non-redundant reasoning pathways—contradicting claims that increased response length is sine qua non for better reasoning in RL training.
• Robustness to catastrophic forgetting, favorable for continual RL post-training.
• Test-time learning (specialization), efficiently discovering solutions in sparse binary-reward settings without the need for teacher intervention.

The algorithmic paradigm aligns with emerging trends in RL for LLMs: moving beyond scalar rewards to dense, structured, and domain-specific feedback, thus bridging practical RL and advanced imitation/self-correction protocols.

Limitations and Forward-Looking Directions

SDPO's performance is contingent upon two factors:

• Base Model Strength: Benefits are dependent upon sufficient in-context learning and retrospection capabilities—the retrospective self-teacher must be able to correctly diagnose and assign meaningful credit.
• Quality of Feedback: For maximal efficacy, environment feedback must be informative—uninformative or misleading feedback can stall or mis-guide optimization.

Potential avenues for future research include extensions to agentic, partially observable domains (POMDPs), exploring scaling properties with larger models and longer context windows, adaptation to open-ended or continuous-reward tasks, and a principled investigation into behavioral and reasoning differences induced by self-distillation.

Conclusion

The formalization of RLRF and introduction of SDPO provides a significant refinement in RL post-training regimes for LLMs, demonstrating that tokenized environment feedback can be exploited for dense credit assignment using self-distillation. Empirically, SDPO achieves superior wall-clock sample efficiency, final accuracy, and concise policy representations in reasoning-heavy tasks, with gains increasing with model scale. These findings support the continued exploration of feedback-rich RL protocols and emphasize leveraging in-context introspection and self-supervision inherent in advanced LLMs.