From Generic Correlation to Input-Specific Credit in On-Policy Self Distillation

Abstract: On-policy self-distillation has emerged as a promising paradigm for post-training LLMs, in which the model conditions on environment feedback to serve as its own teacher, providing dense token-level rewards without external teacher models or step-level annotations. Despite its empirical success, what this reward actually measures and what kind of credit it assigns remain unclear. Under a posterior-compatibility interpretation of feedback conditioning, standard in the implicit-reward literature, we show that the self-distillation token reward is a Bayesian filtering increment whose trajectory sum is exactly the pointwise mutual information between the response and the feedback given the input. This pMI can be raised by input-specific reasoning or by input-generic shortcuts, so we further decompose the teacher log-probability along the input axis. Based on this analysis, we propose CREDIT (Contrastive REward from DIsTillation), which isolates the input-specific component with a batch-contrastive baseline. At the sequence level, CREDIT is a teacher-side surrogate for a contrastive pMI objective that also penalizes responses remaining likely under unrelated inputs. Across coding, scientific reasoning, and tool-use benchmarks on two model families, CREDIT delivers the strongest aggregate performance at negligible additional compute.

• The paper introduces Credit, a method that subtracts generic reward components to focus on input-specific credit in self-distillation processes.
• It formalizes the reward as a Bayesian filtering increment, linking token rewards to pointwise mutual information between responses and feedback.
• Empirical results show that Credit enhances training stability and performance across reasoning, coding, and tool-use benchmarks compared to vanilla self-distillation.

Input-Specific Credit Assignment for On-Policy Self-Distillation

Introduction and Problem Statement

On-policy self-distillation (OPSD) is a thriving paradigm in reinforcement learning (RL) fine-tuning for LMs, leveraging dense token-level rewards derived from model-environment interactions, typically without requiring an external teacher or step-level process supervision. Despite empirical success, the underlying semantics and limitations of the self-distillation reward, particularly regarding input specificity versus input-generic bias, are underexplored. This paper formalizes the reward structure of OPSD, identifying that it acts as a Bayesian filtering increment whose trajectory sum recovers pointwise mutual information (pMI) between the response and feedback, conditioned on the input. Crucially, the pMI signal can be elevated both by genuinely input-specific reasoning and by input-generic patterns persisting across inputs—leading to reward misassignment.

To resolve this, the authors propose Credit: a batch-contrastive reward correction that isolates input-specific credit by subtracting input-generic components, yielding a reward signal better concentrated on problem-relevant tokens.

Figure 1: Overview of Credit's contrastive reward structure, isolating input-specific credit by subtracting a generic baseline using contrastive input samples.

Theoretical Framework

Bayesian Filtering and Token-Level Mutual Information

Under posterior compatibility, the self-distillation token reward is demonstrated to correspond to the Bayesian filtering increment between feedback-conditioned and unconditioned log-likelihoods. The central equality presented is: $$r_t(\hat{y}_t) = \log \frac{\pi(\hat{y}_t \mid x, y_{<t}, z)}{\pi(\hat{y}_t \mid x, y_{<t})} = Q_t^z(\hat{y}_t, x) - V_{t-1}^z(x)$$ where the summation across all positions gives the sequence-level reward as the pointwise mutual information: $$\sum_t r_t(y_t) = \log \frac{P_\pi(z \mid x, y)}{P_\pi(z \mid x)} = \mathrm{pMI}_\pi(y; z \mid x)$$ Consequently, the self-distillation reward measures informativeness with respect to observed feedback, with every vocabulary token's advantage reflecting its marginal contribution to making the feedback $z$ predictable.

Input-Generic vs. Input-Specific Reward Decomposition

A key insight is that high token-level pMI can be attributable both to input-specific deductive steps and generic sequence fragments that correlate with successful feedback across inputs. The decomposition of teacher log-probabilities is: $$\log \pi_{\text{ref}}(\hat{y}_t \mid x, y_{<t}, z) = S_t(\hat{y}_t, x) + G_t(\hat{y}_t)$$ where $S_t$ captures input-specific reasoning and $G_t$ the input-generic (shortcut) component. The Credit reward subtracts this generic baseline, estimated via contrastive averaging over batch-sampled alternate inputs, thus: $$R_t(\hat{y}_t) = r_t(\hat{y}_t) - \frac{\lambda}{C} \sum_{k=1}^C \log \pi_{\text{ref}}(\hat{y}_t \mid x'_k, y_{<t}, z)$$ This design ensures only those tokens whose reward signal vanishes when the input changes receive significant positive credit.

Figure 2: Visualization of raw self-distillation reward (near-uniform across tokens) vs. Credit's input-specific signal, where problem-relevant tokens are highlighted and shortcuts are suppressed.

The Credit Algorithm

Credit executes $C$ additional reference-model forward passes per training example, each time conditioning feedback and the generated response on different contrastive inputs sampled from the batch (excluding the current input). The average teacher likelihood under these mismatched contexts estimates $G_t$. For efficiency, per-token rewards are computed only for high-probability tokens (top-$K_v$). Empirically, $$\sum_t r_t(y_t) = \log \frac{P_\pi(z \mid x, y)}{P_\pi(z \mid x)} = \mathrm{pMI}_\pi(y; z \mid x)$$0 suffices, as higher $$\sum_t r_t(y_t) = \log \frac{P_\pi(z \mid x, y)}{P_\pi(z \mid x)} = \mathrm{pMI}_\pi(y; z \mid x)$$1 values give diminishing returns for compute spent.

Empirical Results and Analysis

Performance on Reasoning, Coding, and Tool-Use Benchmarks

The effectiveness of Credit is assessed on LiveCodeBench v6, SciKnowEval, and ToolAlpaca, comparing base models Qwen3-8B and OLMo-3-7B. Baselines include GRPO (scalar reward) and SDPO (vanilla self-distillation).

• Aggregate Performance: Credit consistently yields the highest average scores across domains, particularly outperforming in settings where vanilla self-distillation exhibits significant input-generic noise.
• Calibration and Efficiency: Credit delivers more stable training dynamics, with shorter model responses and lower output entropy, indicating strong regularization of reward attribution.

  Figure 3: LiveCodeBench training trajectories; Credit outpaces SDPO and GRPO in validation accuracy, stabilizing sequence length and entropy.

Qualitative Analysis

Token-level visualizations clarify Credit's practical impact: while raw self-distillation indiscriminately rewards both templates and problem-specific content, Credit's signal is sharply focused, reinforcing tokens directly related to input reasoning.

(Figures 7–13)

Figures 7–13: Credit's input-specific signal concentrates on key entities and suppresses boilerplate or misframed steps across a variety of coding benchmarks, consistently aligning reward with correct reasoning chains.

Ablations

• Contrastive Sample Count: Increasing $$\sum_t r_t(y_t) = \log \frac{P_\pi(z \mid x, y)}{P_\pi(z \mid x)} = \mathrm{pMI}_\pi(y; z \mid x)$$2 (number of contrastive samples) does not yield meaningful improvements beyond $$\sum_t r_t(y_t) = \log \frac{P_\pi(z \mid x, y)}{P_\pi(z \mid x)} = \mathrm{pMI}_\pi(y; z \mid x)$$3, due to rapid variance reduction in the batch-averaged baseline.
• Debiasing Strength $$\sum_t r_t(y_t) = \log \frac{P_\pi(z \mid x, y)}{P_\pi(z \mid x)} = \mathrm{pMI}_\pi(y; z \mid x)$$4: Moderate values ($$\sum_t r_t(y_t) = \log \frac{P_\pi(z \mid x, y)}{P_\pi(z \mid x)} = \mathrm{pMI}_\pi(y; z \mid x)$$5) maximize performance. Excessive debiasing collapses the input-specific signal, underscoring the need for balance.
• Self-Teacher Context: Including richer context in the self-teacher (e.g., solution traces as well as thinking traces) further enhances Credit's reward alignment and training stability.

  Figure 4: Over-debiasing (large $$\sum_t r_t(y_t) = \log \frac{P_\pi(z \mid x, y)}{P_\pi(z \mid x)} = \mathrm{pMI}_\pi(y; z \mid x)$$6) monotonically diminishes both Credit's input-specific signal and benchmark accuracy.

  

  Figure 5: Ablation of self-teacher context highlights that solution-only or full (think+solution) contexts maintain stability, while think-only collapses.

Theoretical Implications and Limitations

The Bayesian filtering interpretation elucidates the implicit reward model realized by self-distillation: maximizing sequence-level pMI. However, this quantity is not robust to reward-shaping artifacts introduced by input-generic regularities, justifying the need for input-specific decompositions. Credit, through a contrastive teacher-side surrogate, approximates the ideal of maximizing contrastive pMI with an additional anti-genericity bonus (penalizing responses that are not surprising under unrelated inputs).

Practically, Credit achieves improved reward localization with negligible additional computation, as contrastive inputs are sampled from within-batch problems. Theoretically, this work offers a new framework for decomposing implicit credit signals in RL for LMs.

Potential Extensions and Future Work

The current formulation assumes sufficient batch diversity and focuses on single-turn tasks with structured feedback. Logical extensions include:

• Exploring adaptive debiasing strength ($$\sum_t r_t(y_t) = \log \frac{P_\pi(z \mid x, y)}{P_\pi(z \mid x)} = \mathrm{pMI}_\pi(y; z \mid x)$$7) schedules.
• Systematic investigation of negative sampling strategies for baseline estimation.
• Application to multi-turn agent tasks and other policy-ratio implicit reward settings.
• Larger-scale studies beyond 7–8B parameter models and evaluation on less structured or human-in-the-loop feedback.

Conclusion

Credit provides a theoretically principled and empirically validated approach to correcting on-policy self-distillation rewards in LLMs, isolating input-specific credit from confounding input-generic shortcuts. This contrastive adjustment yields improved RL fine-tuning efficiency and response quality in tasks demanding genuine reasoning and alignment with input-specific feedback, without requiring external teacher models or expensive step-level annotation. The implications for RL fine-tuning protocols are substantial: Credit offers a path towards more faithful, efficient credit assignment for autonomous and continual learning in LMs.

Figure 6: Empirical validation of the theoretical assumption underpinning Credit’s Bayesian filtering interpretation using projected compatibility checks at the answer position.