Embarrassingly Simple Self-Distillation Improves Code Generation

Abstract: Can a LLM improve at code generation using only its own raw outputs, without a verifier, a teacher model, or reinforcement learning? We answer in the affirmative with simple self-distillation (SSD): sample solutions from the model with certain temperature and truncation configurations, then fine-tune on those samples with standard supervised fine-tuning. SSD improves Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6, with gains concentrating on harder problems, and it generalizes across Qwen and Llama models at 4B, 8B, and 30B scale, including both instruct and thinking variants. To understand why such a simple method can work, we trace these gains to a precision-exploration conflict in LLM decoding and show that SSD reshapes token distributions in a context-dependent way, suppressing distractor tails where precision matters while preserving useful diversity where exploration matters. Taken together, SSD offers a complementary post-training direction for improving LLM code generation.

• The paper presents a simple self-distillation strategy that leverages a model’s own outputs to enhance code generation.
• It employs controlled decoding techniques, using temperature and truncation during data synthesis followed by standard supervised fine-tuning.
• Empirical results demonstrate significant improvements on hard benchmarks, with gains in performance and maintained diversity in solution outputs.

Embarrassingly Simple Self-Distillation for Code Generation: Algorithmic, Empirical, and Theoretical Insights

Introduction and Motivation

"Embarrassingly Simple Self-Distillation Improves Code Generation" (2604.01193) introduces Simple Self-Distillation (SSD), a paradigm for post-training improvement of LLMs on code generation, leveraging only the model's own raw outputs—absent any external verifier, teacher, reward model, or reinforcement learning. SSD generates solutions with controlled decoding (temperature, truncation), then performs standard supervised fine-tuning (SFT) on these unfiltered outputs. This method is algorithmically minimalistic but achieves nontrivial gains, most notably on challenging code generation benchmarks and in the performance regime where further progress is costly via traditional means requiring expensive human labeling or complex RL workflows.

Methodology: Simple Self-Distillation (SSD)

SSD operates in three straightforward phases:

1. Data Synthesis: Given a frozen pre-trained model $p_\theta$ and a prompt set $\mathcal{X}$, sample one solution per prompt using decoding at temperature $T_{\text{train}}$ and truncation parameters $\rho_{\text{train}}$ (top-$k$/top-$p$), producing the dataset $\mathcal{D}_{\mathrm{SSD}}$. No correctness filtering is used.
2. Supervised Fine-Tuning: Fine-tune the model on the synthetic dataset $\mathcal{D}_{\mathrm{SSD}}$ with standard cross-entropy loss, as in SFT. No modification of the training target or weighting is performed beyond what emerges from the raw self-sampled outputs.
3. Inference: Evaluate using the fine-tuned model $p_{\theta^*}$ with optimized decoding configuration $(T_{\text{eval}}, \rho_{\text{eval}})$.

This procedure differs fundamentally from self-training protocols relying on explicit correctness signals, reward shaping, or privileged teacher models.

Figure 2: The vLLM decoding pipeline used for sampling in SSD, showing temperature scaling, top-$\mathcal{X}$0 and top-$\mathcal{X}$1 filtering, and Gumbel-max sampling.

Empirical Results

SSD demonstrates robust improvements across five models in the Llama and Qwen families (scales 4B–30B, covering both "instruct" and "thinking" variants), using the LiveCodeBench~v6 (LCB~v6) and v5 benchmarks. Notable findings include:

• Qwen3-30B-Instruct: Pass@1 rises from 42.4% to 55.3% (+12.9pp, +30.4% relative); pass@5 surges from 53.5% to 71.6%. Gains are especially pronounced on the "hard" subset, with pass@1 increasing by +15.3pp and pass@5 by +23pp.
• Generalization: All five models tested improved, establishing SSD’s applicability across architectures and scales.
• Hard Example Advantage: SSD’s relative improvement intensifies with problem difficulty, indicating acquisition of new or more robust solution strategies rather than simply memorizing easier patterns.
• Diversity Preservation: Gains in pass@5 typically exceed those in pass@1, indicating that SSD does not collapse diversity and may even enhance coverage of viable solution modes.

  Figure 3: SSD outperforms the best base-model decoding sweep, with greater margins for harder problems and top-$\mathcal{X}$2 generation.

(Figure 2) is referenced the first time above, displayed immediately after.

Critically, SSD’s empirical improvements cannot be attained by optimal tuning of the base model's global decoding parameters $\mathcal{X}$3. The temperature/truncation sweep curves of the frozen model are nearly flat, with SSD consistently outperforming the strongest base configurations on every difficult metric. This establishes that SSD induces intrinsic model changes not achievable via inference-time policies alone.

Precision-Exploration Conflict: Theoretical Mechanism

The core mechanism underlying SSD is the precision-exploration conflict in next-token prediction distributions during code generation:

• Locks (Precision Contexts): Positions in code where continuation is unambiguous (e.g., closing a syntactic construct or following a function signature). Here, most mass ought to be concentrated on a single token; temperature or aggressive truncation helps by suppressing long distractor tails.
• Forks (Exploration Contexts): Tokens at which multiple structurally distinct and plausible continuations exist (e.g., choice of algorithm or control structure), necessitating diversity in the token distribution to achieve broad solution coverage.

A fixed global decoding temperature cannot simultaneously optimize both—colder decoders eliminate useful forks, while hotter decoders reintroduce distractor mass at locks.

SSD, through fine-tuning on outputs sampled under nontrivial temperature and truncation, reshapes the model’s token distributions contextually: suppressing long tails at locks, but preserving flattened, diverse heads at forks.

Figure 4: A single global evaluation temperature cannot satisfy both exploration at forks and precision at locks; low temperature collapses fork diversity, while high temperature revives lock distractors.

This effect has been precisely observed both in a theoretical toy FSM and in real models:

Figure 5: SSD turns forks into plateaus—preserving several top alternatives—and locks into spikes—suppressing distractors and sharpening the dominant mode.

Figure 6: Real-model evidence shows SSD compresses distractor tails and makes evaluation-time temperature more effective for top alternatives.

Theoretical Analysis: Objective Decomposition

SSD’s cross-entropy loss, after data synthesis via temperature/truncation-altered sampling, decomposes into three interpretable terms at each context:

1. Support Compression ($\mathcal{X}$4): Pushes mass onto the truncated support, eliminating tail tokens.
2. Within-Support Reshaping ($\mathcal{X}$5): Adjusts entropy within the retained support, typically flattening the head for $\mathcal{X}$6 (Renyi entropy of order $\mathcal{X}$7).
3. KL Alignment: Keeps reshaped distribution anchored to the base model on the supported set.

This is a context-adaptive reshaping not attainable by fixed decoding posteriors. The result is a student model whose induced distributions at locks are sharper and more temperature-inert (lower entropy, less susceptible to evaluation-time over-exploration), while fork distributions retain broader, more uniform heads—enabling effective exploration without reviving unwanted distractors.

Temperature interaction analysis empirically confirms that training and evaluation temperatures compose to produce an effective temperature regime ($\mathcal{X}$8), but truncation is necessary to raise the performance ceiling.

Figure 1: SSD hyperparameter sweeps: effective temperature governs performance without truncation; truncation further raises achievable pass@1.

Analysis of Data Quality and Robustness

An unexpected finding is SSD’s resilience to pathological training data: when SSD is performed using extremely high temperature ($\mathcal{X}$9) without truncation (where most outputs are syntactically or semantically invalid), the model still improves substantially. The improvement persists so long as per-token distributional reshaping occurs, and appropriate truncation is used at evaluation. This invalidates explanations based solely on copying correct self-samples and implicates the underlying distributional transformation as the operative effect.

Figure 8: In the high-temperature, no-truncation regime, SSD-trained model still achieves gains even though the majority of synthetic data is gibberish.

Practical and Theoretical Implications

Practical Implications

• SSD offers a minimal, infrastructure-light post-training route to nontrivial code generation improvement without additional supervised data or complex pipelines (RL, verifiers, external teachers). This substantially lowers the operational barrier for capability unlocking in code LLMs, especially in domains with scarce gold labels.
• The gains concentrate where they are most valuable: on hard and previously uncovered problems, and in increased solution diversity as measured by pass@5.
• Out-of-domain performance is preserved at large scale (30B models), though smaller models may incur uneven tradeoffs, highlighting the need for further study in cross-domain robustness.

Theoretical Implications

• The necessity of contextually adaptive distributional reshaping, as realized by SSD but not by fixed decoding, challenges the sufficiency of decode-only or inference-time policies for unlocking the latent potential of LLMs.
• The success of SSD even with low-quality or high-temperature data underscores the role of distributional support geometry and entropy regularization over explicit correctness—a result compatible with recent findings on entropy-driven RL for LLM reasoning (Agarwal et al., 21 May 2025, Cui et al., 28 May 2025).
• Theoretically, SSD can be understood as support-adaptive, context-conditioned Renyi entropy regularization plus tail suppression, with aligned KL anchoring to the base.
• The precision-exploration conflict is a general mechanism, and the SSD framework provides a tractable approach to resolve it without domain-specific engineering.

Limitations and Future Directions

While SSD provides practically appealing gains, several open questions and potential directions arise:

• Transfer and Generality: The impact of repeated or multi-stage SSD (e.g., iterated self-distillation) on both in-domain and out-of-domain generalization requires systematic analysis, particularly regarding entropy collapse or mode dropping in longer training horizons.
• Optimal Decoding Policy Synthesis: There is an opportunity to design context-aware decoding or sampling meta-policies, possibly integrating signals from SSD-induced distributional diagnostics.
• Extensions Beyond Code Generation: Applying SSD to non-code LLMs in reasoning, mathematics, or dialogue may lead to task- or domain-specific adaptations of the objective, potentially leveraging explicit or intrinsic uncertainty estimation.
• Relation to RL and Unsupervised Objectives: Connections to entropy minimization, intrinsic motivation, and RL with internal signal (RLVR) could be further elucidated, especially in regimes with sparse ground-truth supervision.

Conclusion

SSD provides compelling evidence that a code LLM can materially improve its own performance across multiple metrics, difficulty regimes, and model scales, solely by re-training on its own unfiltered outputs with temperature/truncation-modified decoding. The mechanism—support compression and within-support entropy regularization—relieves the fundamental precision-exploration tension, unlocking capabilities otherwise inaccessible through conventional scoring, RL, or supervised distillation. These results challenge the necessity of complex or fragile external feedback protocols for further LLM self-improvement in code, and establish SSD as a strong baseline and methodological foundation for future work.

---

References

• "Embarrassingly Simple Self-Distillation Improves Code Generation" (2604.01193)
• "The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning" (Agarwal et al., 21 May 2025)
• "The Entropy Mechanism of Reinforcement Learning for Reasoning LLMs" (Cui et al., 28 May 2025)