SFT Overtraining Predicts Rank Inversion via Entropy Collapse Under RLVR

Abstract: The standard heuristic of selecting the SFT checkpoint with the highest pass@1 for GRPO can fail when SFT compresses the rollout distribution. For binary rewards, the expected within group advantage variance is $p(1{-}p)(g{-}1)/g$; when early GRPO drives $p$ below $p^*(g)$, most groups have identical rewards and provide no group relative signal. We study SFT depth ladders for Qwen2.5-Coder-3B and DeepSeek-Coder-6.7B. We test Qwen2.5-Coder-3B across five depths and three seeds, and DeepSeek-Coder-6.7B across four matched depths and three seeds. On Qwen, pre RL pass@1 rises with SFT depth, but peak GRPO pass@10 falls from $0.806$ to $0.481$ (3 seed mean, $n{=}20$); pre RL entropy is positively associated with the GRPO outcome ($ρ{=}{+}0.69$). On DeepSeek, pass@1 remains far above $p^*(8){=}0.083$, and GRPO outcomes compress rather than invert. A two stage diagnostic, combining pre RL entropy triage with an early GRPO entropy monitor, flags high risk checkpoints and can stop failing runs early. Simple KL to reference regularisation and label smoothing variants do not rescue the collapsed Qwen checkpoint in our setting, suggesting the failure is not a trivial GRPO hyperparameter artefact.

• The paper demonstrates that deeper supervised fine-tuning, while boosting pass@1 scores, paradoxically degrades RLVR performance via entropy collapse.
• It introduces an analytic framework linking low output entropy to a collapse in within-group advantage variance, which nullifies effective reinforcement signals.
• Empirical evaluations reveal that entropy-based diagnostics can recover substantial RL performance, challenging conventional checkpoint selection methods.

SFT Overtraining, Entropy Collapse, and Rank Inversion in RLVR: An Expert Analysis of (2606.18487)

Introduction

The paper "SFT Overtraining Predicts Rank Inversion via Entropy Collapse Under RLVR" (2606.18487) investigates the interplay between supervised fine-tuning (SFT) depth and subsequent reinforcement learning with verifiable rewards (RLVR) in code generation models. The work focuses on two families of code LLMs—Qwen2.5-Coder-3B and DeepSeek-Coder-6.7B—scrutinizing the standard checkpoint selection heuristic for RL initialization based on SFT pass@1, the effect of overtraining on entropy, and the resultant impact on RLVR trainability under the group relative policy optimization (GRPO) framework.

Connection Between SFT Overtraining and RLVR Failures

The core finding is a systematic failure of the widely used heuristic of selecting the highest-scoring SFT checkpoint (by pass@1) for RLVR initialization. The authors demonstrate that for binary-reward RLVR regimes (such as code correctness verification), deeper SFT leads to higher pass@1 prior to RL, but paradoxically results in worse RLVR performance—as evidenced by a monotonic drop in peak GRPO pass@10 from 0.806 to 0.481 across SFT depths on Qwen. Notably, this is more than a superficial metric inversion: it reflects a collapse in output distribution entropy, which suffocates the gradient signal that GRPO relies on for effective optimization.

They formalize this phenomenon via an analytic framework: for group size $g$ and success probability $p$, the expected within-group advantage variance is $p(1-p)(g-1)/g$. When SFT overtraining compresses the model’s output diversity such that $p$ approaches a threshold $p^*(g)$, variance vanishes, rendering advantage-based RL updates ineffectual. This "entropy collapse" translates to the majority of GRPO groups producing no learning signal and the model ceasing to improve—effectively rank inversion where lower-SFT checkpoints are strictly better RL initializers.

Empirical Findings and Model Regime Boundaries

The authors present comprehensive empirical evaluations across the SFT depth ladder on both Qwen2.5-Coder-3B and DeepSeek-Coder-6.7B. On Qwen, the pre-RL entropy was found to be a strong direct predictor of RL outcome (Spearman $\rho=+0.69$), with the standard pass@1 metric being inversely correlated ($\rho=-0.75$) with post-RL capability. This inversion was observed in every random seed and SFT depth tested. In contrast, DeepSeek-Coder-6.7B exhibited a rank compression regime where all SFT checkpoints maintained pass@1 well above $p^*(g)$, thus retaining sufficient entropy and avoiding signal collapse; RL outcomes became bunched, and no inversion occurred.

These results robustly separate two regimes:

• Rank Inversion (Qwen2.5-Coder-3B): SFT overtraining saturates pass@1 but pushes entropy below the safe threshold, causing RLVR failure via gradient death.
• Rank Compression (DeepSeek-Coder-6.7B): At all SFT depths, entropy remains sufficient, post-RL performance is directly predicted by pre-RL outcomes, and no catastrophic collapse is observed.

The transition between these regimes is predicted precisely by the threshold $p^*(g)$ derived analytically.

Diagnostic Protocol and Interventions

The authors operationalize their findings into a two-stage diagnostic protocol:

1. Pre-RL Entropy Triage: Apply a simple next-token entropy probe at rollout temperature ($T=1.0$), flagging checkpoints as high risk if entropy falls below a calibrated threshold (e.g., $p$0 nats for the Qwen ladder).
2. Early GRPO Entropy Monitor: During the first 150 GRPO steps, monitor relative entropy drop; severe early collapse further flags runs likely to fail.

This protocol enables reliable early detection of high-risk SFT checkpoints, preventing wasted RL compute. Notably, substituting entropy triage for the standard pass@1 criterion recovers up to +0.090 absolute in deep eval pass@10 (0.643 vs. 0.553, $p$1, $p$2 seeds).

Two common regularization interventions—KL to reference and label smoothing—were empirically ineffective at rescuing collapsed (low-entropy) checkpoints. Interestingly, label smoothing increased entropy but failed to improve RL trainability, illustrating that entropy alone is necessary but not sufficient if it does not reflect mass concentrated on correct continuations.

Theoretical Mechanism and Broader Implications

The entropy collapse mechanism is articulated via the following self-reinforcing cycle:

• Prolonged SFT compresses the policy; diversity vanishes ($p$3 low entropy).
• Under RLVR, stochastic rollouts become homogeneous, pass@1 collapses below $p$4.
• The within-group advantage variance collapses; the policy gradient vanishes.
• Optimization stalls, and residual momentum further erodes capability.

This mechanism is not rescued by rollout count scaling (increasing $p$5), as $p$6 only scales logarithmically.

Broader theoretical implications include the linkage between entropy loss and model plasticity. The inability to recover from entropy collapse is analogous to non-recoverable loss of plasticity in deep networks studied elsewhere. Entropy-preserving methods (e.g., network resets, regularization toward initial weights) and more sophisticated SFT-to-RL bridges (e.g., PEAR, Clip-Cov) are hypothesized as promising complements, deserving dedicated empirical study.

Limitations and Future Research

The theoretical derivations in this work assume binary rewards and GRPO. Practical RLVR scenarios featuring richer reward signals, partial credit, or non-binary correctness, as well as other RL formulations (PPO, RLOO), may deviate from these results. Furthermore, coverage is restricted to two code LLMs; scaling to other domains, model sizes, and RL settings remains to be established. Future research should empirically test entropy-preserving SFT/RL interventions, investigate cross-domain generality, and extend model selection protocols to account for process/task-specific entropy profiles.

Conclusion

This work provides a rigorous critique of the standard SFT checkpoint selection heuristic for RLVR, demonstrating that SFT overtraining can actively sabotage RL performance via entropy collapse—and that "better" SFT pass@1 scores can mislead model selection. The analytic and empirical evidence strongly supports pre-RL entropy as a more precise, low-cost predictive metric for RLVR outcomes under GRPO. For practitioners designing code generation LLMs or similar RLVR pipelines, adopting entropy-based diagnostics for checkpoint selection is a concretely actionable recommendation. Moreover, the broader theoretical linkage between fine-tuning, output entropy, and RL trainability lays a foundation for methodologically robust future research into post-SFT RL for LLMs.