H$^2$SD: Hybrid Hindsight Self-Distillation

Abstract: Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of LLMs on tasks such as mathematical reasoning and code generation. However, most RLVR methods assign a scalar outcome reward to an entire trajectory, resulting in sparse supervision and limited token-level credit assignment. On-policy distillation (OPD) provides denser supervision by distilling token-level distributions from a stronger teacher model, but requires an additional teacher and typically assumes a shared vocabulary. On-policy self-distillation (OPSD) removes this dependency by conditioning the same model on privileged information to construct a teacher policy. However, directly matching the teacher distribution may cause information leakage and unstable optimization. RLSD avoids direct matching by using the teacher signal only to modulate update magnitudes, but it cannot provide an explicit correction direction when the sampled reasoning fails. To address this tradeoff, we introduce $\mathrm{H}<sup>{2}\mathrm{SD}$, a hybrid hindsight self distillation framework that uses the teacher differently according to trajectory correctness. For successful trajectories, the teacher receives the student response confirmed as correct together with a rephrasing instruction, and its probabilities on the original response tokens are used to modulate update magnitudes without changing the direction determined by the reward. For failed trajectories, we condition the teacher on a reference hint containing key reasoning steps and a verified answer, and minimize the reverse KL divergence from the student to the teacher. Experiments on multiple challenging reasoning benchmarks show that H$<sup>2$SD consistently outperforms representative RLVR, OPSD, and RLSD baselines while maintaining stable optimization and favorable generation efficiency.

• The paper introduces H²SD, a correctness-conditioned method that uses hindsight magnitude weighting for successful trajectories and hint-guided reverse-KL distillation for failed ones.
• H²SD achieved 50.49% overall accuracy versus 24.85% for the strongest baseline, with especially large gains on Sudoku and consistent improvements on unseen Arrow Maze sizes.
• The paper shows that hybrid routing, detailed process-level hints, and response rephrasing are critical, while reversed routing causes entropy collapse and performance below the base model.

Motivation and problem setting

Reinforcement learning with verifiable rewards (RLVR), exemplified by GRPO, has become the dominant paradigm for improving LLM reasoning, but it assigns a single scalar outcome reward to an entire trajectory. This yields sparse supervision and poor token-level credit assignment: all tokens in a response receive the same advantage magnitude regardless of their individual contribution. On-policy distillation (OPD) addresses this by supervising student rollouts with a stronger teacher's token-level distributions, but it requires an external teacher that typically shares the student's vocabulary. On-policy self-distillation (OPSD) removes this dependency by conditioning the same model on privileged information (e.g., ground-truth answers) to construct a self-teacher, yet directly matching such a teacher risks privileged-information leakage and unstable optimization. RLSD sidesteps leakage by using the privileged signal only to modulate token-level update magnitudes while letting the reward determine direction — but for failed trajectories it merely reweights already-sampled errors without indicating which alternatives should be preferred.

The paper identifies this as a genuine trade-off between distribution-level correction and stable credit assignment, and proposes Hybrid Hindsight Self-Distillation (H$^2$SD), which resolves it by routing each trajectory according to its verified correctness.

The H$^2$SD method

H$^2$SD conditions the choice of update rule on trajectory correctness $m=\mathbf{1}[R(x,y)=1]$. For successful trajectories, the reward already supplies a reliable optimization direction; the self-teacher is given the student's own verified-correct response together with a rephrasing instruction, and its token probabilities are used solely to compute RLSD-style magnitude weights $w_t = (P_T(y_t)/P_S(y_t))^{\operatorname{sign}(A)}$, clipped and mixed into the GRPO advantage via $$\widehat{A}_t = A[(1-\lambda)+\lambda\operatorname{clip}(w_t)]$$. For failed trajectories, where the sampled reasoning path is unreliable, H$^2$SD instead minimizes reverse KL divergence from the student distribution $p^{S}_{t,\theta}$ to the hint-conditioned teacher $p^{T,h}_{t,\theta}$ (with stop-gradients through the teacher):

$$\mathcal{L}_{\mathrm{RKL}} = \frac{1}{T}\sum_{t=1}^{T} D_{\mathrm{KL}}\left(p^{S}_{t,\theta} \,\|\, \operatorname{sg}[p^{T,h}_{t,\theta}]\right).$$

The overall objective combines both branches with weight $^2$0: $^2$1.

A second design decision concerns the privileged information itself. Answer-level signals are too coarse for complex reasoning tasks, so an external strong model (Kimi-K2.6 in experiments) generates natural-language hints containing key intermediate steps and a verifier-confirmed answer. Crucially, the hint generator never acts as a teacher and need not share the student's tokenizer; the teacher distribution is always produced by the current policy conditioned on the hint. Since OPSD, RLSD+hint, and H$^2$2SD consume identical hints, performance differences isolate how privileged information is converted into learning signals rather than what information is provided.

Main results

Experiments use Qwen3-30B-A3B-Instruct-2507 on three logical-reasoning benchmarks (Sudoku 6×6/8×8, Calcudoku 5×5/6×6, Arrow Maze across five grid sizes up to held-out 8×8 and 10×10), implemented in VERL with SGLang on four nodes of eight H200 GPUs. To make full-vocabulary OPSD tractable over long chains, exact probabilities are retained only for the student's top-100 tokens with tail mass merged into a bucket.

Method	Sudoku avg.	Calcudoku avg.	Arrow Maze avg.	Overall
Base LLM	19.25	35.00	15.60	22.28
GRPO	21.50	40.00	15.70	24.68
SDPO	12.38	13.17	15.90	13.46
RLSD	21.50	41.00	15.40	24.85
OPSD	26.00	30.83	13.20	24.01
SRPO	21.88	38.50	13.80	24.01
RLSD+hint	21.13	40.83	15.80	24.72
H$^2$3SD	66.88	44.00	24.20	50.49

The headline result is stark: H$^2$4SD more than doubles the overall average of every baseline (50.49 vs. 24.85 for the best baseline), with the largest gains on Sudoku (76.50% on 6×6 versus roughly 27–35% for all baselines). Because all hint-consuming methods share identical privileged information, these gains must be attributed to the objective structure rather than information content. H$^2$5SD also ranks first on every Arrow Maze size, including unseen sizes, indicating generalization beyond training configurations. Notably, SDPO underperforms even the base model, which the authors attribute to misaligned sibling-conditioned supervision suppressing necessary reasoning steps. An accuracy–cost analysis shows H$^2$6SD attains its highest pass@1 with the fewest generated tokens, so improvements do not come from longer chains of thought.

Ablations

Routing matters decisively. Applying magnitude modulation alone or reverse-KL alone to all trajectories yields substantially worse results than hybrid routing, and reversing the routing assignment (distillation on successes, magnitude on failures) collapses performance below the base model everywhere (e.g., 17.00% vs. 76.50% on Sudoku-6×6). Reversed Routing also drives actor entropy rapidly to zero, evidencing early collapse of exploration. This is the paper's strongest causal evidence that the success/failure split, not merely access to hints, drives the gains.

Privileged context quality matters. Holding the recipe fixed, programmatic verifier feedback barely helps (25.50% on Sudoku-6×6) and ground-truth answers help modestly (27.50%), whereas process-level contexts are far more effective: sibling solutions reach 61.75%, and external hints perform best at 76.50%. Fine-grained, directional corrective signals appear essential for failed-trajectory distillation.

Rephrasing interacts with the update rule. Adding rephrasing to Magnitude Only improves Sudoku-6×6 by 23.75 points and Sudoku-8×8 by 32.00 points, and improves full H$^2$7SD by 8.00 and 5.75 points — but slightly hurts Reverse-KL Only. Entropy traces show rephrasing sharply reduces entropy under Reverse-KL Only but not under Magnitude Only. The interpretation is that direct KL matching toward a rephrased teacher overconstrains an already-correct trajectory, whereas magnitude modulation preserves the valid reasoning path while emphasizing its load-bearing tokens.

Limitations and open questions

The evaluation is confined to a single base model (Qwen3-30B-A3B) and to deterministic, verifiably correctable puzzle domains; whether outcome-conditioned routing transfers to open-ended tasks with noisy or partial rewards is untested. The method depends on an external strong model for hint generation, introducing a cost and quality dependency that the paper does not quantify against cheaper privileged sources at scale. The hyperparameters $^2$8 and $^2$9 are not ablated, so sensitivity to the mixing weights remains unknown. Finally, the top-100 truncation used for distillation efficiency is presented as an engineering necessity; its effect on distillation fidelity is not measured. Whether sibling solutions could replace external hint generators when no stronger model is available — they performed well but not optimally here — is left open.

Conclusion

H$^2$0SD reframes self-distilled RLVR around a correctness-conditioned division of labor: reward-determined policy-gradient updates refined by hindsight magnitude modulation for successful trajectories, and hint-conditioned reverse-KL distillation for failed ones. The controlled comparisons — identical hints across methods, reversed-routing controls, and entropy dynamics — support the claim that the routing structure, informative process-level hints, and successful-response rephrasing jointly produce the observed gains, which are large (overall 50.49% vs. ≤24.85% for baselines) and generalize to unseen task sizes.