Recursive Self-Improvement via On-Policy Distillation for Reasoning

Abstract: On-policy distillation (OPD) trains a student model by having it generate trajectories, then matching its next-token predictions with an external teacher's next-token predictions. This provides dense, token-level supervision to the student. On-policy self-distillation (OPSD) eliminates the need for the external teacher. Specifically, a second frozen copy of the student model, now given the ground truth in its context, serves as the teacher. The student model only receives the problem and learns to mimic the privileged teacher model, while the teacher remains frozen throughout training. Previous work showed that freezing the teacher is useful for training stability, but we argue that this can prevent the teacher from incorporating the improvements learned by the student during training. Our primary contribution is to address this limitation with a recursive framework built around two complementary components. First, we let the privileged teacher co-evolve with the student so that revision learned in one round can guide the next, a process we refer to as Dynamic Co-Evolution (DCE). Second, because stronger revision can also make responses too verbose and self-critical, we additionally train on shorter, verified rewrites of the model's own on-policy responses. We call this complementary objective Self-Refined Concise Learning (SRCL). Overall, our comprehensive evaluations show that DCE+SRCL outperforms OPSD across multiple model scales and four competition-level mathematics benchmarks. Specifically, on Qwen3-8B, DCE+SRCL reaches 65.97% Average@12, outperforming OPSD by 35.62 percentage points while reducing mean output length by 7.80% relative to DCE alone.

• The paper introduces Dynamic Co-Evolution (DCE) and Self-Refined Concise Learning (SRCL) for improving reasoning tasks in large language models. DCE involves a recursive mechanism where the checkpoint is periodically updated, acting as both the next student and privileged teacher, which results in a more adaptive supervision tailored to evolving student behaviors.
• DCE+SRCL shows significant improvements compared to OPSD, achieving a 35.62% increase in Average@12 accuracy and a 7.80% reduction in output length on the Qwen3-8B model. This improvement is attributed to DCE's ability to incorporate revision abilities into future supervision and SRCL's effectiveness in filtering out redundant reasoning.
• The study demonstrates that the static maintenance of the teacher instead of refreshing it results in a teacher-student mismatch to a relatively high degree, impacting the overall accuracy.

Problem setting and central claim

“Recursive Self-Improvement via On-Policy Distillation for Reasoning” (2609.30652) addresses a specific weakness of on-policy self-distillation (OPSD): the privileged teacher is conditioned on a verified solution but remains frozen at the initial checkpoint. The student, by contrast, evolves during training and may acquire behaviors—reflection, backtracking, self-verification, and error correction—that are absent or weakly represented in the frozen teacher. The resulting teacher–student mismatch is particularly consequential on incorrect student trajectories, where useful supervision should indicate how to continue reasoning rather than simply terminate.

The paper’s central claim is that privileged self-distillation should be recursive. A checkpoint updated using privileged guidance should become both the next student and the next privileged teacher. This mechanism, termed Dynamic Co-Evolution (DCE), is combined with Self-Refined Concise Learning (SRCL), which trains on shorter, verified rewrites of the model’s own on-policy responses. DCE is intended to improve the quality of revision, whereas SRCL constrains the inference cost associated with increasingly frequent revision.

The empirical claim is substantial: across four competition-level mathematics benchmarks, DCE markedly outperforms OPSD across Qwen3 model scales, and DCE+SRCL generally improves or preserves accuracy while reducing output length relative to DCE. On Qwen3-8B, DCE+SRCL reaches 65.97% Average@12 accuracy, compared with 30.35% for OPSD and 20.28% for GRPO. These results are obtained in non-thinking mode, using 12 sampled responses per problem and a 32K generation cap.

Frozen privileged teachers and the motivation for co-evolution

OPSD provides dense token-level supervision by comparing the student’s next-token distribution on an on-policy trajectory with the distribution of a privileged teacher that additionally receives a verified solution. Unlike standard OPD, it does not require a separately trained external model. The privileged context is intended to make the teacher’s predictions informative at prefixes where the student is uncertain or incorrect.

The paper’s diagnostic analysis challenges the assumption that a frozen privileged teacher remains useful throughout training. On fixed incorrect Qwen3-8B trajectories from AIME24, AIME25, and AIME26, the initial frozen teacher assigns an average probability of 94.5% to EOS at incorrect response endpoints, while assigning only 22.0% probability to observed reflection cues such as “Wait.” Thus, even with access to the verified solution, the teacher often favors terminating an incorrect response over initiating revision. Meanwhile, the evolving student becomes increasingly likely to reflect, creating an expanding mismatch between the student’s behavior and the teacher’s target distribution.

This observation gives the paper’s recursive mechanism a concrete interpretation. If the student acquires revision behavior through training, a frozen teacher cannot incorporate that behavior into subsequent supervision. DCE instead refreshes the privileged branch after every update: the resulting checkpoint initializes both the next student and a detached, gold-conditioned teacher. The teacher is therefore not trained by gradients during a given update, but its parameters change between rounds as a consequence of the shared checkpoint update. This preserves the stabilizing role of a detached teacher while allowing the supervision policy to evolve.

The distinction between gradient detachment and parameter freezing is important. In DCE, the privileged predictions are stop-gradient targets within each update, but the target-generating checkpoint is periodically—or, in the main implementation, continually—replaced by the updated model. The method consequently implements recursive teacher improvement without allowing the teacher to chase the student within the same gradient computation.

Dynamic Co-Evolution

At training round $k$, the current checkpoint $\theta_k$ generates an on-policy response $y^{(k)}$ from the problem alone. For each retained prefix, the student predicts the next token using the problem and the prefix, while the privileged teacher uses the same prefix together with the verified solution and a transition instruction. Both distributions are produced by $\theta_k$, but gradients are stopped through the privileged distribution.

The main guidance objective uses Forward KL,

$D_{\mathrm{KL}}(q \,\|\, p),$

where $q$ is the gold-conditioned teacher distribution and $p$ is the problem-only student distribution. The asymmetry is operationally important. Forward KL weights discrepancies according to teacher-supported tokens, allowing the loss to emphasize alternatives that the student assigns low probability but that the privileged branch supports. Reverse KL instead emphasizes modes already represented in the student, while JSD weakens the directional transfer when teacher and student distributions differ substantially.

After optimization, $\theta_{k+1}$ replaces $\theta_k$ in both roles. This procedure allows revision behavior learned at one round to become part of the privileged supervision available at the next round. The paper’s interpretation is supported by fixed-trace probes. Across incorrect trajectories from three AIME cohorts, the macro-averaged endpoint EOS probability of the privileged teacher decreases from 90.4% at initialization to 41.3% at step 200. Its probability on observed reflection tokens rises from 32.8% to 77.4%. The corresponding student values change from 92.6% to 26.3% for EOS and from 30.1% to 77.0% for reflection tokens.

These measurements do not establish that the teacher has learned generally correct revision policies: they are behavior probes on fixed trajectories and use observed reflection tokens rather than a complete vocabulary of revision actions. They do, however, directly support the narrower claim that co-evolution changes the teacher’s local termination and reflection preferences in the direction required for continued revision.

Self-Refined Concise Learning

DCE improves revision but does not itself penalize redundant verification, repeated branches, or continued reasoning after a correct solution has been reached. The paper therefore introduces SRCL as a complementary self-training objective.

For each on-policy response, the current checkpoint is prompted to produce a direct, self-contained rewrite without access to the verified solution. A rewrite is retained only if it is shorter than its source response, terminates naturally, satisfies structural constraints, contains no prohibited reflection language, and passes answer verification. Accepted rewrites are used as autoregressive supervised targets. The ground-truth solution is used only by the acceptance procedure; it is not exposed during rewriting.

This design separates two forms of supervision. DCE supplies dense, gold-conditioned guidance at prefixes of the original on-policy trajectory, including incorrect prefixes. SRCL supplies a filtered target representing a shorter successful solution. Consequently, SRCL is not intended to teach revision from scratch. The ablation confirms this dependency: SRCL without DCE collapses to 2.36% Average@12 accuracy on Qwen3-8B with 1,116 mean tokens, and to 0.76% on Qwen3-4B with 531 mean tokens. SRCL therefore functions as a concision objective only when DCE has developed the underlying reasoning and revision capability.

The filtering process is relatively permissive in volume but strict in content. In the Qwen3-8B run, 2,296 of 3,200 candidate rewrites were accepted, giving a 71.75% acceptance rate. Accepted targets were 83.74% shorter than their source rollouts. Most rejected candidates failed structural checks; 76 retained explicit revision language and two failed to terminate naturally. These statistics indicate that SRCL receives a substantial amount of training data, but they also mean that the method depends on a verifier and on hand-specified structural criteria, including boxed-answer parsing and rejection of explicit markers such as “Wait,” “Actually,” and “start over.”

Main empirical results

The main experiments use 14,717 OpenThoughts mathematical-reasoning problems for training and evaluate on AIME24, AIME25, AIME26, and HMMT February 2025. Accuracy is reported as Average@12, with 12 sampled responses per problem. The main ranking uses non-thinking decoding and a 32K generation cap.

Model	Method	Average@12	Mean tokens
Qwen3-8B	OPSD	30.35%	6,009
Qwen3-8B	DCE	65.76%	19,046
Qwen3-8B	DCE+SRCL	65.97%	17,561
Qwen3-4B	OPSD	22.85%	8,350
Qwen3-4B	DCE	60.00%	19,360
Qwen3-4B	DCE+SRCL	61.88%	17,265
Qwen3-1.7B	OPSD	10.35%	5,380
Qwen3-1.7B	DCE	23.33%	12,600
Qwen3-1.7B	DCE+SRCL	26.88%	19,504

At 8B, DCE+SRCL improves over OPSD by 35.62 percentage points. At 4B and 1.7B, the improvements are 39.03 and 16.53 points, respectively. The 4B result is especially notable because SRCL improves both accuracy and length relative to DCE: accuracy rises from 60.00% to 61.88%, while mean output decreases by 10.82%, from 19,360 to 17,265 tokens. At 8B, SRCL preserves essentially the same accuracy as DCE while reducing mean output by 7.80%.

The scale dependence is material. At 1.7B, SRCL increases both accuracy and length, and the training trajectory becomes unstable after early gains. DCE reaches its maximum at step 30, while DCE+SRCL peaks at step 50 and later declines. The reported 1.7B result therefore depends on checkpoint selection and does not support a general claim that SRCL uniformly improves the accuracy–length frontier.

The method also transfers across model families. On Gemma-4-12B-IT, DCE reaches 62.01% Average@12 and DCE+SRCL reaches 63.61%, compared with 51.04% for OPSD. SRCL contributes 1.60 points while reducing mean output from 8,812 to 8,526 tokens. The cross-family experiment supports the portability of the training principle, although it remains a single additional model family and uses a distinct optimization configuration.

The paper further reports results on MATH-500, GPQA-Diamond, and AMC23. DCE+SRCL improves over OPSD on all three benchmarks for both Qwen3-8B and Qwen3-4B. On Qwen3-8B, the gains are modest on MATH-500, from 88.58% to 88.95%, but larger on GPQA-Diamond, from 50.29% to 58.29%, and AMC23, from 78.54% to 95.42%. These improvements generally involve longer outputs, indicating that the concision effect observed on the competition-math suite does not transfer uniformly to broader tasks.

Separating learned revision from longer generation

A central potential confound is that DCE produces much longer responses than OPSD. The paper addresses this with test-time-scaling controls that force OPSD responses to continue to exact 8K or 16K budgets using a “Wait” cue. The controls do not update model parameters.

For Qwen3-8B, standard OPSD obtains 30.35% Average@12 with 6,009 tokens. Forced 16K OPSD reaches only 30.76%, despite generating 16,384 tokens per response. By contrast, DCE+SRCL obtains 65.97% with 17,561 mean tokens. Under an 8K target, OPSD-TTS obtains 28.47%, whereas DCE+SRCL with a continuation cue obtains 35.07%. At 16K, the corresponding figures are 30.76% and 57.64%.

The implication is specific: additional tokens alone do not reproduce the performance gains. The learned policy must use the additional budget differently, particularly by allocating probability to revision rather than merely continuing an unproductive trajectory. This conclusion is strengthened by the matched-budget comparisons, but it remains conditioned on the particular continuation intervention and on the evaluated benchmarks.

Teacher refresh, divergence choice, and privileged-context placement

The teacher-refresh ablations provide the strongest evidence that recursive co-evolution, rather than on-policy distillation alone, is responsible for the gains. On Qwen3-8B, frozen-target DCE reaches 47.99% Average@12, compared with 65.76% for fully dynamic DCE. With SRCL, frozen-target performance is 41.04%, compared with 65.97% for dynamic DCE+SRCL. On Qwen3-4B, the corresponding comparison is 25.28% versus 60.00% for DCE and 27.29% versus 61.88% for DCE+SRCL.

EMA and periodic refresh schedules also outperform a frozen teacher. However, the fully dynamic teacher performs best in the reported main comparison: 65.97% with 17,561 tokens, versus 65.35% for the best EMA configuration and 63.47% for the best periodic snapshot. The authors appropriately caution that each schedule is represented by a single trajectory, so small differences should not be treated as definitive. The large gap between frozen and any evolving teacher is much more robust than the smaller gap among evolving schedules.

The choice of divergence is also consequential. Forward KL reaches 65.97% with 17,561 tokens. Reverse KL reaches 60.42% with 23,748 tokens, a 5.55-point reduction while using 6,187 more tokens. JSD reaches only 17.99% with 3,767 tokens, indicating premature shortening rather than useful concision. The result supports the paper’s interpretation that Forward KL is better suited to transferring low-probability teacher-supported alternatives, including revision actions.

The serialization of the verified solution has an unexpectedly large effect. The strongest configuration places the solution in prior assistant context, followed by a transition instruction and the on-policy prefix. Relative to user-side instruction-last conditioning, this improves Average@12 by 9.79, 6.39, and 14.66 points for Qwen3-8B, Qwen3-4B, and Qwen3-1.7B, respectively. On Qwen3-8B, assistant-side conditioning achieves 65.97%, compared with 56.18% for user-side instruction-last and 61.74% for dynamic reference-last conditioning.

The analysis suggests that assistant-side placement creates a strong early “route initialization” effect. However, the paper does not establish a purely causal account of this phenomenon. Position-resolved probes show larger early distribution shifts, but KL magnitude is not equivalent to correctness, and the three placement curves come from separately trained models. The result is therefore an important engineering finding rather than a complete explanation of why the serialization works.

Accuracy–length tradeoffs and optimization behavior

The paper’s results reject a simple interpretation of concision as direct termination pressure. An auxiliary EOS-specific loss produces shorter responses but substantially worse accuracy than SRCL. On Qwen3-8B, EOS-Penalty reaches 61.11% with 16,067 tokens, compared with 65.97% and 17,561 tokens for DCE+SRCL. On Qwen3-4B, it reaches 32.71% with 7,169 tokens, far below DCE+SRCL’s 61.88%.

This contrast indicates that useful concision requires preserving successful reasoning while eliminating redundant detours. Directly increasing EOS probability does not distinguish a completed correct derivation from an incorrect trajectory that should continue. SRCL makes this distinction indirectly through shorter rewrites that must remain answer-correct.

The loss-weight sweeps show broad but non-monotonic operating regions. For Qwen3-8B, DCE guidance weights from $10^4$ to $\theta_k$0 produce Average@12 values between 63.96% and 65.97%, while SRCL weights from 12.5 to 500 remain within 2.57 points of the main result. For Qwen3-4B, the best tested SRCL weight is 35, producing 61.88% with 17,265 tokens. These findings indicate local robustness but not a scale-independent hyperparameter rule. The framework remains sensitive to the balance between privileged guidance and concise-target learning, especially at smaller scales.

The Qwen3-14B experiment extends the method to a larger model. DCE+SRCL raises Average@12 from 21.39% at initialization to 42.64% at step 20 and 66.39% at step 30, a 45.00-point improvement. AIME26 increases by 50.28 points, from 19.44% to 69.72%. At step 50, accuracy remains comparable at 65.21%, but mean output increases from 15,877 to 20,050 tokens. This result reinforces the importance of early stopping: recursive improvement can rapidly improve accuracy while subsequently degrading the accuracy–length frontier.

Limitations and open questions

The evaluation is concentrated on automatically verifiable mathematical reasoning. The training data, acceptance filters, and benchmark metrics all depend on exact-answer verification, boxed-answer conventions, and competition-style problem structure. Although results on GPQA-Diamond and other benchmarks provide some evidence of transfer, the paper does not establish whether DCE remains effective when correctness is subjective, partially verifiable, or difficult to decompose into a final scalar answer.

The use of the verified solution in the privileged teacher also creates an unusually strong training signal. The student does not see the solution, but the teacher does, and the teacher’s predictions are distilled along student-generated prefixes. The method therefore demonstrates improvement under privileged information rather than eliminating supervision requirements. SRCL additionally depends on a verifier and manually specified structural filters. How performance changes with noisy, incomplete, or adversarial verification is not evaluated.

The recursive update can also amplify undesirable behaviors. The paper reports instability at Qwen3-1.7B and increasing output length at several scales. Since the evolving checkpoint becomes the next teacher, errors or pathological revision strategies could potentially be propagated into later supervision. The experiments compare several refresh schedules, but they do not characterize failure modes under systematically corrupted teacher targets or imperfect answer verification.

Finally, the fixed-trace probes establish changes in EOS and reflection-token probabilities, not successful error correction in every individual trajectory. A higher probability of “Wait” or a lower probability of EOS can represent useful revision, unproductive looping, or length-induced overthinking. The benchmark improvements show that the aggregate policy becomes more effective, but the causal relationship between specific token-level interventions and final correctness remains unresolved.

Conclusion

The paper presents DCE as a recursive extension of OPSD in which the privileged teacher co-evolves with the student, and SRCL as a complementary mechanism for learning shorter verified solutions. Its main empirical contribution is the demonstration that teacher refresh is not a minor implementation detail: on Qwen3-8B, dynamic co-evolution raises Average@12 accuracy from 30.35% for OPSD to 65.76% for DCE, while DCE+SRCL reaches 65.97% and reduces output length relative to DCE.

The ablations support three specific conclusions. First, frozen privileged teachers become misaligned with evolving revision behavior. Second, Forward-KL distillation from an evolving privileged branch transfers useful low-probability alternatives more effectively than the tested alternatives. Third, concise self-refinement improves the accuracy–length frontier only when dynamic guidance has already established competent revision. The remaining question is how reliably this recursive mechanism transfers beyond verifier-based mathematical reasoning and how its stability can be controlled when the privileged signal or acceptance verifier is imperfect.