On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models

Abstract: On-policy self-distillation enables post-training foundation models without relying on stronger external teachers. However, existing approaches rely on ground-truth question-answer pairs, human-curated problem sets, or other supervised signals. In this work, we reformulate on-policy self-distillation as a self-training framework, driven by self-generated question-answer pairs and privileged reasoning demonstrations. We push supervision to the limit: guided solely by coarse, high-level topic descriptions, the model synthesizes its own curriculum of queries and privileged reasoning traces without access to any human-curated prompts or answers. We find that self-distillation on fully synthetic privileged information enables a model to improve across multiple domains while retaining its capabilities on domains it is not self-trained on. We demonstrate the generality of this framework across text and multimodal domains, achieving consistent self-improvement on standard language benchmarks and Visual Question Answering. Our findings demonstrate that foundation models possess rich latent capabilities that can be amplified without external task supervision or curated problem distributions.

• The paper introduces On-Policy Self-Training (OPST), a data-free post-training method for large language models that generates synthetic problems and uses teacher-student distillation.
• OPST outperforms traditional methods like SFT and GRPO, achieving up to 10.2 points improvement in MMLU benchmark, and up to 6.9 points in VQAv2; it relies on synthetic context and parameters filtering, not literal target answers
• Experiments show OPST's effectiveness without curated supervision or answers across various benchmarks, including transfer to coarse Qwen3-4B model

Problem setting and central contribution

On-Policy Self-Training (OPST) addresses data-free post-training for language and vision-LLMs. Existing self-improvement methods typically eliminate one source of supervision while retaining another: SFT requires curated demonstrations, RLVR requires verifiable answers, on-policy distillation requires a stronger teacher, and on-policy self-distillation generally requires privileged reference solutions. OPST removes both the curated problem distribution and externally supplied answers. Its only textual input is a set of coarse topic descriptors, such as “Abstract Algebra” or “Commonsense Causal Reasoning.” From these descriptors, the model generates its own questions, rationales, and candidate answers, filters the resulting data, and trains through on-policy self-distillation (2610.32770).

The paper’s principal claim is that a model can improve by exploiting latent capabilities already encoded in its parameters, provided that synthetic information is used as privileged context for a teacher rather than treated as an unquestionable training target. The distinction is important. SFT on a generated rationale assumes that every generated token is correct; reinforcement learning based on agreement with a generated answer compresses the information into a scalar reward. OPST instead conditions an EMA teacher on the synthetic rationale and answer, while the student sees only the question. The teacher consequently supplies token-level distributions along trajectories sampled from the student’s current policy. This permits the teacher to complete omitted reasoning steps, reinterpret imperfect candidate answers, and express supervision in its own distribution rather than in the surface form of the synthetic trace.

The contribution is therefore not merely synthetic data generation. It is the combination of self-generated curriculum construction, intrinsic filtering, privileged-information distillation, and EMA stabilization into an iterative training loop.

On-policy self-distillation with synthetic privileged information

For a generated problem $x$ and privileged information $z$, consisting of a rationale and candidate answer, the student samples a response from $\pi_\theta(\cdot \mid x)$. At each visited prefix, the student distribution is compared with the teacher distribution $\pi_\phi(\cdot \mid x,z,y_{<t})$. The student is thus trained only on states that it visits, avoiding the teacher-forcing mismatch associated with conventional SFT, while the teacher receives information unavailable to the student.

The training objective uses a skewed Jensen–Shannon divergence rather than an unbounded KL divergence. With $\alpha=0.8$, the objective emphasizes the mode-seeking regime while remaining bounded by the entropy of the mixing coefficient. This limits the effect of localized teacher artifacts or incorrect synthetic hints: a token assigned negligible probability by one distribution cannot produce an arbitrarily large gradient. The teacher is updated through an EMA of student parameters, with the paper’s default rate $\beta=0.05$. The EMA serves two functions. It supplies a temporally stable target despite noisy synthetic batches, and it acts as the deployed model because its averaged parameters outperform the final student parameters.

The ablations establish that this stabilization is consequential rather than cosmetic. Without EMA tracking, the MMLU score increases only from 37.5 to 38.1, whereas the default configuration reaches 47.7. Excessive smoothing also reduces progress: $\beta=0.01$ yields 45.2, compared with 47.7 at $\beta=0.05$. The EMA teacher is itself 1.8 MMLU points better than the student at the end of training, indicating that parameter averaging removes update noise that remains in the student.

Self-generated curriculum and intrinsic filtering

OPST generates a question–privileged-information pair $(x,z)$ from a topic descriptor. The generator may be the evolving EMA teacher or a frozen copy of the initial model. Questions can be multiple-choice or open-ended, and sampling temperature is used to produce diversity. Since generation is no longer controlled by human-written prompts, the method requires filters that simultaneously assess data quality and training value.

Pseudo-verification tests whether the candidate answer is reproducible independently of the generation context and robust to paraphrase. The generator must recover the candidate answer when solving the question in a fresh context and again after paraphrasing the question. A sample is retained only when both answers agree with the candidate answer. This procedure rejects some malformed or ambiguous questions, but it does not establish objective correctness; it establishes only self-consistency.

The consensus gate addresses a different problem. For each pseudo-verified question, the student produces multiple on-policy rollouts. Samples for which every rollout agrees with the candidate answer are discarded, because the teacher has little additional information to provide on a problem the student already solves reliably. The retained curriculum therefore targets an approximate capability frontier. As the student improves, the gate changes which generated questions are retained, causing the effective difficulty of the curriculum to evolve.

The incremental ablation supports both filters. On Gemma 4 E2B, OPST without filtering improves MMLU by 5.2 points over the base model. Pseudo-verification adds 2.7 points, and the consensus gate adds a further 2.3 points, producing the full 10.2-point improvement. This result also shows that quality filtering is not an auxiliary engineering detail: it materially determines the learning signal.

Language-model results

The main language experiments use Gemma 4 E2B and E4B models, with topic descriptors corresponding to MMLU, GPQA, GSM8K, and HellaSwag. OPST is compared with SFT and GRPO on identical self-generated data, isolating the effect of using generated information as teacher context rather than as a direct target or reward.

The strongest result occurs for Gemma 4 E2B without thinking. OPST raises MMLU from 37.5 to 47.7, an improvement of 10.2 points; GPQA rises by 4.0 points, HellaSwag by 7.5 points, and the aggregate score by 7.3 points. For Gemma 4 E4B without thinking, OPST improves the aggregate by 6.1 points, with gains of 7.6 on MMLU, 6.1 on GPQA, and 4.5 on HellaSwag. In the thinking regime, Gemma 4 E2B improves by 8.2 points on GSM8K, 6.5 on GPQA, and 4.6 points on average. Gemma 4 E4B reaches 93.0 on GSM8K, an 8.6-point increase, and improves its aggregate by 4.7 points.

Model and regime	OPST aggregate gain	Largest reported task gains
Gemma 4 E2B, without thinking	+7.3	MMLU +10.2; HellaSwag +7.5
Gemma 4 E2B, with thinking	+4.6	GSM8K +8.2; GPQA +6.5
Gemma 4 E4B, without thinking	+6.1	MMLU +7.6; GPQA +6.1
Gemma 4 E4B, with thinking	+4.7	GSM8K +8.6; MMLU and GPQA +4.1

The comparison against alternative uses of the same synthetic data is particularly informative. OPST exceeds SFT on the aggregate score by 5.6 points for the 2B-scale model without thinking and by 1.3 points for the 4B-scale model without thinking. With thinking, the corresponding margins are 2.7 and 3.8 points. GRPO is consistently weaker than OPST and sometimes degrades individual benchmarks. These comparisons support the paper’s mechanistic interpretation: the benefit derives substantially from conditioning the teacher on synthetic privileged information, not simply from exposing the model to additional generated questions.

The privileged-information ablations further isolate this effect. Removing privileged information reduces the MMLU score from 47.7 to 40.6, only 3.1 points above the base model. A placebo context drawn from another problem on the same topic performs slightly worse, at 40.2, demonstrating that the useful signal is sample-specific rather than merely topical. Rationale-only context reaches 45.8, whereas answer-only context reaches 44.6; full rationale-plus-answer context performs best. Thus, the rationale contains more useful information than the candidate answer, but their combination is most effective.

OPST is also comparatively robust to incorrect synthetic answers. When 75% of candidate answers are replaced with wrong answers, OPST loses 1.9 MMLU points and remains 3.3 points above the base model. SFT loses 5.7 points and falls 4.4 points below the base model. This supports the claim that teacher conditioning treats generated answers as hints filtered through the teacher’s parametric prior rather than as literal ground truth. It does not, however, show that OPST can reliably correct systematic errors; it shows only that its degradation is less severe under the specified corruption experiment.

Comparison with curated supervision and domain transfer

Across supervision regimes, OPST performs best among methods without human-curated reference answers in all four reported model settings. Relative to u-OPSD, which uses curated questions and self-generated answers, OPST improves the aggregate by 1.0 to 2.8 points. Relative to R-Zero, which also operates without external answers but uses reinforcement learning on self-generated problems, OPST improves the aggregate by 1.4 to 5.2 points.

The comparison with fully supervised OPSD is substantial. On Gemma 4 E2B without thinking, OPST obtains a gain of 7.3 points over the base model, compared with 8.3 points for supervised OPSD, recovering 88% of the supervised improvement. In the thinking regime, OPST matches or slightly exceeds supervised OPSD at both model sizes, with aggregate differences of +0.2 and +0.7 points. Under these experimental conditions, curated questions and reference solutions are not necessary to approach the performance of supervised on-policy self-distillation. This claim should be interpreted within the narrow benchmark and model scale studied; the experiments do not establish equivalence under broader task distributions or larger models.

Multi-domain training produces limited interference. Combining all three principal topic groups yields a 7.6-point in-domain improvement, only 1.0 point below the best single-domain configuration. Out-of-domain retention is mixed rather than uniformly positive. The average OOD improvement is 0.8 points, but MATH-500 declines by 4.9 points. The MMLU-plus-GSM8K mixture gives the strongest OOD average gain, +3.9 points, and improves MMLU by 11.3 points. Even without MMLU topics, MMLU improves by 5.1 points, suggesting that some gains arise from general reasoning transfer rather than direct topic matching. At the same time, the MATH-500 regression demonstrates that OPST does not guarantee preservation of every pre-existing capability.

The appendix reports transfer to Qwen3-4B-Base, a pretrained model without instruction tuning. OPST increases the aggregate score from 66.7 to 72.7, with gains of 13.3 points on GSM8K, 5.1 on GPQA, 3.1 on MMLU, and 2.5 on HellaSwag. Because multiple-choice tasks are evaluated through log-likelihood over answer options, these improvements cannot be attributed to better answer formatting. This result broadens the evidence beyond the Gemma instruction-tuned models, although it remains a single additional model family and scale.

Extension to vision-LLMs

Visual OPST introduces unlabeled images and visual topic descriptors, such as object counting, attribute recognition, and spatial relations. The generator produces image-grounded questions and candidate answers. Privileged information is either a detailed scene caption, detected objects with bounding boxes, or both. Visual verification evaluates answer stability under image augmentations, excluding horizontal flips because they alter left–right relations.

The student receives only the image and question, while the teacher additionally receives the privileged caption and/or spatial relations. On Gemma 4 E2B, spatial privileged information raises VQAv2 accuracy from 49.1 to 55.7, while captions raise it to 54.0. Combining both reaches 56.0, a 6.9-point improvement. POPE F1 increases from 82.2 to 84.6, with recall rising by 4.5 points. The recall increase is consistent with improved suppression of object hallucination, although the paper does not provide a causal decomposition separating better visual recognition from altered response calibration.

The complementarity of the two privileged-information forms is not consistent across scales. At E4B, spatial information alone gives the highest VQAv2 accuracy, 59.6, whereas combining spatial information and captions reaches 59.2. The combined configuration also underperforms spatial information alone on POPE. Thus, multimodal privileged information is useful, but adding more context does not monotonically improve distillation.

Limitations and open questions

The paper explicitly characterizes OPST as an amplifier of existing knowledge. It cannot be expected to acquire information absent from the model’s parametric representations, particularly when no retrieval, tool execution, or external verifier is available. This limitation is central to the data-free formulation rather than an implementation defect.

Pseudo-verification measures reproducibility and paraphrase invariance, not correctness. A model can consistently generate and verify an incorrect answer, allowing systematic errors to enter the curriculum. The wrong-answer ablation demonstrates robustness to substantial random corruption, but it does not resolve correlated or adversarially persistent errors. The proposed consensus gate can also exclude questions that are difficult but already solved consistently, and it may retain questions whose answer distribution is unstable for reasons unrelated to useful learning signal.

The topic descriptors are hand-selected, so the system is not fully autonomous in curriculum specification. The study is also limited primarily to small Gemma models, one additional Qwen model, standard benchmarks, and a restricted set of visual tasks. Although OOD performance is often preserved or improved, the MATH-500 regression under the all-domain mixture shows that retention is not guaranteed. The paper leaves open whether scalable, label-free stopping criteria can detect when further rounds cease to improve the model, and whether tool- or retrieval-grounded verification can correct the systematic-error problem without sacrificing the data-free character of the method.

Conclusion

OPST formulates post-training as an iterative cycle in which a model generates its own problems and privileged reasoning information, filters them through self-consistency tests, and distills an EMA teacher into an on-policy student. Its central technical result is that synthetic rationales and answers are more effective as privileged teacher context than as direct SFT targets or rewards. Across language and multimodal experiments, the method produces substantial gains without curated questions or answers, including a 10.2-point MMLU improvement and a 6.9-point VQAv2 improvement. The evidence supports data-free self-improvement as a viable post-training regime, while also showing that correctness verification, curriculum specification, capability retention, and scale remain unresolved constraints.