On-Policy Self Distillation (OPSD)

• OPSD is a self-distillation technique where one model serves as both student and teacher, using privileged context for enriched token-level guidance.
• It operates on-policy by training on its own rollouts, aligning the unprivileged student outputs to the privileged teacher’s distribution with dense supervision.
• Recent OPSD variants demonstrate notable reasoning gains and effective compression, reducing GPU memory use by 40%-60% compared to traditional distillation.

Searching arXiv for recent OPSD papers and related variants to ground the article in current literature. On-Policy Self-Distillation (OPSD) is a post-training paradigm in which a single model serves as both student and teacher, with the teacher receiving privileged context unavailable at deployment and the student trained on its own rollouts to match the teacher’s token-level or step-level distribution. In the recent literature, OPSD has been instantiated with verified reasoning traces, world feedback, system prompts, persona information, visual thoughts, multimodal targets, and self-generated future answers, and has been used both as a reasoning-improvement method and as a compression mechanism for verbose chain-of-thought policies (Zhao et al., 26 Jan 2026, Cui et al., 18 May 2026).

1. Formal framework and defining properties

In its canonical language-model form, OPSD defines a student policy that conditions only on the task input $x$, and a self-teacher that shares the same base parameters but conditions on $x$ plus privileged context $c$ or $y^*$. Training samples trajectories from the student itself,

$\hat y \sim \pi_S(\cdot \mid x),$

and minimizes a per-token divergence between teacher and student along the student’s own prefixes. One generic formulation is

$$\mathcal{L}_{\text{OPSD}}(\theta) = \mathbb{E}_{(x,y^*)}\,\mathbb{E}_{\hat y\sim \pi_S(\cdot\mid x)} \left[ \frac{1}{|\hat y|}\sum_{n=1}^{|\hat y|} D\!\big(\pi_T(\cdot\mid x,y^*,\hat y_{<n})\,\|\,\pi_S(\cdot\mid x,\hat y_{<n})\big) \right],$$

with gradients applied only to the student branch (Zhao et al., 26 Jan 2026). In other OPSD instantiations, the divergence is written in reverse-KL form from the student distribution to a stop-gradient teacher distribution on on-policy data, or as generalized JSD (He et al., 28 May 2026).

Two properties distinguish OPSD from adjacent paradigms. First, it is on-policy: the training state distribution is induced by the current student, not by expert traces or teacher-generated trajectories. Second, it is self-distillation: the teacher is not a separate stronger model, but the same model under richer conditioning, often with a frozen or EMA teacher copy for stability (He et al., 28 May 2026, Cui et al., 18 May 2026). This architecture removes the explicit capacity gap of classical teacher–student distillation and, in the overview literature, is described as reducing GPU memory consumption by approximately $40\%-60\%$ compared to standard OPD (Cui et al., 18 May 2026).

The operational contrast with RLVR or GRPO is equally sharp. RLVR consumes outcome-level rewards, often binary, and broadcasts trajectory-level advantage information across tokens. OPSD instead uses dense token-level supervision induced by the teacher’s privileged context, with no explicit reward model or advantage computation in the basic formulation (He et al., 28 May 2026, Zhao et al., 26 Jan 2026).

2. Privileged context as the central design variable

The defining degree of freedom in OPSD is the privileged context supplied only to the teacher. In the literature, this context ranges from verified reasoning traces to environment feedback and multimodal auxiliary channels.

In coding-focused OPSD, “world feedback” is treated as first-class text rather than collapsed into a scalar reward. The teacher may receive compilation/runtime errors, unit test results, stack traces, explanatory messages, peer or own solutions from previous rollouts, and hybrid templates such as “Peer Solution + Feedback,” while the student remains unprivileged. The teacher’s advantage is explicitly attributed to in-context learning over $c$, not to a stronger base network (He et al., 28 May 2026). This design reframes OPSD as a mechanism for internalizing the effects of rich environment interaction into the base policy.

In reasoning settings, privileged context often consists of a verified solution, a final answer, or a reference chain-of-thought. Early OPSD formulations used problem-solution pairs so that the same model, once shown the solution, could rationalize it and provide a more informative distribution over the student’s own prefixes (Zhao et al., 26 Jan 2026). Other work uses system prompts or alignment directives as privileged information, turning OPSD into a form of prompt internalization rather than solution imitation (Zhu et al., 11 May 2026).

Recent multimodal and generative extensions broaden the notion of privileged context further. Visual-OPSD conditions the teacher on privileged visual-thought traces while the student is text-only at inference (Li et al., 17 Jun 2026). D-OPSD uses multimodal features of text plus target image for the teacher and text-only conditioning for the student in few-step diffusion models (Jiang et al., 6 May 2026). In d-OPSD for diffusion LLMs, the teacher is constructed from partially revealed tokens of the model’s own final answer, so that the student learns from “self future-experience” via suffix conditioning rather than privileged prefixes (Luo et al., 16 Jun 2026).

A recurring implication across these papers is that OPSD is less about a fixed loss than about a training-time asymmetry in information access. Different forms of world knowledge, feedback, or latent structure become OPSD-compatible as long as they can be encoded into the teacher context and yield a more useful teacher distribution on the student’s visited states.

3. Objectives, update rules, and major variants

Although the one-model-two-context pattern is stable across the literature, OPSD objectives have diversified substantially. Reverse KL and JSD dominate the basic formulations. Some papers use full-vocabulary token-level divergence; others adopt sampled-token estimators or PPO-like policy-gradient views in which the teacher–student log-probability gap acts as a dense per-token advantage (Zhao et al., 26 Jan 2026, Zhu et al., 11 May 2026). In diffusion settings, token-level KL is replaced by MSE on velocity fields or by step-level KL aligned to denoising transitions (Jiang et al., 6 May 2026, Luo et al., 16 Jun 2026).

Several influential variants modify where and how distillation is applied rather than changing the core self-teacher idea. ROSD replaces full-solution imitation with reflection-guided, error-localized distillation: a self-reflector extracts a corrective idea $e$ and an error quote $q$, and masking restricts distillation to the erroneous suffix, preserving valid prefixes (Zhao et al., 27 May 2026). OGLS-SD uses verifiable outcome rewards to construct positive and negative guidance pools, then steers teacher logits by contrasting successful and failed privileged contexts, with the explicit goal of counteracting reflection-induced bias and response-template artifacts (Yang et al., 12 May 2026). DemoPSD moves from teacher matching to a disagreement-modulated reverse-KL barycenter target, interpolating geometrically between teacher and student to attenuate privileged-information leakage and preserve exploration (Li et al., 2 Jul 2026). RLCSD introduces a correct-versus-wrong hint contrast, arguing that much of the raw OPSD signal is privilege-induced style drift rather than task-bearing supervision (Pan et al., 10 Jun 2026). TS-OPSD discards external privileged data altogether and uses a high-temperature version of the model’s own logits as self-teacher, treating OPSD as a policy-reheating stage after RL entropy collapse (Yang et al., 30 May 2026).

Framework	Teacher modification	Reported emphasis
"Self-Distilled Reasoner" (Zhao et al., 26 Jan 2026)	Verified reasoning trace in teacher context	Token-efficient reasoning post-training
"A Predictive Law for On-Policy Self-Distillation From World Feedback" (He et al., 28 May 2026)	EMA self-teacher with world-feedback context	Gap-to-improvement predictability
"ROSD" (Zhao et al., 27 May 2026)	Reflection-conditioned teacher plus error-span mask	Error-localized correction and OOD robustness
"DemoPSD" (Li et al., 2 Jul 2026) / "RLCSD" (Pan et al., 10 Jun 2026)	Barycentric or contrastive target shaping	Leakage and style-drift mitigation
"EDGE-OPD" (Lazaridis et al., 22 May 2026)	Guided rollouts plus evidence mask	Rare-token identity internalization
"Visual-OPSD" (Li et al., 17 Jun 2026) / "D-OPSD" (Jiang et al., 6 May 2026) / "d-OPSD" (Luo et al., 16 Jun 2026)	Privileged visual, multimodal, or self-future context	Multimodal and diffusion adaptation

These variants collectively show that OPSD has evolved from a single reverse-KL recipe into a family of algorithms whose differences lie in teacher construction, token selection, divergence choice, and the degree to which outcome information or verifier signals shape the dense supervision.

4. Empirical regimes: reasoning gains, predictive laws, and compression

Empirically, OPSD has been reported in several distinct operating regimes. An early reasoning-focused formulation showed that OPSD can outperform off-policy distillation and achieve $x$0 token efficiency compared to GRPO-like RL on mathematical reasoning benchmarks, while slightly exceeding GRPO at 4B and 8B scale on average@16 across four math benchmarks (Zhao et al., 26 Jan 2026).

A separate line of work identifies a striking predictive law for OPSD from world feedback. On LiveCodeBench, the initial self-teacher minus student validation gap,

$x$1

almost linearly predicts the final OPSD improvement after 50 steps. For Qwen3-8B, the fitted line is intercept $x$2, slope $x$3, $x$4; for Olmo-3-7B-Instruct, $x$5, $x$6, $x$7. The relation also holds across Qwen3 model sizes $x$8B, $x$9B, $c$0B, and $c$1B with slope $c$2 and $c$3, suggesting that teacher–student in-context gap is a useful pre-training diagnostic for OPSD configuration quality (He et al., 28 May 2026).

A third regime treats OPSD primarily as reasoning compression. OPSDC conditions the same model on a “be concise” instruction to obtain teacher logits and then distills them back into the default policy on on-policy rollouts. On Qwen3-8B and Qwen3-14B, it reports $c$4 token reduction on MATH-500 while improving accuracy by $c$5 points absolute; on AIME 2024, the 14B model gains $c$6 points with $c$7 compression (Sang et al., 5 Mar 2026). The paper further argues that OPSDC compresses easy problems more aggressively while preserving longer deliberation on harder ones, without explicit difficulty estimation.

That compression interpretation is reinforced by work on thinking-enabled mathematical reasoning after RL. When OPSD is separated into correct-only and incorrect-only rollouts, training only on correct rollouts preserves accuracy while substantially shortening responses, whereas training only on incorrect rollouts damages accuracy. This leads to the proposal that, for long mathematical chain-of-thought, OPSD is most reliable as a post-RL compaction stage rather than as the main correctness-improving mechanism, motivating the pipeline $c$8 (Kim et al., 7 May 2026).

Taken together, these studies suggest that OPSD is not a single empirical phenomenon. It can behave as dense post-training supervision, as a predictable world-feedback internalizer, or as a compactor of already competent reasoning policies, depending on the nature of the privileged context and the error profile of the target domain.

5. Relation to RLVR, OPD, and multimodal or diffusion extensions

OPSD occupies an intermediate position between reward-based RL and teacher-based distillation. Relative to RLVR, it replaces scalar outcome supervision with dense distributional guidance. Relative to standard OPD, it removes the external teacher and instead derives the teacher from the same model under privileged information. The overview literature explicitly frames this as a way to preserve on-policy alignment while avoiding the memory and realizability costs of cross-model distillation (Cui et al., 18 May 2026).

This relation is not purely competitive. Several papers treat OPSD as a complement to RL rather than a replacement. RLCSD embeds contrastive self-distillation into a GRPO-style PPO objective, using verifier-anchored rollout advantages for direction and contrastive OPSD signals for token-level modulation (Pan et al., 10 Jun 2026). TS-OPSD uses OPSD after entropy collapse in RLVR, with a temperature-scaled self-teacher that restores exploration and yields a stronger initialization for continued RL than both standard continued RL and rollout-level temperature reheating (Yang et al., 30 May 2026). The post-RL compaction study for thinking-enabled math reaches a similar systems conclusion from the opposite direction: RLVR expands the frontier of solvable trajectories, whereas OPSD is better suited to compress and regularize them afterward (Kim et al., 7 May 2026).

The multimodal and non-autoregressive generalizations are especially significant because they show that OPSD is not tied to left-to-right language modeling. Visual-OPSD applies cross-modal OPSD inside a unified multimodal model where the teacher sees privileged visual-thought traces and the student is text-only at inference. Across nine benchmarks, it improves over its generative teacher by $c$9 percentage points with a $y^*$0 speedup, and the authors report $y^*$1 closure of the KL gap between VT-conditioned and text-only distributions (Li et al., 17 Jun 2026). D-OPSD adapts OPSD to step-distilled diffusion models by aligning student and teacher velocity predictions on the student’s own few-step trajectories; the teacher is conditioned on multimodal text-plus-target-image features, and the method is presented as preserving few-step inference quality during continual tuning (Jiang et al., 6 May 2026). d-OPSD extends OPSD to diffusion LLMs by replacing prefix-conditioned self-teachers with suffix-conditioned self-future teachers and moving from token-level to step-level supervision; it is reported to outperform RLVR and SFT baselines across four reasoning benchmarks while requiring only around $y^*$2 of RLVR’s optimization steps (Luo et al., 16 Jun 2026).

These extensions suggest that OPSD is best understood as a generic privileged-information distillation template rather than a language-only algorithm. The common invariant is on-policy supervision from a self-teacher with richer conditioning; the modality-specific implementation can vary from token distributions to velocity fields or denoising-step distributions.

6. Failure modes, controversies, and open questions

The contemporary OPSD literature is unusually explicit about failure modes. A comprehensive empirical study argues that OPSD fails in tested reasoning settings when privileged information is instance-specific and absent at test time, but works when privileged information encodes a shared latent rule such as a system prompt or alignment preference (Zhu et al., 11 May 2026). That paper identifies three mechanisms: distribution mismatch caused by teacher conditioning on student-generated prefixes, optimization instability from biased TopK reverse-KL gradients, and an OPSD-specific limitation in which the student learns a PI-free policy that aggregates incompatible PI-conditioned teachers (Zhu et al., 11 May 2026).

Several more recent methods can be read as direct responses to these criticisms. ROSD argues that standard OPSD overfits because conditioning on a verified solution encourages reference-trajectory imitation and full-response distillation overwrites valid prefixes; its reflection-guided, error-localized masking is meant to convert “imitate the reference solution” into “correct the specific error” (Zhao et al., 27 May 2026). OGLS-SD diagnoses reflection-induced bias and response-template mismatch in privileged self-teachers, then uses outcome-guided logit steering to subtract negative guidance and calibrate token-level supervision (Yang et al., 12 May 2026). DemoPSD addresses privileged-information leakage and exploration loss by moving to a disagreement-modulated reverse-KL barycenter target, proving leakage attenuation and exploration preservation under its formal setup and reporting stronger OOD robustness than SDPO on GPQA (Li et al., 2 Jul 2026). RLCSD introduces the notion of privilege-induced style drift, arguing that the teacher–student gap in standard OPSD concentrates on style tokens rather than task-bearing tokens; its correct-versus-wrong hint contrast is intended to cancel generic hint-induced style shifts (Pan et al., 10 Jun 2026).

A different but related failure mode appears in rare-token identity internalization. In a setting where the target self-name “EdgeRunner AI” has near-zero base probability, unguided OPSD and unguided RLSD-no-verifier completely fail to learn the target identity, because the student never visits the relevant states. EDGE-OPD resolves this with guided rollouts and an evidence mask that updates only on tokens for which privileged context provides positive evidence, and mask-region ablations show that the persona signal is localized to the positive-evidence tail (Lazaridis et al., 22 May 2026). This suggests that support mismatch, not only teacher bias, can be a first-order obstacle for OPSD.

Open questions remain broad. The predictive-law work is explicit that its evidence is confined to LiveCodeBench, 50-step training horizons, and model sizes up to 8B, with preliminary 14B behavior more hyperparameter-sensitive (He et al., 28 May 2026). The “Many Faces” study leaves unresolved how to handle instance-specific privileged information without collapsing to a weak geometric average of incompatible teacher policies (Zhu et al., 11 May 2026). The post-RL compaction work leaves open whether step-level verifiers or richer hindsight signals could make OPSD genuinely corrective on long failed traces (Kim et al., 7 May 2026). Multimodal and diffusion variants raise additional questions about how teacher construction, denoising schedules, and auxiliary modalities alter the stability–exploration trade-off (Li et al., 17 Jun 2026, Jiang et al., 6 May 2026, Luo et al., 16 Jun 2026).

The current consensus is therefore conditional rather than universal. OPSD is neither a drop-in replacement for RL nor a uniformly reliable self-improvement primitive. It is a flexible family of on-policy privileged-context distillation methods whose effectiveness depends on the relationship between privileged information, visited trajectories, and the deployment-time policy class. Where that relationship is well posed, OPSD can be efficient, predictable, and compact; where it is not, the method can overfit, collapse exploration, or internalize non-transferable artifacts.