Privileged Self-Distillation in Deep Learning

• Privileged self-distillation is a training paradigm where the teacher model uses additional training-only information to guide a student that operates on limited test-time inputs.
• It converts sparse sequence-level supervision into dense token- or turn-level learning signals, enhancing long-horizon reasoning and multimodal tasks.
• Corrective designs like selective attenuation and directional routing are applied to ensure that the privileged signals remain transferable without causing leakage or degraded performance.

Privileged self-distillation is a post-training paradigm in which a model is trained under an asymmetric information pattern: a student policy acts under the test-time input alone, while a teacher—often the same model, or a detached or EMA copy—conditions on additional information available only during training. In recent language-model and multimodal policy-optimization work, the privileged context has included verified answers, reference solutions, action-only traces, source documents, self-play-generated trajectories, rationales, hints, constitutions, and other auxiliary views. The central purpose is to convert sparse sequence-level supervision into dense token- or turn-level learning signals, especially in long-horizon reasoning, tool use, and multimodal settings where ordinary outcome-based RL suffers from weak credit assignment (Penaloza et al., 4 Feb 2026, Tian et al., 8 Jun 2026).

1. Conceptual lineage and formal setting

The broader idea belongs to the learning-using-privileged-information tradition. In learning-to-rank, privileged features distillation treats a feature set $z$ as privileged when it is available during training but unavailable at test time, formalized by $I(y;z\mid x)>0$. A teacher is trained on $(x,z)$, and a student is then distilled to operate on $x$ alone (Yang et al., 2022). In multimodal expression recognition, multi-teacher privileged knowledge distillation similarly exploits modalities present only during training and includes a self-distillation stage that aligns modality-specific teachers to a fused teacher representation before distillation to a student restricted to prevalent modalities (Aslam et al., 2024).

Modern privileged self-distillation in LLMs sharpens this asymmetry into an on-policy or self-evolutionary setting. A single model frequently plays both roles: the student produces rollouts under plain context $x$, while the teacher scores those same rollouts under $x$ plus privileged information such as a verified answer $y^*$, a reference trace $r$, or another training-only context. This formulation appears in $\pi$-Distill and OPSD for agentic environments, in which a teacher $\pi_\theta^T(o\mid s,I)$ sees privileged information $I(y;z\mid x)>0$0 while the student $I(y;z\mid x)>0$1 does not (Penaloza et al., 4 Feb 2026). Closely related asymmetric tutor–student formulations also appear in document-grounded question answering, where a tutor conditions on $I(y;z\mid x)>0$2 and the student on $I(y;z\mid x)>0$3 alone, despite the absence of verified answers or external graders (Stein et al., 24 Feb 2026).

The paradigm is therefore more specific than generic knowledge distillation. It is not merely teacher compression, because the teacher’s additional conditioning is intentionally non-deployable. It is also not merely supervised imitation, because many methods are integrated with RLVR, GRPO, PPO-style surrogates, or verifier-gated updates. This suggests that privileged self-distillation is best understood as a family of asymmetric training objectives for transferring the utility of training-time-only evidence into a policy that must ultimately act without that evidence.

2. Core objectives and token-level credit assignment

A common formalization samples a rollout $I(y;z\mid x)>0$4, defines a student distribution

$I(y;z\mid x)>0$5

and a privileged teacher distribution

$I(y;z\mid x)>0$6

then minimizes a dense token-level divergence such as

$I(y;z\mid x)>0$7

This is the canonical OPSD-style objective: the student learns on its own trajectory distribution, but each token is supervised by a teacher run with privileged conditioning (Li et al., 2 Jul 2026).

A second line of work reframes privileged self-distillation as credit assignment rather than direct teacher matching. PBSD defines a trajectory-level Bayesian evidence score

$I(y;z\mid x)>0$8

and, via Bayes’ rule, converts it into a likelihood ratio

$I(y;z\mid x)>0$9

Autoregressive decomposition yields the turn-level evidence score

$(x,z)$0

which is then used to modulate a trajectory-level advantage into turn-level advantages. Positive $(x,z)$1 marks a turn as supporting the verified outcome; negative $(x,z)$2 marks it as undermining the outcome (Tian et al., 8 Jun 2026).

Other frameworks preserve sparse verifier direction while letting privileged signals set only local magnitude. RLSD computes token-level privileged information gain

$(x,z)$3

and reweights a GRPO-style sequence advantage $(x,z)$4 by

$(x,z)$5

so that update direction remains anchored to the verifier while privileged evidence adjusts per-token credit magnitude (Yang et al., 3 Apr 2026). SDPG combines group-relative standardized verifier advantages, exact full-vocabulary reverse-KL self-distillation, and reference-policy KL regularization; it further gates distillation to positive-advantage rollouts through $(x,z)$6 (Liu et al., 2 Jun 2026).

A further variant dispenses with verified labels entirely and derives supervision from teacher consensus. GATES samples multiple privileged tutor trajectories, extracts their modal answer, defines a question-level gate $(x,z)$7, and distills full tutor trajectories to the student only when tutor agreement exceeds threshold and guardrails pass. In this setting, privileged self-distillation functions as a reliability-filtered asymmetric transfer mechanism rather than as reward shaping from an external verifier (Stein et al., 24 Feb 2026).

3. Sources and representations of privileged information

Privileged information in this literature ranges from raw oracle-like context to compressed, non-spoiling, or self-generated evidence. The choice of representation is central because it determines both the density of supervision and the risk of train–test mismatch.

Privileged signal	Representative frameworks	Teacher-side use
Verified answer, final answer, reference solution, solution steps	OPSD/$(x,z)$8-Distill, PBSD, Purified OPSD, DASD, SDPG	Token supervision or Bayes-calibrated evidence
Action-only traces, source document, question construction path	$(x,z)$9-Distill, GATES, $x$0-Play	Asymmetric tutor/teacher scoring on student or tutor trajectories
Rationales, hints, constitutions, skills, multiple views	OmniOPSD, PTD-PO, COPSD, SGSD, AVSD, AMR-SD	Privileged evidence, gated supervision, or reconstructed teacher targets

Some methods expose the teacher to raw privileged context. Examples include reference answers or reasoning traces in OPSD-style math training, source documents in asymmetric question answering, and full question construction paths in self-play, where the QCP is the examiner’s reverse solution process $x$1 and is concatenated into the teacher prompt while remaining hidden from the student (Zhang et al., 15 Apr 2026).

Other methods deliberately compress or sanitize privileged context before teacher conditioning. AMR-SD inserts a reflection bottleneck that converts verifier outcomes, peer rollouts, or reference feedback into concise self-generated Socratic hints or critiques rather than exposing the raw oracle trace (Wei et al., 18 May 2026). PTD-PO similarly uses structured privileged hints with a zero-spoiler rule, visual grounding, and high-level reasoning instructions so that the teacher is guided by “the way, not the answer” (Xiang et al., 5 Jun 2026). OmniOPSD conditions the teacher on frontier-generated evidence-aware rationales, but uses them only as training-time privileged evidence for rescoring student rollouts rather than as direct imitation targets (Cheng et al., 14 Jun 2026).

A third pattern treats privileged information as multi-view or weakly trusted evidence. AVSD constructs several privileged views—such as full solution, partial solution, final answer, demonstration, or execution feedback—and reconstructs a supervision signal by separating cross-view consensus from view-specific residuals (Nguyen et al., 20 May 2026). SGSD uses retrieved skill–mistake pairs as teacher-side context $x$2, treating each skill-conditioned teacher as a hypothesis whose stance must be validated by the verifier rather than blindly imitated (Huang et al., 27 May 2026). COPSD uses a safety constitution as privileged context, but calibrates the teacher through a Cross-SFT cold start before on-policy distillation (Wen et al., 2 Jun 2026).

This variety of representations suggests that privileged self-distillation is not tied to any single notion of “privileged information.” It is a broader architectural principle: the teacher is allowed to condition on information that would be useful for learning but unavailable, too costly, or undesirable to expose at inference.

4. Failure modes, leakage, and theoretical diagnoses

A major theme in the literature is that privileged self-distillation is not uniformly beneficial. Strong positive results coexist with repeated evidence that naïve privileged teacher imitation can damage long-horizon reasoning, generalization, or deployable behavior.

The clearest empirical challenge is the degradation of strong thinking models under dense privileged conditioning. “Rethinking On-Policy Self-Distillation for Thinking Models” reports that privileged-context distillation causes a relative drop of up to 17% in avg@16 accuracy across five Qwen3 and OLMo thinking models on AIME24, AIME25, and HMMT25. The degradation grows with the amount of privileged context withheld from the student, is most pronounced at long rollout budgets, and is associated with reduced fork rates, shorter responses, and fewer verification, backtracking, and hedging markers even after length normalization (Kaur et al., 6 Jul 2026).

Several papers provide theoretical or mechanistic accounts of this pathology. RLSD derives a KL decomposition

$x$3

arguing that the conditional mutual information term is irreducible and that late training becomes dominated by variance from privileged-information-dependent gradients, leading to leakage and collapse (Yang et al., 3 Apr 2026). Purified OPSD decomposes the teacher signal into a reference-induced component

$x$4

and an inference-transferable residual

$x$5

and reports that standard OPSD updates are dominated by the non-transferable reference component rather than by question-conditioned correction (Shen et al., 2 Jul 2026).

Related analyses locate the problem at particular token classes. DASD argues that uniform attraction to the privileged teacher suppresses high-entropy exploratory tokens and harms complex reasoning because high-entropy “forking” tokens and low-entropy “scaffolding” tokens should not receive the same direction of supervision (Zhang et al., 21 May 2026). AntiSD interprets the teacher–student log-probability gap

$x$6

as $x$7, and argues that default self-distillation reinforces structural or answer-template tokens while suppressing deliberation tokens such as “Wait” and “Maybe” (Shen et al., 12 May 2026). DemoPSD identifies privileged information leakage as a direct problem in dense token-level OPSD and proves a leakage attenuation property for its disagreement-modulated alternative (Li et al., 2 Jul 2026).

The same concern appears outside reasoning. In safety alignment, COPSD argues that constitution-conditioned reverse-KL OPSD contracts the teacher distribution toward short, overly conservative responses; in its geometric account, non-orthogonality between safety and expressiveness axes causes safety pressure to “leak” into the expressiveness dimension, producing reduced length and entropy (Wen et al., 2 Jun 2026). In ranking, privileged features distillation exhibits a non-monotone dependence on the predictive power of the privileged feature: as privileged signal strength increases, student performance first improves and then declines because a very predictive privileged teacher produces high-variance targets and inferior testing performance (Yang et al., 2022).

Taken together, these results contradict the simple intuition that a stronger privileged teacher should monotonically improve a weaker student. A plausible implication is that privileged information is useful only insofar as it can be converted into a signal that remains transferable to the student’s test-time information set.

5. Corrective designs and major algorithmic variants

A large fraction of recent work can be read as a sequence of attempts to retain dense teacher-side guidance while removing or attenuating non-transferable privileged pressure.

One strategy is selective attenuation rather than exact teacher imitation. DemoPSD replaces direct fitting of the privileged teacher with a reverse-KL barycenter target

$x$8

where $x$9 increases with the Jensen–Shannon disagreement between teacher and student. This weakens teacher influence on high-disagreement tokens and is proved to provide both leakage attenuation and exploration preservation (Li et al., 2 Jul 2026). PTD-PO also routes distillation selectively, applying hint-conditioned token-distribution alignment only to failed rollouts and using a bounded Top-$x$0 Jensen–Shannon divergence to stabilize teacher–student alignment under answer-free versus hint-augmented context shift (Xiang et al., 5 Jun 2026).

A second strategy is directional routing. DASD assigns a signed weight $x$1 to each token-level KL term, pushing high-entropy tokens away from the privileged teacher and pulling low-entropy tokens toward it. In its sampled-token PPO form, the effective advantage becomes

$x$2

with $x$3, so privileged supervision changes sign according to token uncertainty (Zhang et al., 21 May 2026). AntiSD takes a stronger stance by ascending a bounded Jensen–Shannon divergence instead of descending a KL, so that negative-PMI deliberation tokens receive positive reward; it adds an entropy-triggered gate to disable the term once teacher entropy collapses (Shen et al., 12 May 2026).

A third strategy is purification or decomposition of the privileged signal. Purified OPSD constructs a reference-only teacher $x$4, subtracts the reference-induced component, and then exponentiates the residual into a PMI-based target

$x$5

thereby filtering out reference-specific shortcut pressure while retaining question-conditioned correction (Shen et al., 2 Jul 2026). PBSD likewise avoids direct privileged-policy matching: it converts answer-side posterior-to-prior ratios into tractable teacher–student likelihood ratios and then into turn-level Bayesian evidence scores that reweight standard policy optimization rather than distilling the privileged policy itself (Tian et al., 8 Jun 2026). RLSD embodies the same design intuition in a different form by using privileged self-distillation only for update magnitude while letting the sparse verifier determine the direction of the policy update (Yang et al., 3 Apr 2026).

A fourth strategy is to gate or reconstruct supervision from multiple or weakly trusted sources. AVSD computes a geometric consensus target across privileged views, then selectively adds a view-specific residual only when alignment and magnitude gates are satisfied, guaranteeing that the reconstructed advantage preserves the sign of the consensus term (Nguyen et al., 20 May 2026). GATES uses question-level tutor consensus and trajectory-level eligibility to decide when full privileged tutor trajectories are reliable enough to distill at all (Stein et al., 24 Feb 2026). SGSD validates each skill-conditioned teacher’s polarity against the environment outcome, reversing or ignoring teacher influence when the skill’s stance is misleading or weak (Huang et al., 27 May 2026).

A fifth strategy is to restrict the bandwidth of teacher-side privileged access. AMR-SD replaces raw oracle traces with short Socratic hints or critiques and converts teacher–student log-probability differences into sparse ReLU-gated Causal Information Gain adjustments with temporal annealing (Wei et al., 18 May 2026). HDPO limits privileged self-distillation to “cliff” prompts on which all ordinary rollouts fail, generates privileged rollouts only there, filters for $x$6, and uses the distillation weight $x$7 to control the exploration–exploitation tradeoff (Ding, 25 Mar 2026).

These variants collectively indicate that privileged self-distillation has evolved from teacher imitation toward teacher-side evidence shaping. This suggests that the field increasingly treats privileged context as a noisy instrument for credit assignment, not as a target distribution to be matched uncritically.

6. Empirical landscape, applications, and open questions

The empirical record is broad rather than uniform. In long-horizon search agents, PBSD improves in-domain validation on BrowseComp synthetic from 40.87% vs. GRPO’s 38.25% (+2.6 pts) and out-of-domain BC300 from 35.83% vs. 32.33% (+3.5 pts), while also transferring from 64K-context training to 256K-context evaluation and inducing more—but shorter—search turns focused on evidence gathering (Tian et al., 8 Jun 2026). In self-play, $x$8-Play uses the question construction path as intrinsic privileged context and reports that data-free self-play surpasses fully supervised search agents while improving evolutionary efficiency by 2–3$x$9 over conventional self-play (Zhang et al., 15 Apr 2026). In frontier-agent distillation, $x$0-Distill and, in some cases, OPSD outperform SFT followed by RL even when the baseline assumes full chain-of-thought supervision, across TravelPlanner, $x$1-Bench Retail, OOD airline, and other agentic benchmarks (Penaloza et al., 4 Feb 2026).

In reasoning tasks, results are mixed in a way that mirrors the theoretical debates. DemoPSD improves over SDPO by +1.68 pp mean@16, +1.68 pp maj@16, and +2.82 pp best@16 on SciKnowEval, remains +7.9 pp above SDPO on average on GPQA Extended, and maintains 32–98 % higher policy entropy than SDPO (Li et al., 2 Jul 2026). DASD achieves the best macro Avg@16 across six mathematical reasoning benchmarks and, for Qwen3-8B, raises Avg@16 to 52.5 versus 46.5 for GRPO and 32.8 for OPSD; on AIME24, Pass@16 rises from roughly 80% under GRPO to roughly 87% under DASD while OPSD plateaus near 44% (Zhang et al., 21 May 2026). Purified OPSD improves every tested long-CoT model on both DASD-10K and Math-CoT-20K, with significance at $x$2, while standard OPSD often yields negligible gains or degradation (Shen et al., 2 Jul 2026). AntiSD reaches the GRPO baseline in 2 to 10x fewer training steps and improves final accuracy by up to 11.5 points across five math-reasoning models (Shen et al., 12 May 2026). By contrast, standard privileged OPSD can harm strong thinking models at long budgets, as already noted (Kaur et al., 6 Jul 2026).

In label-scarce or multimodal settings, privileged self-distillation has also been effective. GATES improves held-out in-domain asymmetric evaluation from 46.0% to 62.0% and public document-free math benchmark maj@8 accuracy from 20.2% to 35.4%, despite never using ground-truth answers (Stein et al., 24 Feb 2026). OmniOPSD reaches a state-of-the-art 84.19 average score on MER-UniBench and shows that rationale-privileged teacher guidance outperforms simply appending rationales into GRPO (Cheng et al., 14 Jun 2026). PTD-PO raises the 4B LVLM average from about 65.8% under GRPO to about 69.1%, while mitigating entropy collapse and outperforming RLVR and distillation baselines across 2B–8B models (Xiang et al., 5 Jun 2026). In safety alignment, COPSD achieves a stronger safety–helpfulness trade-off than standard OPSD or RL baselines and sharply reduces the “alignment tax” on MathVista relative to GRPO or Safe-RLHF-V (Wen et al., 2 Jun 2026).

Open questions are now explicit in the literature. Several papers ask whether learned routers could replace entropy heuristics, whether weaker or partial privileged signals are sufficient, how adaptive rationale selection or teacher refinement loops should be designed, how to analyze gate strictness as a coverage–reliability trade-off, and how these methods behave at scales beyond the reported $x$3B range in some studies (Zhang et al., 21 May 2026, Cheng et al., 14 Jun 2026, Stein et al., 24 Feb 2026, Penaloza et al., 4 Feb 2026). HDPO proposes an “expand-then-sharpen curriculum” as a possible way to recover pass@1 after widening solution support on cliff prompts (Ding, 25 Mar 2026). More generally, the combined evidence suggests that future work will likely focus less on making the privileged teacher stronger and more on making the privileged signal transferable, selective, uncertainty-aware, and compatible with the student’s inference-time information set.