---
title: On-Policy Distillation Explained
url: https://www.emergentmind.com/topics/on-policy-distillation
type: topic
---

# On-Policy Distillation Explained

On-policy distillation (OPD) is a post-training paradigm in which a student model is optimized based on dense, token-level supervision from a stronger policy—either an external teacher, a self-teacher, or a hybrid—applied to trajectories sampled from the student's own policy. This approach fundamentally differs from off-policy knowledge distillation by providing the student with feedback on states it actually visits, rather than on fixed teacher-generated traces. OPD encompasses a spectrum of objectives, teacher access modes, and feedback signals, and has catalyzed rapid advances in model alignment, capacity transfer, and efficient reasoning across large-scale language and vision-language models.

## 1. Foundations and Core Principles

At its core, OPD defines an autoregressive student policy $\pi_\theta(y|x) = \prod_{t=1}^T \pi_\theta(y_t|x,y_{<t})$ and minimizes a divergence—commonly the reverse KL—between the student's trajectory distribution and a teacher policy (which may receive privileged context):

\[
\mathcal{L}_{\rm OPD}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, \, y \sim \pi_\theta(\cdot|x)} \left[\sum_{t=1}^T D_{\rm KL}\big(\pi_\theta(\cdot|x,y_{<t}) \, \| \, \pi^*(\cdot|x,y_{<t})\big)\right].
\]

By sampling rollouts on-policy from $\pi_\theta$, OPD eliminates exposure bias—where off-policy students never learn to recover from their own mistakes. Iteratively updating the student on distributions it generates itself, OPD reduces the theoretical compounding error rate from $O(\epsilon T^2)$ (off-policy) to $O(\epsilon T)$, consistent with DAgger-style analysis [2604.00626].

Dense supervision is typically at the token or chunk level, with the teacher providing probabilities, preferences, or structured feedback on each student-induced state. This process generalizes to self-distillation, context distillation, and various reward-regularized extensions.

## 2. Privileged Information and the "Privilege Illusion"

To expand the performance frontier, practices have emerged for infusing privileged information into either the teacher or student policy during training—examples include chain-of-thought, verified hints, tool descriptions, or bounding box annotations [2606.30626]. However, naive application of OPD under contextual privilege leads to the "privilege illusion": students appear to close the teacher–student gap during training by mimicking privileged cues unavailable at test time, rather than acquiring transferable capability.

The privilege illusion arises due to two factors:

- **Information asymmetry:** Tokens predicted with high confidence by a privileged teacher may encode shortcuts or context-specific artifacts not reproducible in deployment.
- **Non-uniform token-level signal:** Only a minority of tokens along a trajectory are truly capability-bearing; indiscriminate distillation over all tokens risks overfitting privilege-correlated behaviors and collapsing policy diversity.

Empirical analyses reveal entropy collapse, reduced exploration, and degraded generalization when all tokens are treated equally under dense privileged supervision [2606.30626].

## 3. Dual On-Policy Distillation: DOPD Paradigm

DOPD (Dual On-Policy Distillation) introduces an advantage-aware, token-wise distillation mechanism that routes supervision between a privileged teacher and a privileged student branch based on their relative advantage and token-wise confidences [2606.30626].

For each student-sampled trajectory and token $n$, DOPD computes:

\[
\mathcal{A}_n = |\log q_T - \log q_S| = \Big|\ln\frac{q_T}{q_S}\Big|
\]

where $q_T$ and $q_S$ are the privileged teacher and privileged student probabilities for $y_n$ given the contextual inputs (including privilege).

DOPD assigns each token to one of four regimes using empirical batch means $\bar{\mathcal{A}}$, $\bar{q}_T$, $\bar{q}_S$:

- **Low-Gap, High-Conf (LH):** Teacher and student both confident with small gap. Weak teacher distillation via light top-$K$ reverse KL.
- **Low-Gap, Low-Conf (LL):** Both uncertain. Self-regularization toward privileged student output.
- **High-Gap, Teacher-Favored (HT):** Large gap, high teacher confidence. Strong, full-vocabulary JS divergence distillation.
- **High-Gap, Student-Favored (HS):** Large gap, but student more confident. Light self-regularization.

This regime-routing ensures that genuine capability-bearing tokens (high advantage gap, high teacher confidence) receive maximal supervision, while privilege-dominant or noisy tokens receive lighter or self-distillation signals.

The final per-token DOPD objective is:

\[
\mathcal{L}_n^{\mathrm{DOPD}} = \mathbb{I}^{\mathrm{LH}}_n\,\mathcal{L}_n^{\mathrm{LH}} + \mathbb{I}^{\mathrm{LL}}_n\,\mathcal{L}_n^{\mathrm{LL}} + \mathbb{I}^{\mathrm{HT}}_n\,\mathcal{L}_n^{\mathrm{HT}} + \mathbb{I}^{\mathrm{HS}}_n\,\mathcal{L}_n^{\mathrm{HS}}
\]
with the global loss averaged over on-policy rollouts.

This architecture generalizes to vision-language and other multimodal settings, where, for example, object labels or bounding boxes serve as privileged context during DOPD [2606.30626].

## 4. Empirical Performance, Robustness, and Scalability

Extensive experiments validate DOPD across LLM and VLM tasks using multiple teacher–student scale pairs (up to 8B→0.6B) and diverse privileged information:

| Setting        | Vanilla OPD | DOPD   | Gap Recovery |
|:---------------|:-----------:|:------:|:------------:|
| LLM (Qwen3)    | 43.9        | 51.4   | 89.8%        |
| VLM (Qwen3-VL) | 52.4        | 58.4   | 69.2%        |

- DOPD consistently surpasses standard OPD and recent variants (ExOPD, Uni-OPD, EOPD, Self-Distillation) by large margins (4–8 points absolute), both in matched-scale and cross-scale settings.
- DOPD's gains increase as the teacher–student size ratio grows, maintaining robustness and stability where vanilla OPD plateaus or regresses.
- DOPD shows superior continual learning (retains nearly all acquired abilities across multi-stage curricula) and improved out-of-distribution generalization; e.g., training on reasoning, testing on coding yields 3–4 point improvements over leading baselines.
- Policy entropy curves under DOPD exhibit an initial controlled rise and moderate decay, avoiding the sharp entropy collapse seen in self-distillation, indicating healthy exploration and stable convergence.
- Ablations confirm that top-advantage tokens account for 50–80% of DOPD's relative performance; restricting strong teacher imitation to these tokens yields most of the measured gain.

## 5. Methodological Innovations and Theoretical Underpinnings

DOPD's core innovation is token-wise regime routing, justified by the observation that only tokens with high privilege advantage correspond to transferable capabilities. Unlike uniform KL minimization, DOPD's adaptive strategy mitigates privilege illusion by:

- Decoupling information-asymmetry from real competence: tokens where both privileged teacher and student agree (low gap) are likely privilege-dominated and do not warrant strong distillation.
- Assigning regime-specific divergence: strong teacher imitation (full-vocab JS or KL) for high-gap, teacher-favored tokens; lightweight self-regularization elsewhere.
- Empirical evidence indicates this approach captures nearly all the possible teacher–student gap, and ablation studies demonstrate its necessity for effective, robust distillation [2606.30626].

DOPD is algorithmically compatible with existing OPD architectures: tokens are dynamically classified per batch, with regime-specific loss terms aggregated in the main optimization loop.

## 6. Broader Implications and Generalization

The DOPD framework—optimal routing of token-level supervision between privileged teachers and privileged students via advantage gap—offers a blueprint for selective, context-aware knowledge transfer beyond LLMs and VLMs. The same principles can be adapted to continuous-control RL: on-policy agent updates can be informed by the per-state advantage gap between a privileged demonstrator and the current policy, focusing strong supervision on states exhibiting true capability distinctions [2606.30626].

This reframing aligns with recent perspectives that position OPD as a modular interface for routed feedback—where choices of state source, credit assignment, privilege masking, and routing are dynamically determined per context [2606.22793]. A plausible implication is the emergence of a new design space for hybrid OPD methods, integrating regime-aware routing, counterfactual probability reassignment, and privilege compressibility diagnostics.

## 7. Practical Considerations and Limitations

- DOPD's implementation presupposes batch-wise computation of the privilege advantage gap, as well as routine availability of both privileged teacher and student policies during training.
- The weights governing regime strength ($\beta_w, \beta_\ell$) can be tuned, with empirical optima reported at $(0.3, 0.6)$ [2606.30626].
- Forms of privileged information most beneficial for distillation are task-dependent: step-wise hints (sans execution traces) in LLMs and bounding boxes with labels in VLMs yield largest gains; over-privileging (e.g., final answer access) degrades performance.
- DOPD generalizes across model scales, domains, continual learning scenarios, and demonstrates strong OOD resilience, but requires further validation at very large scales and in multi-agent or partially observed domains. Residual privilege effects at extreme model gaps and stability in highly dynamic environments warrant further study.

---

**References**:

- "DOPD: Dual On-Policy Distillation" [2606.30626]
- "A Formula-Driven Survey and Research Agenda for On-Policy Distillation" [2606.22793]
- "A Survey of On-Policy Distillation for Large Language Models" [2604.00626]
- "Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization" [2605.05040]

Source: https://www.emergentmind.com/topics/on-policy-distillation