Multi-Teacher On-Policy Distillation

• Multi-Teacher On-Policy Distillation is an on-policy training method that leverages multiple specialized teachers to supervise student rollouts.
• It uses diverse teacher aggregation strategies—such as confidence-weighted debates, hard routing, and decision-attention—to provide dense token- or trajectory-level guidance.
• Empirical results across language, image, and RL tasks show that this approach overcomes single-teacher limitations, enhancing performance and training stability.

Multi-Teacher On-Policy Distillation (OPD) trains a student on its own generated distribution while multiple teachers supply supervision on the student’s trajectories. In LLMs, the supervision is commonly token-level on student-generated rollouts; in flow-matching text-to-image models it is dense trajectory-level supervision over student-sampled latent paths; and in online reinforcement-learning settings it can take the form of group-derived targets computed on the states each policy visits. The method is used to break the single-teacher capability ceiling, integrate multiple domain-specialized RL teachers into one student, and replace sparse scalar rewards with dense teacher signals (Wang et al., 2 May 2026, Ma et al., 29 Jun 2026, Fang et al., 8 May 2026, Yu et al., 2024).

1. Formal definition and objective

The standard OPD formulation trains a student $\pi_\theta$ on its own rollouts while a teacher $\pi^*$ provides dense, token-level supervision by force-decoding the student’s tokens. In one canonical formulation,

$$\mathcal{L}_{opd}(\theta) = E_{x,\,\hat{y} \sim \pi_\theta} \left[ \frac{1}{|\hat{y}|} \sum_{t=1}^{|\hat{y}|} D\big(\pi^*(\cdot | x, c, \hat{y}_{<t}) \;\big\|\; \pi_\theta(\cdot | x, \hat{y}_{<t})\big) \right],$$

where $D$ is a divergence and $c$ denotes privileged information available only to the teacher. In the debate-based formulation, $c$ is the multi-agent debate transcript (Wang et al., 2 May 2026).

A sequence-level view expresses OPD as reverse-KL on student-generated trajectories: $$\mathcal{J}_{\mathrm{OPD}}(\theta) = \min_{\theta}\; \mathbb{E}_{x\sim D,\,y\sim \pi_{\theta}(\cdot\mid x)} \!\left[ \mathrm{KL}\big(\pi_{\theta}(y\mid x)\,\|\,\pi_T(y\mid x)\big) \right].$$ This framing is central to Generalized On-Policy Distillation, which shows that OPD is a special case of dense KL-constrained RL in which the reward function and the KL regularization are always weighted equally and the reference model can be any model (Yang et al., 12 Feb 2026).

Multi-teacher extensions preserve the on-policy condition but replace the single teacher with either a weighted set of teacher divergences or a routed specialist teacher. One explicit language-model formulation is

$$\mathcal{L}_{mad\text{-}opd}(\theta) = E_{s_m \sim \mathcal{D},\, \hat{y} \sim \pi_\theta(\cdot \mid s_m)} \!\left[ \frac{1}{|\hat{y}|} \sum_{t=1}^{|\hat{y}|} \sum_{k=1}^{K} w_k \cdot D\big(p_{T_k}(\cdot \mid s_m, \mathcal{H}_m^R, \hat{y}_{<t}) \;\big\|\; p_S(\cdot \mid s_m, \hat{y}_{<t})\big) \right].$$

A related context-distillation extension defines a weighted sum of reverse KLs against several context-conditioned teachers, and also describes a forward-KL-to-mixture alternative when a mode-covering target is preferred (Wang et al., 2 May 2026, Ye et al., 12 Feb 2026).

2. Teacher aggregation and routing

Multi-teacher OPD differs most sharply in how it constructs the effective teacher. Some methods aggregate teachers simultaneously; others route each prompt, trajectory, or state to a single teacher; still others select one rollout from a candidate pool.

Mechanism	Representative formulation	Representative papers
Confidence-weighted debate	$K=2$, $R=2$, multi-round teacher debate, post-debate confidence softmax weights	(Wang et al., 2 May 2026)
Hard routing	Prompt or trajectory routed to a domain-matched teacher	(Ma et al., 29 Jun 2026, Fang et al., 8 May 2026)
Decision-Attention	Leave-one-out, state-dependent attention weights over peer policies	(Yu et al., 2024)
Best-of-$\pi^*$0 selection	Correctness first, student alignment second	(Zhang et al., 10 May 2026)

In MAD-OPD, the teacher is recast as a deliberative collective. At each decision point, teachers run an $\pi^*$1-round debate; the full debate history $\pi^*$2 becomes the teacher-only privileged context; and each teacher self-reports a confidence score extracted from its structured output. Scores are scaled, softmax-normalized with $\pi^*$3, and used to weight the per-teacher divergences. This turns supervision into a confidence-weighted, debate-conditioned token-level signal rather than a naive uniform ensemble (Wang et al., 2 May 2026).

In MOPD and Flow-OPD, the defining mechanism is routing rather than simultaneous aggregation. MOPD uses hard routing: each prompt carries a known domain label, and only the dispatched teacher supervises the student’s rollout. Flow-OPD likewise routes each prompt to the unique appropriate teacher by a rule-based mapping over dataset partitions, then applies dense supervision along the student’s sampled latent trajectory (Ma et al., 29 Jun 2026, Fang et al., 8 May 2026).

Online Policy Distillation with Decision-Attention replaces a fixed teacher pool with a cohort of policies trained online in the same environment. For each student policy and state, Decision-Attention computes state-dependent weights over the other policies, forms leave-one-out group-derived actor, critic, and feature targets, and uses those targets as auxiliary distillation losses inside PPO or DQN training. The design goal is to avoid the rapid homogenization caused by naive averaging (Yu et al., 2024).

BRTS uses a different construction: it samples a small pool of teacher trajectories and selects the auxiliary trajectory using a simple priority rule, correctness first, student alignment second. If unconditioned teacher samples fail on harder prompts, it invokes a ground-truth-conditioned recovery step and then uses the selected trajectory in a teacher-context supervision branch alongside the standard student-context OPD branch (Zhang et al., 10 May 2026).

3. Divergence design and optimization geometry

A major theme in the literature is that multi-teacher OPD is not defined solely by teacher aggregation; it is also defined by the divergence geometry used to compare student and teacher distributions. MAD-OPD derives a task-adaptive divergence principle. For agentic tasks it selects $\pi^*$4, with the bound

$\pi^*$5

and argues that the per-token logit gradient is uniformly bounded independent of support overlap. For code generation it selects reverse KL, and proves a mode-concentration property for teacher mixtures with disjoint supports: gradient descent on $\pi^*$6 drives $\pi^*$7 to concentrate on the dominant mode, which is presented as the reason reverse KL avoids splicing across incompatible coherent code paths (Wang et al., 2 May 2026).

Generalized On-Policy Distillation broadens this view by introducing a flexible reference model and a reward scaling factor $\pi^*$8. Its generalized objective is

$\pi^*$9

When $$\mathcal{L}_{opd}(\theta) = E_{x,\,\hat{y} \sim \pi_\theta} \left[ \frac{1}{|\hat{y}|} \sum_{t=1}^{|\hat{y}|} D\big(\pi^*(\cdot | x, c, \hat{y}_{<t}) \;\big\|\; \pi_\theta(\cdot | x, \hat{y}_{<t})\big) \right],$$0, the method becomes ExOPD, or reward extrapolation. In the multi-teacher setting studied there, the paper uses a mixture-of-domains formulation in which a domain is sampled, the student rolls out on-policy on a prompt from that domain, and only the corresponding teacher contributes to the dense reward term (Yang et al., 12 Feb 2026).

In Flow-OPD, the optimization geometry is continuous-time rather than autoregressive. Under SDE-based stochasticity, student and teacher transitions at each step are Gaussians with shared isotropic covariance, and the per-step reverse KL reduces to a scaled $$\mathcal{L}_{opd}(\theta) = E_{x,\,\hat{y} \sim \pi_\theta} \left[ \frac{1}{|\hat{y}|} \sum_{t=1}^{|\hat{y}|} D\big(\pi^*(\cdot | x, c, \hat{y}_{<t}) \;\big\|\; \pi_\theta(\cdot | x, \hat{y}_{<t})\big) \right],$$1 difference of means, which in turn reduces to a scaled $$\mathcal{L}_{opd}(\theta) = E_{x,\,\hat{y} \sim \pi_\theta} \left[ \frac{1}{|\hat{y}|} \sum_{t=1}^{|\hat{y}|} D\big(\pi^*(\cdot | x, c, \hat{y}_{<t}) \;\big\|\; \pi_\theta(\cdot | x, \hat{y}_{<t})\big) \right],$$2 difference of vector fields: $$\mathcal{L}_{opd}(\theta) = E_{x,\,\hat{y} \sim \pi_\theta} \left[ \frac{1}{|\hat{y}|} \sum_{t=1}^{|\hat{y}|} D\big(\pi^*(\cdot | x, c, \hat{y}_{<t}) \;\big\|\; \pi_\theta(\cdot | x, \hat{y}_{<t})\big) \right],$$3 This is the basis for its dense trajectory-level supervision in flow-matching models (Fang et al., 8 May 2026).

Uni-OPD adds a distinct criterion: aggregated token-level guidance should remain order-consistent with the outcome reward. It defines a trajectory-level distillation return

$$\mathcal{L}_{opd}(\theta) = E_{x,\,\hat{y} \sim \pi_\theta} \left[ \frac{1}{|\hat{y}|} \sum_{t=1}^{|\hat{y}|} D\big(\pi^*(\cdot | x, c, \hat{y}_{<t}) \;\big\|\; \pi_\theta(\cdot | x, \hat{y}_{<t})\big) \right],$$4

and then calibrates prompt-level margins so that correct trajectories are ranked above incorrect ones with a safety margin $$\mathcal{L}_{opd}(\theta) = E_{x,\,\hat{y} \sim \pi_\theta} \left[ \frac{1}{|\hat{y}|} \sum_{t=1}^{|\hat{y}|} D\big(\pi^*(\cdot | x, c, \hat{y}_{<t}) \;\big\|\; \pi_\theta(\cdot | x, \hat{y}_{<t})\big) \right],$$5 (Hou et al., 5 May 2026).

4. Domain-specific realizations

In LLM post-training, multi-teacher OPD is used primarily for capability integration and agentic supervision. MOPD organizes this as a three-stage pipeline: Stage 1 general SFT, Stage 2 domain-specialized RL to obtain frozen teachers, and Stage 3 on-policy distillation on the student’s own rollouts. The paper reports that MOPD has been deployed in the post-training of MiMo-V2-Flash, an industrial-scale frontier model, and emphasizes that teachers are trained independently and in parallel, each with its own RL recipe, before a later integration stage (Ma et al., 29 Jun 2026).

MAD-OPD extends this paradigm to agentic tasks by introducing On-Policy Agentic Distillation (OPAD), which adds step-level environment interaction and per-step debate-supervision. At step $$\mathcal{L}_{opd}(\theta) = E_{x,\,\hat{y} \sim \pi_\theta} \left[ \frac{1}{|\hat{y}|} \sum_{t=1}^{|\hat{y}|} D\big(\pi^*(\cdot | x, c, \hat{y}_{<t}) \;\big\|\; \pi_\theta(\cdot | x, \hat{y}_{<t})\big) \right],$$6, the student samples an action $$\mathcal{L}_{opd}(\theta) = E_{x,\,\hat{y} \sim \pi_\theta} \left[ \frac{1}{|\hat{y}|} \sum_{t=1}^{|\hat{y}|} D\big(\pi^*(\cdot | x, c, \hat{y}_{<t}) \;\big\|\; \pi_\theta(\cdot | x, \hat{y}_{<t})\big) \right],$$7, the environment returns observation $$\mathcal{L}_{opd}(\theta) = E_{x,\,\hat{y} \sim \pi_\theta} \left[ \frac{1}{|\hat{y}|} \sum_{t=1}^{|\hat{y}|} D\big(\pi^*(\cdot | x, c, \hat{y}_{<t}) \;\big\|\; \pi_\theta(\cdot | x, \hat{y}_{<t})\big) \right],$$8, teachers debate on $$\mathcal{L}_{opd}(\theta) = E_{x,\,\hat{y} \sim \pi_\theta} \left[ \frac{1}{|\hat{y}|} \sum_{t=1}^{|\hat{y}|} D\big(\pi^*(\cdot | x, c, \hat{y}_{<t}) \;\big\|\; \pi_\theta(\cdot | x, \hat{y}_{<t})\big) \right],$$9 to produce $D$0, and the total trajectory loss sums per-step contributions. The explicit target is to stabilize training when errors compound across long trajectories (Wang et al., 2 May 2026).

Tool-use models expose a narrower but especially revealing multi-teacher setting. In the two-teacher setup studied in behavior leverage imbalance, one teacher specializes in tool calls and one in direct responses. Routing is example-level rather than soft. The paper shows that vanilla generalized knowledge distillation improves tool-call recall but also moves the model toward over-calling, which motivated a new per-token divergence calibration method rather than a new teacher router (Shen et al., 8 Jul 2026).

CaMOPD addresses a different LLM scenario: a domain-specialized student that has lost general capability. It uses two branches, a general-recovery branch supervised by a general teacher and a domain-preservation branch supervised by a domain teacher, together with a 3-1 alternating schedule and branch-specific sample selection under incomplete prompt coverage. The stated goal is general capability recovery with domain preservation (Chen et al., 26 May 2026).

In flow-matching text-to-image models, Flow-OPD is presented as the first unified post-training framework that integrates on-policy distillation into Flow Matching models. It first cultivates domain-specialized teacher models via single-reward GRPO fine-tuning, then establishes a robust initial policy through a Flow-based Cold-Start scheme, and finally consolidates heterogeneous expertise into a single student via on-policy sampling, task-routing labeling, and dense trajectory-level supervision. Manifold Anchor Regularization adds a task-agnostic teacher to anchor generation to a high-quality manifold (Fang et al., 8 May 2026).

CollectionLoRA specializes the same idea to customized image editing. It treats each effect LoRA as a teacher, distills up to 50 different effect LoRAs together with few-step generation into a single student LoRA, and introduces three task-specific components: Probabilistic Dual-Stream Routing, Asymmetric Orthogonal Prompting, and a Coarse-to-Fine Distillation Objective. The formulation is still on-policy: teacher signals are evaluated on states the student itself visits during training (Wu et al., 25 May 2026).

In online RL, OPD-DA realizes multi-teacher OPD by training a cohort of policies in the same environment and letting each policy distill from the group’s decisions with state-dependent attention weights, while preserving on-policy updates for the base RL algorithm, primarily PPO. The emphasis there is not domain transfer but online peer-to-peer knowledge transfer without a pre-trained teacher (Yu et al., 2024).

5. Failure modes, diagnostics, and corrective mechanisms

The earliest limitation emphasized in recent LLM work is the single-teacher capability ceiling. In single-teacher OPD, when the teacher errs on the student’s on-policy state, the per-token divergence becomes small and the student inherits the error. The paper presents this as especially severe in agentic tasks, where per-step mistakes derail subsequent steps, and reports that the strongest single-teacher OPD falls short of the debate-based multi-teacher alternative across settings (Wang et al., 2 May 2026).

A common aggregate explanation is that behavior shifts are explained by total token exposure or average per-token divergence. The behavior leverage imbalance study rejects that explanation in a two-teacher tool-use setting. It reports that tool-call samples do not receive more token exposure, and full-sequence average per-token JSD is not larger for the tool-call teacher. Instead, local token-level signals at mode-entry and structural positions, such as <tool_call> and function names, can have disproportionate control over the global generation mode. Soft Clamp addresses this by dynamically compressing extreme token-level JSD while preserving nonzero gradients; on APIGen-MT it reduces over-calling from 13.7% to 9.0% relative to vanilla GKD while matching its decision accuracy (Shen et al., 8 Jul 2026).

CaMOPD identifies two further failure modes under incomplete teacher coverage: recovery-preservation counteraction and weak-signal flattening. Recovery-preservation counteraction arises when recovery and preservation gradients are mixed in the same update and remain negatively aligned; weak-signal flattening arises when full-batch averaging assigns equal weight to samples with unequal correction demand. The proposed response is decoupled alternating training plus gap-based sample selection, with branch-specific scoring functions and a mass-target selection rule (Chen et al., 26 May 2026).

Teacher mismatch can also destabilize otherwise effective on-policy distillation. MOPD reports that same-origin teachers are critical to stable optimization. Replacing the Math teacher with a stronger but distributionally different model raises the initial KL, destabilizes training, and can lead to catastrophic collapse in the Top-k MOPD form. The paper treats this as a distribution-mismatch failure rather than a simple teacher-strength effect (Ma et al., 29 Jun 2026).

Other limitations are infrastructural rather than algorithmic. MAD-OPD requires teacher access to token-level distributions and shared vocabulary, which excludes black-box API-only teachers; its training cost scales linearly with $D$1; and excessive debate depth harms performance because of prompt-context bloat. Flow-OPD similarly notes routing misassignment, teacher ceilings, and risk of over-anchoring with MAR as open constraints (Wang et al., 2 May 2026, Fang et al., 8 May 2026).

6. Empirical record and deployment practice

Across representative systems, the empirical record shows that multi-teacher OPD is used both to improve absolute capability and to integrate heterogeneous teachers into one student.

System	Setting	Reported outcome
MAD-OPD	Qwen3 14B+8B$D$24B	Ag-Avg 23.26 $D$3 25.69; Co-Avg 44.41 $D$4 48.12
Flow-OPD	Stable Diffusion 3.5 Medium	GenEval 63 $D$5 92; OCR 59 $D$6 94
MOPD	Qwen3-30B-A3B	$D$7 vs Mix-RL 0.882 and Off-Policy Finetune 0.824

MAD-OPD reports that, across six teacher-student configurations and five agentic and code benchmarks, it ranks first across all six configurations, and in the highlighted Qwen3 14B+8B$D$84B setting it lifts the agentic average by $D$9 and the code average by $c$0 over the stronger single-teacher OPD. It also reports cases where a 4B student exceeds its 14B teacher on LiveCodeBench v6 (Wang et al., 2 May 2026).

Flow-OPD reports that, built upon Stable Diffusion 3.5 Medium, it raises the GenEval score from 63 to 92 and the OCR accuracy from 59 to 94, yielding an overall improvement of roughly 10 points over vanilla GRPO, while preserving image fidelity and human-preference alignment and exhibiting an emergent teacher-surpassing effect. CollectionLoRA reports that it distills all customized effects and few-step generation into a single LoRA, with EffectBench metrics of CLIP 0.727, DreamSim 0.425, VSA 4.380, EditReward 1.052, BCR 0.087, and NFE 8, while reducing storage and routing overhead relative to loading many separate effect LoRAs (Fang et al., 8 May 2026, Wu et al., 25 May 2026).

MOPD reports that, on Qwen3-30B-A3B, it outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines, inheriting nearly all of each teacher’s capability. It also reports that MOPD reaches teacher-level plateau on IF in approximately 25K domain samples and SWE in approximately 30K, versus 150–180K for Mix-RL. Uni-OPD, in turn, reports consistent gains over standard OPD in both single-teacher and multi-teacher settings across LLMs and MLLMs, including strong-to-weak and cross-modal distillation (Ma et al., 29 Jun 2026, Hou et al., 5 May 2026).

Practical recommendations recur across these systems. MAD-OPD recommends starting with $c$1, $c$2, post-debate self-reported confidence scores, and full-vocabulary logits from teachers for correct gradients. MOPD emphasizes same-origin teachers, asynchronous teacher prefill services, and payload reduction through top-k teacher outputs. Flow-OPD reduces overhead by hard routing to a single teacher per trajectory or prompt rather than querying all teachers, and CollectionLoRA uses a single deployable LoRA rather than runtime composition of many effect LoRAs and an accelerator (Wang et al., 2 May 2026, Ma et al., 29 Jun 2026, Fang et al., 8 May 2026, Wu et al., 25 May 2026).

Taken together, these results define Multi-Teacher On-Policy Distillation not as a single algorithm but as a class of on-policy training procedures in which teacher multiplicity is exploited through routing, aggregation, debate, attention, or selection, while stability is managed through divergence choice, order calibration, sample selection, and teacher-distribution alignment.