Multi-Teacher On-Policy Distillation

• Multi-Teacher On-Policy Distillation (MOPD) is a paradigm that combines knowledge distillation with reinforcement learning for adaptive teacher selection.
• It employs an RL-guided policy to dynamically weight multiple teachers, improving training convergence, accuracy, and robustness.
• Empirical results show that MOPD achieves superior performance on NLP and reasoning benchmarks while maintaining computational efficiency.

Multi-Teacher On-Policy Distillation (MOPD) refers to a class of training paradigms in which a student model learns from multiple teacher models in a manner guided by on-policy signals—typically leveraging reinforcement learning (RL) policy optimization to adaptively select the most beneficial knowledge transfer sources during training. MOPD unifies knowledge distillation and RL-guided teacher selection, enabling improved performance and robustness in settings such as model compression for NLP or reinforcement learning for multi-step reasoning. Representative frameworks include Reinforced Multi-Teacher Selection (RL-KD) and Adaptive Multi-Guidance Policy Optimization (AMPO), which implement dynamic weighting and guidance-on-demand selection over a set of candidate teachers (Yuan et al., 2020, Yuan et al., 2 Oct 2025).

1. Problem Formulation and Core Principles

The general MOPD framework involves a set of $N$ fixed teacher models $\mathcal{T} = \{T_1, T_2, \ldots, T_N\}$, each defining a conditional output distribution, and a student model $S(x; \varphi)$ with parameters $\varphi$. The objective is to optimize the student such that it both matches the performance of the ensemble of teachers and aligns with ground-truth labels, all while maintaining computational efficiency.

Unlike vanilla distillation approaches, which use static or uniformly weighted teacher ensembles, MOPD employs a teacher-selector (policy) $\pi_{\theta}$, which dynamically generates a weight vector $\mathbf{w} = [w_1, \ldots, w_N]$ (either soft or hard assignments) per training instance or mini-batch. The student’s loss function combines the standard supervised term $L_{\textrm{task}}(\varphi)$ and a weighted distillation term $L_{\textrm{KD}}(\varphi; x, \mathbf{w})$:

$$L_{\textrm{total}}(\varphi;\theta) = L_{\textrm{task}}(\varphi) + \lambda\,\mathbb{E}_{x\sim D,\,\mathbf{w}\sim\pi_{\theta}(\cdot\,|\,s(x,\varphi))}\left[L_{\textrm{KD}}(\varphi; x, \mathbf{w})\right]$$

with

$$L_{\textrm{KD}}(\varphi; x,\, \mathbf{w}) = \sum_{i=1}^{N} w_i\;D_{\textrm{KL}}\big(T_i(\cdot|x) \,\|\, S(\cdot|x; \varphi)\big)$$

(Yuan et al., 2020).

AMPO extends these principles, embedding dynamic teacher guidance within an RLVR (RL with Verifiable Rewards) setting; here, a binary flag $I$ governs whether off-policy teacher guidance is injected, dependent on the student's rollout performance (Yuan et al., 2 Oct 2025). Correct teacher traces are selected via a comprehension-based mechanism, emphasizing learnable knowledge transfer.

2. Reinforcement Learning Policy Design for Teacher Selection

The teacher-selection policy $\pi_{\theta}$ maps a state $s_t$—constructed from features such as student loss/logits, teacher outputs, and historical statistics—to a distribution over the teacher set. In RL-KD, the action is a softmax-weighted vector $\mathbf{w}$ output by a lightweight MLP ($\sim$3.1k parameters), operationalized as:

$$\mathbf{w} = \textrm{softmax}\left(f_\theta(s)\right)$$

Immediate reward signals can be either the negative KD loss on the current batch,

$$r_t = -L_{\textrm{KD}}(\varphi_t; x_t, \mathbf{w}),$$

or a dev accuracy increment,

$$r_t = \Delta \textrm{Acc}_{\textrm{dev}} = \textrm{Acc}_{\textrm{dev}}(\varphi_{t+1}) - \textrm{Acc}_{\textrm{dev}}(\varphi_{t}).$$

Policy parameters are updated via Monte-Carlo policy-gradient (REINFORCE):

$$\nabla_\theta J(\theta) = \mathbb{E}_{s,\,\mathbf{w}} \left[\nabla_\theta \log\pi_\theta(\mathbf{w}|s)\;[R(s, \mathbf{w}) - b(s)]\right]$$

with $b(s)$ a running baseline (Yuan et al., 2020).

In AMPO, the process incorporates a guidance-on-demand flag $I$, high-dimensional policy rollouts, and off-policy advantage normalization. Comprehension-based selection chooses only those teacher trajectories the student is most likely to reproduce:

$$r_p(o^{\textrm{off}}) = \textrm{clip}\left(\exp\left(\frac{1}{|y^*|}\sum_{\tau_i \in y^*}\log\pi_\theta(\tau_i | z^{\textrm{off}}, y^*_{<i})\right), 0, 1\right)$$

This directs distillation to comprehensible reasoning patterns, preventing information overload or misalignment (Yuan et al., 2 Oct 2025).

3. On-Policy Optimization Algorithms

Student and policy parameters are updated in alternating steps. The typical pseudocode for RL-KD involves:

1. Sampling a mini-batch and computing states and teacher weights,
2. Updating student parameters via gradient descent on $L_{\textrm{total}}$,
3. Computing policy rewards and updating selector parameters via REINFORCE,
4. Maintaining a baseline for variance reduction.

For AMPO, training unfolds as rollouts followed by evaluation of the guidance flag $I$. If $I=1$, comprehension-based teacher traces replace lowest-performing student trajectories in the batch. The advantage normalization occurs globally over the augmented batch, followed by clipped policy-gradient updates.

Framework	Teacher Selection	On-Policy Signal	Dynamic Weighting
RL-KD	Policy (MLP, softmax)	Immediate RL reward	Yes
AMPO	Guidance-on-demand + Comprehension	RLVR, batch-based	Yes

4. Empirical Results and Benchmark Analysis

Experiments for RL-KD utilized the QQP benchmark (GLUE), along with MNLI and SST-2. Teacher models included BERT$_{12}$, RoBERTa$_{12}$, XLNet$_{12}$, ALBERT$_{12}$; students were BERT$_6$ and BERT$_3$. RL-KD (MOPD) yielded superior QQP accuracy (90.4%) compared to vanilla KD (89.2%) and fine-tuning (87.9%), with statistically indistinguishable inference speeds across methods ($\approx$45 QPS at batch size 32, max length 128). Training overhead for RL-KD was moderate ($1.2–1.5\times$) relative to baseline KD due to on-policy optimization (Yuan et al., 2020).

AMPO was benchmarked on mathematical reasoning and out-of-distribution tasks using Qwen2.5-7B-Ins. AMPO achieved substantial performance improvements:

• In-distribution: 40.4% vs. 36.1% (GRPO baseline), +4.3 points
• Out-of-distribution: 64.2% vs. 52.0% (GRPO), +12.2 points
• Pass@$k$ diversity metrics and entropy confirmed richer exploration and avoidance of late-stage policy collapse (Yuan et al., 2 Oct 2025).

Notably, AMPO attained state-of-the-art results on key tasks with only 8.5k multi-teacher examples, matching single-teacher methods trained on substantially more data.

5. Mechanistic Insights and Implications

MOPD frameworks demonstrate that fixed equal weighting in standard multi-teacher distillation is suboptimal, failing to capitalize on the heterogeneous strengths of individual teachers or their performance on subdomains. On-policy adaptation discovers and exploits these complementary strengths, dynamically adjusting teacher influence in response to student needs and task complexity.

Empirical ablations revealed benefits from on-policy reward-driven teacher selection, showing that instantaneously beneficial teacher guidance accelerates convergence and enhances robustness, particularly on challenging examples where student knowledge is weak. Guidance-on-demand and comprehension-gating in AMPO suggest future scalability, enabling the incorporation of diverse teachers—including heterogeneous reasoning chains—without destabilizing learning or overwhelming the student.

A plausible implication is that as models and tasks grow more diverse, MOPD approaches will prove critical for stability and sample efficiency in large-scale multi-expert RL fine-tuning.

6. Scalability, Diversity, and Future Directions

MOPD frameworks are readily extensible. RL-KD’s teacher selector is a compact model (≲3.1k parameters), imposing negligible run-time cost. AMPO’s guidance-on-demand protocol and comprehension-driven selection are agnostic to teacher set cardinality and architecture, allowing flexible inclusion of LongCoT and ShortCoT models, peer-level or expert-level architectures, or even hybrid pools.

Experiments indicate that MOPD mechanisms such as AMPO preserve self-discovery while exploiting external guidance only when strictly necessary—a principle that maximizes exploration and maintains diversity in student reasoning paths. The approach offers a blueprint for scalable multi-expert distillation in future LLM reasoning tasks, enabling robust generalizability and data-efficient training (Yuan et al., 2 Oct 2025).

7. Related Work and Comparative Analysis

MOPD is closely related to traditional knowledge distillation but distinguishes itself through reinforcement learning-based, adaptive weighting. The RL-KD approach augments vanilla multi-teacher distillation, matching or exceeding performance while keeping inference speed constant (Yuan et al., 2020). AMPO advances prior RLVR work by introducing multi-teacher augmentation and comprehension gating, outperforming single-teacher distillation on critical metrics with a fraction of the data (Yuan et al., 2 Oct 2025). Contemporary trends increasingly focus on dynamic allocation of teacher influence and adaptive, policy-driven knowledge absorption, driven by the rapid evolution of LLM architectures and RL fine-tuning paradigms.

No controversies or misconceptions regarding the methodology are noted in these foundational papers; empirical evidence supports MOPD’s superiority over static multi-teacher baselines in both accuracy and diversity. Future research is likely to explore heterogeneous teacher composition, policy generalization, and data-efficient large-scale reasoning.