MTKD-RL: Adaptive Multi-Teacher Distillation

Updated 15 April 2026

The paper introduces a reinforcement learning policy that dynamically assigns teacher weights, enabling instance-wise adaptive knowledge distillation.
The methodology formulates teacher selection as a Markov decision process using state features and policy gradients to optimize student performance.
The approach yields consistent improvements across NLP, vision, and image forensics, outperforming static or heuristic teacher weighting methods.

Multi-Teacher Knowledge Distillation with Reinforcement Learning (MTKD-RL) is an advanced paradigm for neural network compression and transfer learning wherein a student model assimilates knowledge from a pool of expert teachers. It leverages reinforcement learning (RL) to dynamically adapt or assign teacher contributions, addressing the limitations of static, heuristic, or hand-crafted weighting schemes in conventional multi-teacher knowledge distillation (MTKD). MTKD-RL has been developed in NLP, computer vision, and image forensics, and is also subsumed within recent on-policy distillation formalisms.

1. Problem Formulation and Rationale

The objective in MTKD-RL is to train a compact student model that inherits both generic and specialized capabilities present in a set of pre-trained teacher models $\{T_i\}_{i=1}^N$ . Unlike classic distillation, which often averages teacher predictions or assigns fixed per-teacher weights, MTKD-RL introduces a reinforcement-learned policy to select or weight teachers instance-wise. This dynamic adaptation targets not just a better match to each example’s complexity but also the compatibility between student and teacher distributions.

Mathematically, for input $x$ and label $y$ , the MTKD loss is: $L_{\mathrm{KD}}(x; \theta_s, \phi) = \alpha\,\mathrm{CE}(y, p_s(\cdot\,|\,x;\theta_s)) + (1-\alpha)\sum_{i=1}^N a_i(x;\phi)\, \mathrm{KL}(p_{T_i}(\cdot\,|\,x) \Vert p_s(\cdot\,|\,x;\theta_s))$ where $a_i(x;\phi)$ is determined by an RL-based policy $\pi_\phi(\cdot)$ , and $\mathrm{CE}$ , $\mathrm{KL}$ denote cross-entropy and KL divergence, respectively (Yuan et al., 2020).

This flexible, feedback-driven weighting provides substantial gains over static MTKD, especially in the presence of teachers with diverse competencies and when labeled data is heterogeneous or contains multiple modalities (Yang et al., 22 Feb 2025, Yu et al., 7 Apr 2025).

2. Reinforcement Learning Formulation

MTKD-RL recasts teacher selection or weighting as a Markov decision process, with a “teacher selector” agent parameterized by $\phi$ choosing an action $a$ (e.g., convex mixing weights or binary assignments) given state information $x$ 0. The key RL elements are:

State ( $x$ 1): Typically comprises concatenations of student logits, teacher logits, teacher performance metrics, student-teacher gap features (KL, cosine similarity), and optionally ground-truth labels (Yuan et al., 2020, Yang et al., 22 Feb 2025).
Action ( $x$ 2): Real-valued (softmax) weights for each teacher, or binary indicators for selection (Yuan et al., 2020, Yu et al., 7 Apr 2025).
Reward ( $x$ 3): Measures the student’s improvement or loss after a knowledge distillation update with selected teacher weights. This may be defined as:

$x$ 4

where $x$ 5 is the student after the update (Yuan et al., 2020), or a composite of negative distillation loss and positive task metrics (Yu et al., 7 Apr 2025).
Policy Update: The agent parameters $x$ 6 are updated via policy-gradient (REINFORCE) according to:

$x$ 7

This architecture converges to an adaptive teacher allocation that maximizes student learning progress as observed through reward signals.

3. Algorithmic Frameworks and Variants

3.1. Generic MTKD-RL Workflow

The typical training loop for MTKD-RL (as in (Yuan et al., 2020, Yang et al., 22 Feb 2025, Yu et al., 7 Apr 2025)) comprises:

Compute teacher and student outputs for each data instance; form the composite state.
Sample normalized teacher weights (actions) from the policy network.
Update the student model via the weighted KD loss.
Evaluate post-update rewards (on a held-out batch or via current task metrics).
Update the policy network with policy gradient.
Optionally alternate optimization of student and policy (e.g. freeze one while updating the other for stability).

3.2. Modalities and Specializations

Visual Recognition (MTKD-RL for Vision): States encapsulate features and logit vectors, teacher-student similarity, and direct performance metrics. Actions are per-sample, per-task weightings for both logit- and feature-based KD. Rewards are immediate and negative of the multi-teacher KD loss. Policy networks can be compact with shared trunk and per-teacher heads (Yang et al., 22 Feb 2025).
Image Forgery Detection (Re-MTKD): Combines batchwise teacher selection (binary action per teacher) with an RL agent that incorporates student progress (segmentation F1, accuracy) in the reward. The RL state concatenates static and dynamic representation features and teacher confidence summaries. The student is an encoder-decoder (Cue-Net) with edge-aware modules (Yu et al., 7 Apr 2025).
Language Tasks (Reinforced Multi-Teacher Selection): Actions are convex combinations of teacher logits per instance, learned by a 2-layer MLP policy. Rewards are computed as downstream CE improvement, using a small held-out batch. The process achieves instance-wise teacher adaptation in model compression for transformers (Yuan et al., 2020).

3.3. On-Policy Multi-Teacher Distillation (G-OPD/ExOPD)

Recent advances have shown that multi-teacher distillation can be unified within a KL-constrained RL framework (Yang et al., 12 Feb 2026). Here, each domain expert is first obtained via reward-maximizing RL; then their knowledge is merged into a single student by maximizing: $x$ 8 with $x$ 9 (reward extrapolation). The student samples trajectories on a union of teacher domains, and the teacher corresponding to each domain provides token-level reward signals. Empirical results demonstrate consistent performance boosts over both single-teacher and multi-teacher methods, with the student outperforming all experts (Yang et al., 12 Feb 2026).

4. Quantitative Effects and Empirical Evaluation

MTKD-RL methods consistently yield measurable performance improvements over static or heuristic teacher weighting:

Natural Language Tasks: Achieve +0.8 to +1.1 F1 over vanilla KD, with negligible inference overhead (Yuan et al., 2020).
Visual Recognition: Top-1 accuracy gains of +0.33% (CIFAR-100), +0.49 to +0.80% (ImageNet CNNs/ViTs), and large downstream improvements for detection and segmentation (e.g., +1.1 to +1.5 mAP or mIoU) (Yang et al., 22 Feb 2025).
Forgery Detection: Improvements in detection/segmentation AUC by ~0.16–0.17 on low-level image tampering benchmarks, substantially outperforming uniform KD or single-teacher baselines (Yu et al., 7 Apr 2025).
Math and Code Reasoning (On-Policy ExOPD): Student model achieves +1.7 average gain (math) and +0.8 (code) over the best individual teacher (Yang et al., 12 Feb 2026).

A summary of empirical results in different domains is presented below:

Domain / Task	Gain over Best Baseline	Student Overhead
NLP (GLUE, QQP)	+0.8 to +1.1 F1	Policy: +3.1K params
CIFAR-100	+0.33% top-1 acc	+0.9GB memory
ImageNet	+0.49–0.80% top-1 acc	-
Forgery Detect.	+~0.17 AUC	-
Math/Code (LMMs)	+1.7/0.8 avg points	-

5. Architectures and Implementation Practices

Teacher Pool: Usually heterogeneous, e.g., BERTs, ResNets, Vision Transformers, domain-specific experts.
Policy Network: 2-layer MLP (NLP), trunk/head MLP (Vision), logistic per-teacher policy (Forgery). Lightweight, typically $y$ 0 total parameters.
State Construction: Includes high-dimensional feature fusion. In vision, both teacher performance and teacher-student gap features are required to maximize RL reward and student performance (Yang et al., 22 Feb 2025).
Reward Stabilization: Moving-average baselines, entropy regularization, delayed/mini-batched updates, and multi-modal reward functions are routinely adopted for training stability (Yuan et al., 2020, Yu et al., 7 Apr 2025).
Training Regimes: Alternating updates (student parameters vs. RL policy) shown to improve stability and convergence, especially in large-scale settings (Yang et al., 22 Feb 2025).
Batch Scheduling: Balanced sampling from all teacher domains is used when applicable, ensuring uniform coverage of teacher expertise (Yang et al., 12 Feb 2026).

G-OPD and its extrapolation variant (ExOPD) (Yang et al., 12 Feb 2026) recast on-policy distillation as dense KL-constrained RL, subsuming both classical MTKD-RL and RL-based teacher selection. In this view, reward scaling $y$ 1 and variable references $y$ 2 afford unified control of reward vs. regularization, enabling “learning beyond the teacher” by explicit reward extrapolation. Empirically, this yields not only sizable accuracy gains but enables the student to exceed the teacher in performance bounds, especially when merging orthogonal domain expert RL teachers.

Common themes emerging from ablations include:

RL-based policies consistently outperform static teacher weighting.
Inclusion of both teacher performance and teacher-student fit features in policy state maximizes RL reward (Yang et al., 22 Feb 2025).
Dynamic teacher assignment enables the student to avoid overfitting to weak or redundant teachers, adjusting focus over training (Yu et al., 7 Apr 2025).
Training cost increases moderately (20–70%), but inference overhead is minimal.

7. Impact and Research Significance

MTKD-RL establishes a principled, generalizable framework for adaptive, reward-driven knowledge transfer in student models across diverse modalities and scales. By optimizing teacher allocations end-to-end with respect to student progress, it enables effective compression, domain merging, and performance extrapolation in multi-domain and multi-teacher settings. Ongoing developments—such as reward signal extrapolation, unified KL-constrained objectives, and advanced state/action design—continue to expand its practical and theoretical impact in deep learning (Yuan et al., 2020, Yang et al., 22 Feb 2025, Yu et al., 7 Apr 2025, Yang et al., 12 Feb 2026).