Policy Distillation in Reinforcement Learning

• Policy Distillation is a model compression technique that transfers knowledge from a high-capacity teacher to a smaller student policy using divergence minimization over action distributions.
• It includes various algorithmic variants such as offline teacher-driven, student-driven on-policy, and hybrid methods incorporating regularization to tackle covariate shift and improve robustness.
• This approach enhances sample efficiency, interpretability, and real-world applicability across domains like robotics, multi-agent systems, and federated learning.

Policy distillation is a family of model compression and knowledge transfer techniques in deep reinforcement learning (RL) wherein a high-capacity "teacher" policy is used to supervise a smaller or more interpretable "student" policy, often through supervised learning of action distributions or Q-values. Initially introduced to compress deep Q-networks for Atari games, policy distillation has since evolved to support diverse objectives, architectures, and applications across classic control, robotics, multi-agent RL, federated scenarios, and interpretable policymaking. Methodological innovations have addressed issues such as covariate shift, performance under limited capacity, interpretability, sample efficiency, robustness, and decentralized or online learning paradigms.

1. Conceptual Foundations and Canonical Formulations

The core objective of policy distillation is to transfer policy knowledge from a teacher $\pi_T$ to a student $\pi_S$ by minimizing a divergence between their action distributions across a set of states. The objective commonly takes the form: $$L_{\text{distill}}(\pi_S; \pi_T) = \mathbb{E}_{s\sim D}\left[ D_{\text{KL}}(\pi_T(\cdot|s) \| \pi_S(\cdot|s)) \right]$$ where $D$ is a dataset of states, typically sampled by rolling out $\pi_T$ or $\pi_S$ in the environment. In early approaches, such as "Policy Distillation" (Rusu et al., 2015), the teacher is a fixed high-capacity RL policy, and the distillation is supervised, i.e., conducted offline without further RL interaction. The typical candidate loss functions include KL-divergence over action distributions, mean squared error on Q-values, negative log-likelihood of the teacher's optimal action, or temperature-scaled softmax cross-entropy.

Foundational extensions have introduced on-policy (student-driven) distillation, batch and online variants, regularization (such as entropy maximization), and hybrid schemes directly connecting trajectory distributions, as summarized in (Czarnecki et al., 2019).

2. Algorithmic Variants and Regularization

2.1 Classic Offline and Online Distillation

• Teacher-driven (offline) distillation: Supervised matching of teacher and student policies using trajectories sampled from the teacher, which suffers if the student diverges and visits unmodeled states (Czarnecki et al., 2019, Rusu et al., 2015).
• Student-driven (on-policy) distillation: Student samples trajectories from its own policy and matches the teacher at those states, preventing covariate shift but lacking a true gradient field without reward correction (Czarnecki et al., 2019).
• Hybrid and regularized objectives: Addition of negative entropy terms improves student exploration and convergence (Expected Entropy Regularized Distillation) (Czarnecki et al., 2019). Mixtures of distillation loss and actor-critic objectives allow leveraging both supervision and environmental feedback, as in Proximal Policy Distillation (PPD) (Spigler, 2024).

2.2 Task-Structured and Interpretable Distillation

• Neural-to-tree distillation with policy improvement criterion: Decision tree policies distilled from deep RL teachers are optimized with an advantage-based objective that penalizes poor actions in critical states, rather than simply cloning teacher actions. Regularizing with both advantage and imitation loss further improves generalization and stability, enabling high-fidelity, interpretable trees even with severe capacity constraints (Li et al., 2021).
• Selective Input Gradient Regularization: Combining policy distillation with input gradient regularization yields student policies whose input gradients (saliency maps) approximate computationally expensive perturbation-based saliency while retaining real-time performance and adversarial robustness (Xing et al., 2022).

2.3 Progressive, Multi-Teacher, and Fine-Tuning Approaches

• Multi-policy and scenario-aware distillation: In large-scale RL with domain randomization or federated/multi-agent setups, multiple specialized teachers are distilled into a single generalist student by aggregating their distributions and constructing a KL-based global objective (Khosravi et al., 9 Nov 2025, Jiang et al., 2 Feb 2025, Wadhwania et al., 2019).
• Online and real-time distillation: Teacher and student policies are updated simultaneously, with the student tracking the continuously improving teacher, reducing wall-clock time for distillation and allowing tiny student networks to reach high performance (Sun et al., 2019, Yu et al., 2024).
• Fine-tuning and task adaptation: Distilled students can be further improved by on-policy RL post-distillation, recovering or even exceeding teacher performance at a small fraction of the environment interaction cost (Green et al., 2019, Spigler, 2024).

3. Sample Efficiency, Robustness, and Generalization

Policy distillation is shown to significantly improve sample efficiency and enable model compression without large performance loss. Empirical and theoretical results demonstrate that:

• Distilled students of size 1.7–25% relative to teachers retain >90% performance on Atari and continuous control benchmarks (Rusu et al., 2015, Sun et al., 2019, Green et al., 2019, Spigler, 2024).
• Progressive-resolution or curriculum-based distillation (e.g., across simulator fidelity levels) achieves performance at fine levels with an order-of-magnitude less wall-clock time compared to training fresh or naively transferring coarse policies (Kadokawa et al., 2024, 2207.14561).
• Methods such as selective input-gradient regularization and advantage-based splitting improve transfer fidelity specifically in distribution-shift-prone or adversarial contexts (Li et al., 2021, Xing et al., 2022).

The value of distillation is especially pronounced when small models trained from scratch underperform, whereas distilled students approach or surpass teacher reward, BLER, or generalization metrics across unseen scenarios (Khosravi et al., 9 Nov 2025, Green et al., 2019).

4. Cooperative and Decentralized Distillation

Recent extensions address settings where teacher policies are unavailable or prohibitively expensive to train, focusing on peer-to-peer or federated learning:

• Dual/Peer Distillation: Dual Policy Distillation (DPD) and Online Policy Distillation with Decision Attention (OPD-DA) replace the fixed teacher with dynamically learning peers. Policies are updated using advantage-weighted KL divergence directed at better-performing peer actions in "disadvantageous" states, leading to mutual policy improvement that is empirically and theoretically justified (Lai et al., 2020, Yu et al., 2024).
• Federated Heterogeneous Distillation: Agents with heterogeneous architectures and training hyperparameters share action distributions over a small public set of states. The server averages these and broadcasts a global consensus policy, and each agent aligns locally by minimizing the KL divergence to the consensus (Jiang et al., 2 Feb 2025). Theoretical results show convergence to stationary points and reduced variance in policy gradient updates.
• Multi-agent value matching: In homogeneous multi-agent systems, value-matching complements policy distillation by aligning the critics of agents and the fused student, enabling continued learning post-distillation in possibly changing environments (Wadhwania et al., 2019).

5. Interpretability, Safety, and Real-World Applicability

• Rule Extraction and Decision Trees: Distillation into decision tree policies, as in Dpic and MSVIPER, yields explicit sensor-feature-to-action mappings, enabling human verification, formal safety analysis, and post-training policy modification (e.g., for freezing, oscillation, or vibration in robots) (Li et al., 2021, Roth et al., 2022).
• Saliency-Guided Distillation: Efficient generation of saliency maps from compact student networks facilitates rapid online interpretability, crucial for high-speed safety-critical settings such as autonomous driving, with minimal performance degradation and improved adversarial robustness (Xing et al., 2022).
• Deployment in Resource-Constrained Systems: In real-time radio access networks, distilled students are shown to satisfy tight runtime and memory constraints (<100μs/TTI and <1Mb), while preserving generalization and performance across challenging 5G/4G scenarios (Khosravi et al., 9 Nov 2025).

6. Limitations, Open Problems, and Future Directions

• Capacity and Domain Shift: Although policy distillation can compress policies by 10–30×, very small students can suffer in highly stochastic or covariate-shifted environments. Advantage-based or regularized objectives alleviate but do not always eliminate these issues (Li et al., 2021, Rusu et al., 2015).
• Access to Teacher Q-values: Many approaches require teacher networks that expose their Q-function or soft-Q outputs; policy-gradient-only teachers may need an auxiliary critic (Li et al., 2021).
• Extension to Continuous Action Spaces and Online RL: While classic distillation is well understood for discrete actions, robust approaches for continuous action policy distillation—especially under non-trivial environmental shift—remain an area of active research (Khosravi et al., 9 Nov 2025, Lai et al., 2020).
• Heterogeneous and Large-Scale Settings: Federated and online distillation for highly heterogeneous agents, especially with privacy or communication limits, presents scaling and stability challenges (Jiang et al., 2 Feb 2025, Yu et al., 2024).
• Joint Actor–Critic Distillation: Jointly distilling value (critic) functions and policies remains nontrivial but could further enhance performance and sample efficiency, especially when coupled with RL fine-tuning post-distillation (Wadhwania et al., 2019, Spigler, 2024).
• Interpretability and Verification: While tree and rule-based policies improve interpretability, the translation of neural policies to symbolic forms without excessive growth in tree size or critical error remains a challenge (Li et al., 2021, Roth et al., 2022).

Future work is expected to further unify distillation with curriculum learning, dynamic weighting of distillation signals, adaptive capacity scaling, and integration with meta- and transfer learning frameworks (Li et al., 2021, Spigler, 2024, Kadokawa et al., 2024).

---

References:

• "Policy Distillation" (Rusu et al., 2015)
• "Neural-to-Tree Policy Distillation with Policy Improvement Criterion" (Li et al., 2021)
• "Policy Distillation with Selective Input Gradient Regularization for Efficient Interpretability" (Xing et al., 2022)
• "Practical Policy Distillation for Reinforcement Learning in Radio Access Networks" (Khosravi et al., 9 Nov 2025)
• "FedHPD: Heterogeneous Federated Reinforcement Learning via Policy Distillation" (Jiang et al., 2 Feb 2025)
• "Proximal Policy Distillation" (Spigler, 2024)
• "Dual Policy Distillation" (Lai et al., 2020)
• "Distilling Policy Distillation" (Czarnecki et al., 2019)
• "Distillation Strategies for Proximal Policy Optimization" (Green et al., 2019)
• "Cyclic Policy Distillation: Sample-Efficient Sim-to-Real Reinforcement Learning with Domain Randomization" (2207.14561)
• "Real-time Policy Distillation in Deep Reinforcement Learning" (Sun et al., 2019)
• "Refined Policy Distillation: From VLA Generalists to RL Experts" (Jülg et al., 6 Mar 2025)
• "Progressive-Resolution Policy Distillation" (Kadokawa et al., 2024)
• "Online Policy Distillation with Decision-Attention" (Yu et al., 2024)
• "Policy Distillation and Value Matching in Multiagent Reinforcement Learning" (Wadhwania et al., 2019)
• "MSVIPER: Improved Policy Distillation for Reinforcement-Learning-Based Robot Navigation" (Roth et al., 2022)
• "DisCoRL: Continual Reinforcement Learning via Policy Distillation" (Traoré et al., 2019)
• "Evolutionary Stochastic Policy Distillation" (Sun et al., 2020)
• "WPT: World-to-Policy Transfer via Online World Model Distillation" (Jiang et al., 25 Nov 2025)