Dual Policy Distillation

• Dual Policy Distillation is a framework that trains two or more policies concurrently to share knowledge and improve learning efficiency in tasks like reinforcement learning and diffusion-based modeling.
• It employs loss functions such as weighted regression, KL, and Jensen–Shannon divergence to align policy distributions and selectively update policies based on peer performance.
• Empirical results show that dual policy approaches boost performance, sample efficiency, and inference speed, while theoretical guarantees support policy improvement and stability.

Dual Policy Distillation refers broadly to learning frameworks in which two or more policy models are trained in conjunction, either sharing knowledge with each other or leveraging distinct perspectives to enhance learning and generalization. This stands in contrast to the canonical student-teacher paradigm, which typically distills knowledge from a fixed, pre-trained teacher policy into a student. Dual policy approaches—whether student-student, teacher-teacher, or mixed—have emerged in a range of settings including reinforcement learning (RL), supervised sequence modeling, and diffusion-based policies, yielding improvements in sample efficiency, performance on hard tasks, and distributional robustness.

1. Formal Frameworks for Dual Policy Distillation

Several algorithmic formulations instantiate the concept of dual policy distillation:

• Student–Student Peer Distillation: “Dual Policy Distillation” (DPD) in RL (Lai et al., 2020) maintains two independently-initialized policies (πθ, πϕ), each learning via both an RL objective and an auxiliary distillation loss from its peer. Each model interacts with its own environment copy, collects trajectories, and periodically updates towards the peer’s action distribution—selectively or on all states.
• Teacher–Teacher Dual Distillation: In score-based policy distillation for visuomotor control (Jia et al., 2024), a single-step student generator receives guidance from both a frozen “reference” teacher and an adaptive “adversarial” teacher, each initialized from a common pretrained diffusion policy. The student is optimized to match both the score function and the overall action distribution of these teachers.
• Self–Distillation with Adaptive Contexts: Hybrid Distillation Policy Optimization (HDPO) (Ding, 25 Mar 2026) realizes dual roles within a single parameterization by switching the input context: the base model acts as “student” on ordinary prompts and as “teacher” on privileged (ground-truth-augmented) prompts, distilling the latter’s distributions into the former on cases where exploration fails.
• Mixed-Policy Distillation: ORPO-Distill (Singh et al., 29 Sep 2025) contrasts positive traces from a fixed teacher with negative traces from the student, employing a mixed-policy strategy for sampling student-generated outputs, further blurring the boundaries between student and teacher.

2. Optimization Objectives and Loss Functions

Distinct dual-policy frameworks share a core structural reliance on contrasting, aligning, or otherwise relating the probability distributions of two policies. Three principal classes of loss emerge:

• Weighted Regression or Divergence: DPD minimizes

$$\mathcal J^w_{\theta} = \mathbb E_{s\sim B_{\widetilde\pi_{\phi}}} [ D(\pi_{\theta}(\cdot|s),\widetilde\pi_\phi(\cdot|s)) \cdot w_\phi(s) ]$$

with state-dependent weights $w_\phi(s)$ increasing emphasis on states where the peer outperforms (Lai et al., 2020).

• KL and Jensen-Shannon Divergences: HDPO employs token-level Jensen–Shannon divergence between student and “privileged” teacher policies for distillation on cliff prompts:

$$L_{\text{distill}}(\theta) = \frac{1}{N_{\text{tok}}} \sum_{(x,\bar y) \in T} \sum_{t=1}^{|\bar y|} JSD_k \left( \pi_t(\cdot | \bar y_{<t}) \| \pi_s(\cdot | \bar y_{<t}) \right)$$

with $T$ the set of teacher-generated, reward-positive trajectories (Ding, 25 Mar 2026).

• Odds-Ratio or Preference-based Losses: ORPO-Distill defines the preference optimization objective as a log odds ratio,

$$L_{\text{OR}}(\theta) = \mathbb{E}_{(\tau_T,\tau_S) \sim D} \left[ -\log \sigma \left( \log \frac{ \pi_T(\tau_T) }{ \pi_S(\tau_S) } \right) \right]$$

contrasting teacher and student trajectory likelihoods (Singh et al., 29 Sep 2025).

• Dual-Teacher Adversarial KLs: SDM Policy places the student G_θ in adversarial interplay with a learned teacher D_θ regularized towards a frozen teacher P_θ: $$\min_{G_\theta}\max_{D_\theta} \left\{ D_{\text{KL}}(p_{D_\theta}\|p_{P_\theta}) - \gamma \mathbb{E}_z \|D_\theta(z)-G_\theta(z)\|^2 \right\}$$ enforcing match both to the precise reference and flexible adaptation (Jia et al., 2024).

A common thread is the use of either explicit or implicit weighting to prioritize distillation where one policy exhibits superior returns or better statistical action alignment.

3. Theoretical Properties and Guarantees

Dual policy distillation methods provide theoretical guarantees under simplifying assumptions:

• Policy Improvement & Value Iteration Analogy: In DPD (Lai et al., 2020), the hypothetical “hybrid” policy $\pi^{\rm hypo}(a|s)$, switching between the better of two peers at each state, is shown to dominate both in value under standard RL assumptions. Disadvantageous distillation, performing updates only where the peer’s estimated value is superior, mimics the max-operator of value iteration.
• Tight Realizability Gap: HDPO (Ding, 25 Mar 2026) proves that when teacher and student share parameters and differ only in privileged context, the KL divergence between their distributions is bounded above in terms of the contextual Lipschitz constant and input perturbation. Absence of a model-mismatch term ensures a narrower gap than for cross-model distillation.
• Recovery of KL-Regularized Optima: For binary rewards, HDPO shows that restricting distillation targets to reward-positive, privileged trajectories recovers the optimal KL-regularized policy in the $\beta\to 0^+$ regime of relative entropy regularization.
• Stable Preference-based Learning: In ORPO-Distill (Singh et al., 29 Sep 2025), the log-odds ratio objective ensures gradients remain meaningful even as the probability ratios between positive and negative traces are small, promoting effective optimization across the trajectory space.

These properties explain why dual-policy approaches can outperform naive student-teacher or pure RL alone, as updating selectively or with more tailored feedback increases both learning signal and convergence rate.

4. Algorithmic Implementations and Pseudocode

Dual policy distillation is operationalized via algorithms that alternate between environment interaction, RL updates, and peer-based/specialized distillation steps.

The high-level sequence in DPD (Lai et al., 2020) is:

1. Each policy (πθ, πϕ) collects trajectories in parallel, builds separate buffers.
2. Each updates its own RL loss (e.g., PPO or DDPG).
3. Each samples from the peer’s buffer and updates on a weighted loss toward the peer’s action at sampled states, with weights based on peer advantage.
4. Iteration continues for M steps.

HDPO (Ding, 25 Mar 2026) introduces additional conditioning for “cliff detection,” using privileged rollouts for distillation on batches where all student outputs fail. ORPO-Distill (Singh et al., 29 Sep 2025) samples “negative” traces for preference optimization using an adjustable mixture of the student’s historical and current checkpoints.

SDM Policy (Jia et al., 2024) utilizes a dual-corrector mechanism: a frozen teacher ensures stability, while a learned “adversarial” teacher tracks the student generator; joint minimax objectives align one-step action distributions with the original diffusion teacher.

5. Empirical Performance and Impact

Benchmarks demonstrate the concrete impact of dual policy distillation:

Setting	Baseline	Dual Policy Distillation	Improvement
DDPG (Swimmer-v2)	30±2	36±3 (Lai et al., 2020)	+20%
PPO (HalfCheetah)	2947±201	3051±191 (Lai et al., 2020)	+3.5%
LLM RL (pass@4 OpenMathInstruct-2)	0.7749	0.7861 (Ding, 25 Mar 2026)	+1.1%
TinyLlama-1.1B (QA accuracy)	37.6% (CoT)	43.2% (mixed-policy ORPO) (Singh et al., 29 Sep 2025)	~+15%
Visuomotor (success rate)	72.6% (DP3)	74.8% (SDM Policy) (Jia et al., 2024)	+2.2% + 6x speedup

In all cases, the dual distillation scheme either improves return, coverage, or sample efficiency, and in SDM Policy, enables orders-of-magnitude faster inference with negligible loss (and sometimes improvement) in controller quality.

6. Extensions, Variants, and Current Research

Dual policy distillation has inspired variants and research into:

• Ensembles and Multi-Peer Distillation: Potential extension to more than two parallel policies, including heterogeneous architectures or curricula (Lai et al., 2020).
• Adaptive Weighting and Scheduling: Tuning confidence or loss weights is necessary to balance exploration (diverse learning) and exploitation (fine-tuned accuracy), as demonstrated by the λ parameter in HDPO (Ding, 25 Mar 2026).
• Input Privileging and Self-Distillation: Recasting the duality in input context (privileged vs. standard) rather than parameterization, as in HDPO, and in dual-teacher corrective modules (Jia et al., 2024).
• Distributional Matching Beyond Exact Teacher Cloning: Framing knowledge transfer as matching score functions, outcome distributions, or preferences among generated traces, rather than strict output probability alignment (Jia et al., 2024, Singh et al., 29 Sep 2025).

A plausible implication is that dual policy distillation principles can generalize beyond standard policy-distillation, enabling robust, sample-efficient learning under challenging reward structures and in settings without access to perfect or static teachers.

7. Limitations and Open Problems

Dual policy distillation methods exhibit several limitations:

• Hyperparameter Sensitivity: Performance can depend on weighting parameters (e.g., α, λ), requiring tuning per environment or architecture (Lai et al., 2020, Ding, 25 Mar 2026).
• Scalability to Discrete, Partial, or Multi-Agent Domains: Most empirical evaluations use continuous control or RL scenarios; transferability to discrete or multi-agent settings remains underexplored.
• Interpretability of Peer Gains: While peer distillation can empirically outperform classical methods, interpreting which knowledge is uniquely contributed by each peer—and how to maximize additive benefit—remains an open challenge.
• Extension to Arbitrary Architectures: While frameworks such as ORPO-Distill handle cross-architecture transfer (Singh et al., 29 Sep 2025), extending dual distillation to truly heterogeneous policies may require new objectives to address the realizability gap.

Ongoing research is directed toward addressing these challenges, informed by new algorithmic formulations and application domains.