Cyclic Policy Distillation for Sim-to-Real RL

• Cyclic Policy Distillation is a reinforcement learning method that partitions wide-ranging domain randomization into overlapping sub-domains for targeted policy training.
• It sequentially trains local policies in a cyclic manner, transferring knowledge between adjacent sub-domains to ensure stable updates and faster convergence.
• By distilling local insights into a unified global policy, CPD achieves significant sample efficiency gains and robust zero-shot sim-to-real performance in diverse tasks.

Cyclic Policy Distillation (CPD) is a sample-efficient reinforcement learning (RL) methodology designed for robust sim-to-real policy transfer under wide domain randomization (DR). CPD addresses the challenges arising from instability and sample inefficiency inherent in standard DR-based training when the randomized parameter range is extensive. By systematically partitioning the parameter space, cycling training across local policies, and leveraging targeted distillation, CPD accelerates convergence and enhances zero-shot deployment performance, both in simulation and on real-robot tasks (2207.14561).

1. Motivation and Conceptual Framework

Zero-shot sim-to-real transfer aims to train a single policy in a simulated environment across a broad range of randomized physical and sensor parameters, ensuring immediate real-world effectiveness without further adaptation. Vanilla DR approaches expose a single agent to the entire variability of the environment but suffer from high sample complexity and unstable policy updates when the DR parameter space is large. CPD mitigates these issues by:

• Partitioning the DR parameter range into a sequence of narrower, overlapping sub-domains.
• Allocating a distinct local policy–value pair, $\{\pi^{(n)}, Q^{(n)}\}$, to each sub-domain.
• Training each local agent in a cyclical sequence, beneficially transferring knowledge only from adjacent sub-domains.
• Distilling the ensemble of specialized local policies into a unified global policy $\pi^{(g)}$ for robust zero-shot transfer.

This cycle ensures knowledge is shared between closely related domains, maintaining learning stability while promoting sample efficiency.

2. Partitioning the Randomized Parameter Space

Consider the full randomized domain parameter space,

$$\Theta = \{\xi \in \mathbb{R}^d : \xi_i \in [\ell_i, u_i] \ \forall i = 1, \ldots, d\},$$

where each $\xi_i$ encodes a physical or sensor parameter, such as gravity, mass, friction, or sensor bias. CPD selects one or more target parameters $\xi_j$ and splits their ranges into $N$ overlapping intervals, yielding sub-domains:

$$\Theta = \bigcup_{n=1}^{N} \Theta_n, \hspace{1cm} \Theta_n = \{\xi : \xi_j \in [\ell_j^n, u_j^n], \ \xi_{-j} \in [\ell_{-j}, u_{-j}]\}.$$

Here, $\xi_{-j}$ denotes all non-split parameters, which remain fully randomized. In practice, hyperparameter $N$ is selected based on cross-validation to balance sample efficiency and final policy performance (e.g., $N=4$ for Pendulum, $\pi^{(g)}$0 for MuJoCo domains). Overlap and splitting structure can follow planes, grids, or edges, provided adjacent sub-domains remain proximal in parameter space, thereby facilitating stable knowledge transfer.

3. The CPD Algorithm: Local Policy Training and Cyclic Transitions

Each sub-domain $\pi^{(g)}$1 is assigned a policy–value network pair, $\pi^{(g)}$2, both randomly initialized. Training proceeds by alternating forward and backward cycles across the sub-domains ($\pi^{(g)}$3), with the following key mechanisms:

• On entering a new sub-domain, $\pi^{(g)}$4 is initialized from $\pi^{(g)}$5 (forward) or $\pi^{(g)}$6 (backward) to exploit policy continuity.
• In each cycle, for a fixed number of episodes and time steps, transitions $\pi^{(g)}$7 are collected under policy $\pi^{(g)}$8 within $\pi^{(g)}$9.
• The critic $$\Theta = \{\xi \in \mathbb{R}^d : \xi_i \in [\ell_i, u_i] \ \forall i = 1, \ldots, d\},$$0 is updated using the standard RL loss (e.g., Soft Actor-Critic, SAC, critic loss).
• A monotonic policy-improvement distillation term is incorporated into the actor update:

$$\Theta = \{\xi \in \mathbb{R}^d : \xi_i \in [\ell_i, u_i] \ \forall i = 1, \ldots, d\},$$1

where $$\Theta = \{\xi \in \mathbb{R}^d : \xi_i \in [\ell_i, u_i] \ \forall i = 1, \ldots, d\},$$2 indexes the adjacent sub-domain. The policy update objective is

$$\Theta = \{\xi \in \mathbb{R}^d : \xi_i \in [\ell_i, u_i] \ \forall i = 1, \ldots, d\},$$3

ensuring that knowledge transfer is guided by expected performance improvement, analogous to Conservative Policy Iteration (CPI). Cycling proceeds until convergence across all local policies.

4. Global Policy Distillation

Upon local policy convergence, a global policy $$\Theta = \{\xi \in \mathbb{R}^d : \xi_i \in [\ell_i, u_i] \ \forall i = 1, \ldots, d\},$$4 is learned via one-shot distillation. The parameter $$\Theta = \{\xi \in \mathbb{R}^d : \xi_i \in [\ell_i, u_i] \ \forall i = 1, \ldots, d\},$$5 is optimized to minimize the average KL divergence between $$\Theta = \{\xi \in \mathbb{R}^d : \xi_i \in [\ell_i, u_i] \ \forall i = 1, \ldots, d\},$$6 and each $$\Theta = \{\xi \in \mathbb{R}^d : \xi_i \in [\ell_i, u_i] \ \forall i = 1, \ldots, d\},$$7 on rollouts aggregated across all sub-domains:

$$\Theta = \{\xi \in \mathbb{R}^d : \xi_i \in [\ell_i, u_i] \ \forall i = 1, \ldots, d\},$$8

where $$\Theta = \{\xi \in \mathbb{R}^d : \xi_i \in [\ell_i, u_i] \ \forall i = 1, \ldots, d\},$$9 is a replay buffer containing transitions from all sub-domains. This global distillation aggregates specialized skills into a singular policy for real-world deployment.

5. RL Backbone, Architecture, and Implementation

CPD is built atop the Soft Actor-Critic (SAC) off-policy actor–critic methodology. In each sub-domain:

• The critic $\xi_i$0 is updated using mean-squared Bellman residual.
• The actor $\xi_i$1 employs a diagonal-Gaussian output $\xi_i$2, trained by the SAC policy gradient, augmented by the CPD distillation term.
• Both actor and critic networks include an LSTM (64 units) to encode state–action histories, thereby inferring hidden environment parameters.

Key hyperparameters, selected for training stability and efficiency, include $\xi_i$3 (Pendulum) or $\xi_i$4 (MuJoCo tasks), monotonic mix coefficient $\xi_i$5, discount factor $\xi_i$6, episodes per sub-domain shift $\xi_i$7, steps per episode $\xi_i$8, and SAC learning rates of $\xi_i$9. For real-robot experiments, hidden layer sizes are increased to 128 units.

Application	Sub-domain Count ($\xi_j$0)	LSTM Size	Episodes/Shift	Steps/Episode
Pendulum (Gym)	4	64	15	150
Mujoco (Pusher, Swimmer, HC)	6	64	15	150
UR3 Ball-Dispersal	6	128	15	150

6. Empirical Performance and Evaluation

CPD demonstrates substantial improvements in sample efficiency and zero-shot sim-to-real transfer. In four benchmark tasks—OpenAI Gym Pendulum and MuJoCo Pusher, Swimmer, and HalfCheetah—CPD outperforms SAC-DR, Peer-to-Peer DRL, Distilled DR, Divide-and-Conquer, and Active DR. Notable results include:

• Pendulum: CPD achieves a total reward of 190 with approximately 18,000 samples, compared to $\xi_j$1100,000 samples for the next-best approach, representing a %%%%42$$\Theta = \bigcup_{n=1}^{N} \Theta_n, \hspace{1cm} \Theta_n = \{\xi : \xi_j \in [\ell_j^n, u_j^n], \ \xi_{-j} \in [\ell_{-j}, u_{-j}]\}.$$43%%%% efficiency gain.
• HalfCheetah: Only CPD manages to secure a positive reward within 150,000 steps; all baselines fail to do so.
• Ablation experiments confirm that either removing the monotonic neighbor-distillation ($\xi_j$4) or randomizing the order of cycling degrades both convergence speed and final policy performance.

In real-robot ball-dispersal, using a UR3 arm, CPD's global policy achieves 13/14 (juggling) and 11/14 (beads) upper ball teardown success in zero-shot tests, compared to SAC-DR's 6/14 and 1/14, respectively, and 0/14 for SAC without DR.

7. Domain Randomization Scope and Task Environments

CPD's effectiveness is demonstrated across both simulation and real-robot settings with wide-ranging DR:

• Pendulum: gravity [0.7, 1.5], timestep [0.8, 1.2], mass/length [0.8, 1.2], actuator gain/bias.
• MuJoCo suite: gravity, timestep, friction, mass, actuator gain/damping in [0.5, 2].
• UR3 Ball-Dispersal: gravity [9, 11], friction [0.6, 1.0], ball mass [5, 20g], ball radius [25, 30mm], sensor bias $\xi_j$510mm, Z-scale [0.7, 1.3]. Observations include a compressed depth image (5×9) and end-effector position.

CPD robustly adapts to both continuous control and vision-based robotic manipulation.

---

For code, pretrained models, and experimental artifacts, see https://github.com/yuki-kadokawa/cyclic-policy-distillation. The CPD framework provides a principled strategy to trade increased bookkeeping complexity for significant sample savings, stable learning, and enhanced zero-shot sim-to-real RL performance under extensive domain randomization (2207.14561).