Projected Universal Policy (PUP)

• Projected Universal Policy (PUP) is a sim-to-real transfer framework that leverages two-stage system identification and a low-dimensional policy embedding to adapt control policies for dynamic robotic systems.
• The framework integrates model parameter randomization, latent space projection, and Bayesian optimization to effectively bridge the gap between simulation and real hardware performance.
• Experimental evaluations on the Darwin OP2 robot demonstrate that PUP outperforms nominal and robust baselines, achieving robust multi-gait biped locomotion with minimal real-world trials.

The Projected Universal Policy (PUP) is an algorithmic framework for sim-to-real transfer in dynamic robotic control, designed to address domain discrepancy between simulators and real hardware by leveraging two-stage system identification and a low-dimensional task-adaptive policy embedding. Developed in the context of bipedal locomotion for the Darwin OP2 robot, PUP integrates model parameter randomization, projection into a latent space, and Bayesian optimization to enable efficient policy adaptation and robust real-world deployment (Yu et al., 2019).

1. Mathematical Formulation

Let $\mu \in \mathbb{R}^n$ denote the vector of simulation model parameters, including friction, center-of-mass (COM), and actuator/PD control gains. The main components of PUP are:

• Parameter Embedding: A projection network $f_\phi: \mathbb{R}^n \rightarrow \mathbb{R}^d$ maps $\mu$ to a low-dimensional latent variable $\eta$ ($d \ll n$, with $d=3$ in experiments).
• Policy Network: $\pi_\theta(a|s, \eta)$ outputs control actions $a \in \mathbb{R}^m$ given system state $s$ and embedding $\eta$.
• Uniform Sampling: Parameters are sampled from $f_\phi: \mathbb{R}^n \rightarrow \mathbb{R}^d$0, the range identified in pre-sysID.
• Training Objective: The expected simulated return,

$f_\phi: \mathbb{R}^n \rightarrow \mathbb{R}^d$1

is maximized using PPO with the clipped surrogate loss $f_\phi: \mathbb{R}^n \rightarrow \mathbb{R}^d$2 over the joint network $f_\phi: \mathbb{R}^n \rightarrow \mathbb{R}^d$3.

2. Two-Stage System Identification

2.1 Pre-sysID (Generic Data Collection)

Pre-sysID is performed by collecting hardware trajectories $f_\phi: \mathbb{R}^n \rightarrow \mathbb{R}^d$4 using generic joint exercises and standing/falling sequences. The loss function for parameter fitting is:

$f_\phi: \mathbb{R}^n \rightarrow \mathbb{R}^d$5

where $f_\phi: \mathbb{R}^n \rightarrow \mathbb{R}^d$6 are simulated joint positions and torso orientations, $f_\phi: \mathbb{R}^n \rightarrow \mathbb{R}^d$7 is the standing+falling subset, and $f_\phi: \mathbb{R}^n \rightarrow \mathbb{R}^d$8 is the standing subset.

• Nominal parameters $f_\phi: \mathbb{R}^n \rightarrow \mathbb{R}^d$9 are found via

$\mu$0

• $\mu$1 subsets $\mu$2 are sampled; for each,

$\mu$3

• The parameter bounds are set per-dimension as:

$\mu$4

CMA-ES is utilized to minimize $\mu$5, and bounds $\mu$6 are used to characterize simulator uncertainty for domain randomization.

2.2 Post-sysID (Task-Relevant Optimization)

Following PUP training, the latent embedding $\mu$7 is optimized for real-hardware performance by maximizing the real-world return $\mu$8, typically distance walked or time to fall, by running $\mu$9 directly on hardware.

• A Gaussian process surrogate for $\eta$0 is initialized with $\eta$1 uniformly sampled trials, then refined over $\eta$2 Bayesian optimization steps (e.g., Expected Improvement). Each hardware trial runs the controller for a full episode.
• The best $\eta$3 observed is selected for deployment.

The total post-sysID adaptation is completed within 25 hardware trials (≤ 15 min).

3. Neural Architectures and Training

3.1 Architecture Details

Component	Architecture	Dimensions
Projection $\eta$4	2-layer FC (64→32) with tanh, final linear to $\eta$5	$\eta$6
Policy $\eta$7	2-layer FC (256→256) with tanh, input $\eta$8, output 20-d adjustments	$\eta$9
State $d \ll n$0	Motor positions (20), torso Euler angles (3) $d \ll n$1 2 time-steps	$d \ll n$249
Action $d \ll n$3	Joint target adjustments, 11 bins per joint	$d \ll n$4

PPO hyperparameters used: learning rate $d \ll n$5, discount $d \ll n$6, GAE $d \ll n$7, batch size 64 trajectories, horizon $d \ll n$8 steps. The reward combines forward velocity ($d \ll n$9), torque regularization ($d=3$0), torque-velocity ($d=3$1), reference tracking ($d=3$2), and a survival bonus ($d=3$3).

Simulation randomization includes control frequency $d=3$4 Hz and sensor noise (position $d=3$5 rad, orientation bias $d=3$6 rad).

3.2 Pipeline Pseudocode

1. Pre-sysID:
   • Collect data $d=3$7.
   • Find $d=3$8.
   • For each of $d=3$9 subsets $\pi_\theta(a|s, \eta)$0, solve for $\pi_\theta(a|s, \eta)$1 with regularization.
   • Compute per-dimension bounds $\pi_\theta(a|s, \eta)$2.
2. PUP Training:
   • Initialize $\pi_\theta(a|s, \eta)$3.
   • Repeat: sample $\pi_\theta(a|s, \eta)$4, compute $\pi_\theta(a|s, \eta)$5, run $\pi_\theta(a|s, \eta)$6 in simulation, optimize $\pi_\theta(a|s, \eta)$7.
3. Post-sysID:
   • Initialize GP with $\pi_\theta(a|s, \eta)$8 random $\pi_\theta(a|s, \eta)$9.
   • For $a \in \mathbb{R}^m$0 iterations: propose new $a \in \mathbb{R}^m$1 via acquisition, run on hardware, update GP.
   • Deploy best $a \in \mathbb{R}^m$2.

4. Experimental Evaluation on Darwin OP2

4.1 Gait Results

After post-sysID, stable forward, backward, and sideways walking were achieved on the Darwin OP2. For a trip-wire at $a \in \mathbb{R}^m$3 m:

Gait	Mean Distance to Fall (m)	Elapsed Time (s)
Forward	$a \in \mathbb{R}^m$4	$a \in \mathbb{R}^m$5
Backward	$a \in \mathbb{R}^m$6	$a \in \mathbb{R}^m$7
Sideways	$a \in \mathbb{R}^m$8	$a \in \mathbb{R}^m$9

Walking speed was $s$0 m/s.

4.2 Baseline Comparison

Three approaches were compared:

Method	Distance Traveled (m) Mean $s$1 Std (5 trials)
Nominal	$s$2
Robust	$s$3
PUP+BO	$s$4

PUP with Bayesian optimization substantially outperformed both nominal and robust (domain randomization only) baselines.

4.3 Ablation: NN-PD Actuator Model

Pre-sysID loss $s$5 was compared for two actuator models:

• PD Only: Fixed $s$6.
• NN-PD: $s$7 predicted by a 1-layer NN with 5 units ($s$8).

NN-PD achieved approximately $s$9 lower $\eta$0 compared to PD only over 500 CMA-ES iterations. Hip position step responses and torso pitch were closer to real data for NN-PD.

4.4 Parameter Range Analysis

Some parameters (e.g., ankle damping $\eta$1) had tight pre-sysID bounds, indicating high confidence. Others (e.g., torque limit $\eta$2, COM$\eta$3) exhibited wide bounds, reflecting greater simulation–real mismatch. The low-dimensional latent $\eta$4 allows PUP to compensate for these ambiguities.

5. Key Principles and Significance

Projected Universal Policy enables robust sim-to-real transfer by decoupling the modeling and control challenges into:

• Broad system identification (pre-sysID) to span plausible simulation variations.
• Training a policy family sensitive to a low-dimensional projection of the high-dimensional model uncertainty.
• Efficient real-world specialization (post-sysID) via Bayesian optimization in the latent space.

A plausible implication is that PUP serves as a general template for sim-to-real adaptation wherever model uncertainty can be characterized and projected, potentially benefiting domains beyond biped locomotion.

6. Conclusions

Projected Universal Policy combines principled domain randomization, differentiable model-to-embedding projection, and sample-efficient hardware adaptation, yielding robust and adaptive controllers on real robotic systems. On low-cost hardware, PUP enabled successful multi-gait biped locomotion using fewer than 25 real-world trials and demonstrated substantially improved performance over nominal or robust baselines (Yu et al., 2019).