Self-Supervised Active Domain Randomization

• SS-ADR is a reinforcement learning methodology that combines self-play with active domain randomization via SVPG sampling to create co-adaptive curricula.
• It jointly evolves environment parameters and task goals, aligning task difficulty with the agent’s learning frontier for improved performance.
• Empirical results show SS-ADR achieves robust sim-to-real transfer and lower variance compared to traditional uniform randomization methods.

Self-Supervised Active Domain Randomization (SS-ADR) is a methodology for generating automatic curricula in goal-directed reinforcement learning (RL) by coupling self-play with active domain randomization. SS-ADR simultaneously evolves both the environment parameters and task goals, resulting in a co-adaptive curriculum that directly targets the agent’s current learning frontier. By integrating environment domain parameters into the self-play framework using stochastic value particle gradient (SVPG) sampling, SS-ADR addresses the limitations of traditional self-play and uniform domain randomization in sim-to-real transfer and in training agents on progressively harder tasks (Raparthy et al., 2020).

1. Formal Framework and Problem Setting

SS-ADR operates within a family of goal-conditioned Markov Decision Processes (MDPs), each indexed by an environment parameter vector $\theta\in \Theta \subset \mathbb R^{N_{\rm rand}}$. The MDP is defined as $$\mathcal M_\theta = (\mathcal S, \mathcal A, \mathcal T_\theta, \mathcal R, \gamma)$$, where:

• $\mathcal S$: State space (e.g., robot joint states and object positions).
• $\mathcal A$: Continuous action space (e.g., actuation torques).
• $\mathcal T_\theta$: Transition kernel, with $\theta$ parameterizing simulator dynamics (e.g., friction, mass).
• $\mathcal G$: Goal space, typically a subset of states constituting the desired task objective.
• $\mathcal R(s, g, a)$: Reward function, commonly $-\|\mathrm{dist}(s, g)\|$ or a binary success indicator.
• $\gamma$: Discount factor.

An episode consists of two phases. In the first, an agent ("Alice") proposes a goal $$\mathcal M_\theta = (\mathcal S, \mathcal A, \mathcal T_\theta, \mathcal R, \gamma)$$0 in a reference environment ($$\mathcal M_\theta = (\mathcal S, \mathcal A, \mathcal T_\theta, \mathcal R, \gamma)$$1). In the second, another agent ("Bob") attempts this goal in a randomized environment $$\mathcal M_\theta = (\mathcal S, \mathcal A, \mathcal T_\theta, \mathcal R, \gamma)$$2, where $$\mathcal M_\theta = (\mathcal S, \mathcal A, \mathcal T_\theta, \mathcal R, \gamma)$$3 is an SVPG-sampled distribution over $$\mathcal M_\theta = (\mathcal S, \mathcal A, \mathcal T_\theta, \mathcal R, \gamma)$$4. The SVPG particle system $$\mathcal M_\theta = (\mathcal S, \mathcal A, \mathcal T_\theta, \mathcal R, \gamma)$$5 with parameters $$\mathcal M_\theta = (\mathcal S, \mathcal A, \mathcal T_\theta, \mathcal R, \gamma)$$6 constitutes the active domain randomization module, adapting $$\mathcal M_\theta = (\mathcal S, \mathcal A, \mathcal T_\theta, \mathcal R, \gamma)$$7 to concentrate on maximally informative environment parameters.

2. Self-Play and Joint Curriculum Evolution

SS-ADR extends classical asymmetric self-play by intertwining the selection of both environment parameters and goals. The self-play loop comprises three principal policies:

• Alice’s acting policy, $$\mathcal M_\theta = (\mathcal S, \mathcal A, \mathcal T_\theta, \mathcal R, \gamma)$$8, proposes a sequence of actions in $$\mathcal M_\theta = (\mathcal S, \mathcal A, \mathcal T_\theta, \mathcal R, \gamma)$$9 until issuing a STOP.
• Alice’s stopping policy, $\mathcal S$0, determines when her current state becomes the next goal ($\mathcal S$1).
• Bob’s acting policy, $\mathcal S$2, executes goal-reaching behavior in a sampled environment $\mathcal S$3.

For each episode:

1. Alice acts in $\mathcal S$4 and stops to set $\mathcal S$5 at $\mathcal S$6.
2. Environment samples are drawn from $\mathcal S$7; $\mathcal S$8 is instantiated at $\mathcal S$9.
3. Bob is tasked to reach $\mathcal A$0 in $\mathcal A$1, and the time taken $\mathcal A$2 is measured.
4. Rewards are assigned as follows:
   • Alice’s curriculum reward: $\mathcal A$3, incentivizing her to create goals just beyond Bob’s capabilities.
   • Bob’s RL reward: $\mathcal A$4, the environment reward.

This architecture enables environment goals and dynamics to co-evolve, preventing degenerate solutions (e.g., insoluble tasks or trivial environments).

3. Optimization Objectives and Learning Dynamics

The SS-ADR optimization decomposes into several interacting objectives:

• Bob’s RL Losses: Standard DDPG objectives are used with Q-function $\mathcal A$5 and deterministic policy $\mathcal A$6. The critic loss is

$\mathcal A$7

and the actor loss is

$\mathcal A$8

• Alice’s Stop Policy: Treated as a two-action (STOP/don’t-stop) episodic task, optimized via REINFORCE with gradient

$\mathcal A$9

• SVPG-based ADR Sampler: Each SVPG particle $\mathcal T_\theta$0 updates via

$\mathcal T_\theta$1

where $\mathcal T_\theta$2 is an RBF kernel enforcing diversity, and $\mathcal T_\theta$3.

• Bilevel Optimization View:
  • Inner loop (goal-conditioned RL):

  $\mathcal T_\theta$4 - Outer loop (curricula/ADR):

  $\mathcal T_\theta$5

4. Implementation and Algorithmic Workflow

A succinct pseudocode representation:

$\theta$3

Inputs, episodic role assignment, reward flow, and parameter updates are synchronized between policies and the ADR sampler. Alice’s acting policy is periodically set as a delayed copy of Bob.

5. Empirical Evaluation and Comparative Results

Experiments focus on robotic control in simulated and physical domains:

• Benchmarks:

  • ErgoReacher: 4 DoF robotic arm with goal-reaching.
  • ErgoPusher: 3 DoF robotic arm pushing a puck to a goal.
• Domain Randomization:
  • Pusher: puck friction in $\mathcal T_\theta$6.
  • Reacher: 8 simulator dynamics parameters randomized.
• Baselines:
  • Uniform Domain Randomization (UDR): $\mathcal T_\theta$7.
  • Unsupervised-Default: self-play over goals only, $\mathcal T_\theta$8.
• Metrics:
  • Final distance to goal (lower is better).
  • Success rate on held-out "hard" environments.
  • Sample efficiency (performance versus simulator steps).
  • Variance across 4 seeds.

Simulation Findings:

Method	In-domain performance	Hard env. generalization	Variance
SS-ADR	Fast, stable	Robust convergence	Low
UDR	Unstable	Often fails	High
Unsupervised-Default	Stable	Poor generalization	High

• Self-play alone and SS-ADR perform similarly in-domain.
• Only SS-ADR reliably reaches low error in unseen hard environments.

Sim-to-Real Transfer:

• Transfer to real Poppy Ergo Jr. robots in three friction/torque regimes shows SS-ADR outperforms both baselines, with lower and more consistent final distances on target.
• Error reduction is often 20–50% over baselines on hardest real settings.

Self-Calibration Study:

• UDR can collapse if $\mathcal T_\theta$9 includes "impossible" or unsolvable $\theta$0 parameters.
• SS-ADR’s SVPG sampler avoids unsolvable extremes, concentrating on a viable "band" of environments.

6. Mechanistic Insights and Limitations

The coupling of goal and environmental curriculum in SS-ADR avoids degenerate scenarios where goals or environment variations become unlearnably hard. The single self-play reward signal $\theta$1 unifies goal and environment adaptation, aligning the difficulty levels adaptively.

Empirically, SS-ADR achieves lower performance variance than either ADR or goal-space self-play isolated, a property important for safety and reliability in zero-shot sim-to-real transfer.

Limitations include:

• Necessity for a realistic bounding range $\theta$2; excessively broad or unrealistic domains slow or stall progress.
• Reliance on a time-delayed copy of Bob for Alice’s acting policy may introduce goal proposal bias; improved synchronization or joint optimization could enhance performance.
• Extensions to tasks beyond reachability (e.g., deformable objects, multi-object manipulation) require new definitions for the goal-stopping criterion and possibly for the reward structure.

These factors underscore the importance of ongoing calibration, careful domain space specification, and further architectural innovation for broader task classes (Raparthy et al., 2020).