---
title: Self-Supervised Active Domain Randomization
url: https://www.emergentmind.com/topics/self-supervised-active-domain-randomization-ss-adr
type: topic
---

# Self-Supervised Active Domain Randomization

Self-Supervised Active Domain Randomization (SS-ADR) is a methodology for generating automatic curricula in goal-directed reinforcement learning (RL) by coupling self-play with active domain randomization. SS-ADR simultaneously evolves both the environment parameters and task goals, resulting in a co-adaptive curriculum that directly targets the agent’s current learning frontier. By integrating environment domain parameters into the self-play framework using stochastic value particle gradient (SVPG) sampling, SS-ADR addresses the limitations of traditional self-play and uniform domain randomization in sim-to-real transfer and in training agents on progressively harder tasks [2002.07911].

## 1. Formal Framework and Problem Setting

SS-ADR operates within a family of goal-conditioned Markov Decision Processes (MDPs), each indexed by an environment parameter vector $\theta\in \Theta \subset \mathbb R^{N_{\rm rand}}$. The MDP is defined as $\mathcal M_\theta = (\mathcal S, \mathcal A, \mathcal T_\theta, \mathcal R, \gamma)$, where:

- $\mathcal S$: State space (e.g., robot joint states and object positions).
- $\mathcal A$: Continuous action space (e.g., actuation torques).
- $\mathcal T_\theta$: Transition kernel, with $\theta$ parameterizing simulator dynamics (e.g., friction, mass).
- $\mathcal G$: Goal space, typically a subset of states constituting the desired task objective.
- $\mathcal R(s, g, a)$: Reward function, commonly $-\|\mathrm{dist}(s, g)\|$ or a binary success indicator.
- $\gamma$: Discount factor.

An episode consists of two phases. In the first, an agent ("Alice") proposes a goal $g$ in a reference environment ($\theta_{\rm ref}$). In the second, another agent ("Bob") attempts this goal in a randomized environment $\theta \sim \mu_\phi$, where $\mu_\phi$ is an SVPG-sampled distribution over $\Theta$. The SVPG particle system $\{\xi_i\}$ with parameters $\phi_i$ constitutes the active domain randomization module, adapting $\mu_\phi$ to concentrate on maximally informative environment parameters.

## 2. Self-Play and Joint Curriculum Evolution

SS-ADR extends classical asymmetric self-play by intertwining the selection of both environment parameters and goals. The self-play loop comprises three principal policies:

- **Alice’s acting policy**, $\pi_a(a \mid s_0, s_t; E_{\rm ref})$, proposes a sequence of actions in $E_{\rm ref}$ until issuing a STOP.
- **Alice’s stopping policy**, $\pi_a^s(\mathrm{STOP} \mid s_0, s_t; E_{\rm ref})$, determines when her current state becomes the next goal ($g$).
- **Bob’s acting policy**, $\pi_b(a \mid s_t, g; E_{\rm rand})$, executes goal-reaching behavior in a sampled environment $E_{\rm rand} = S(\theta)$.

For each episode:

1. Alice acts in $E_{\rm ref}$ and stops to set $g$ at $s_{t_a}$.
2. Environment samples are drawn from $\mu_\phi$; $E_{\rm rand}$ is instantiated at $\theta$.
3. Bob is tasked to reach $g$ in $E_{\rm rand}$, and the time taken $t_b$ is measured.
4. Rewards are assigned as follows:
   - Alice’s curriculum reward: $r_a = \nu \max(0, t_b - t_a)$, incentivizing her to create goals just beyond Bob’s capabilities.
   - Bob’s RL reward: $r_b = \mathcal R(s, g, a)$, the environment reward.

This architecture enables environment goals and dynamics to co-evolve, preventing degenerate solutions (e.g., insoluble tasks or trivial environments).

## 3. Optimization Objectives and Learning Dynamics

The SS-ADR optimization decomposes into several interacting objectives:

- **Bob’s RL Losses**: Standard DDPG objectives are used with Q-function $Q(s, a \mid g, \theta)$ and deterministic policy $\pi_b(s \mid g, \theta)$. The critic loss is

  $$
  \mathcal L_Q(\psi) = \mathbb E_{(s, a, r, s')} \left[ Q_\psi(s, a \mid g, \theta) - (r + \gamma Q_{\bar\psi}(s', \pi_b(s'\mid g, \theta) \mid g, \theta)) \right]^2
  $$

  and the actor loss is

  $$
  \mathcal L_{\pi_b} = -\, \mathbb E_s \left[ Q_\psi(s, \pi_b(s \mid g, \theta) \mid g, \theta) \right].
  $$

- **Alice’s Stop Policy**: Treated as a two-action (STOP/don’t-stop) episodic task, optimized via REINFORCE with gradient

  $$
  \nabla_{\phi_s} J(\phi_s) = \mathbb E \left[ r_a \sum_{t=0}^{t_a-1} \nabla_{\phi_s} \log \pi_a^s(d_t) \right]
  $$

- **SVPG-based ADR Sampler**: Each SVPG particle $\xi_i$ updates via

  $$
  \Delta \xi_i = \frac{\epsilon}{N} \sum_{j=1}^N \left[ \nabla_{\xi_j} J(\xi_j) k(\xi_i, \xi_j) + \alpha \nabla_{\xi_j} k(\xi_i, \xi_j) \right]
  $$

  where $k(\cdot, \cdot)$ is an RBF kernel enforcing diversity, and $J(\xi_j) \approx r_a^{(j)}$.

- **Bilevel Optimization View**:

  - *Inner loop (goal-conditioned RL)*:
    $$
    \max_{\pi_b, \pi_a} \mathbb E_{\theta \sim \mu_\phi,\,g \sim \pi_a} \left[\sum_t \mathcal R(s_t, g, a_t)\right]
    $$
  - *Outer loop (curricula/ADR)*:
    $$
    \max_{\pi_a^s, \mu_\phi} \mathbb E_{g \sim \pi_a,\, \theta \sim \mu_\phi} \left[ r_a(g, \theta) \right]
    $$

## 4. Implementation and Algorithmic Workflow

A succinct pseudocode representation:

```
Input:
  - Randomization space Ξ, simulator S: Ξ → Envs.
  - Reference parameter θ_ref.
  - Init π_a, π_a^s, π_b, SVPG particles {ξ_i}.

for iteration = 1 to MaxSteps:
    1. π_a ← (old copy of) π_b
    2. Build E_ref ← S(θ_ref); observe s_0.
    3. Alice acts under π_a, π_a^s in E_ref (STOP at t_a → goal g).
    4. Sample θ ~ {ξ_i}; build E_rand = S(θ).
    5. Bob executes π_b in E_rand (record t_b to reach g).
    6. Compute r_a = ν · max(0, t_b - t_a).
    7. Update π_a^s using policy gradient on r_a.
    8. Update {ξ_i} using SVPG step on r_a.
    9. Collect standard RL rollouts and update π_a, π_b via DDPG.
end for
```

*Inputs, episodic role assignment, reward flow, and parameter updates are synchronized between policies and the ADR sampler. Alice’s acting policy is periodically set as a delayed copy of Bob.*

## 5. Empirical Evaluation and Comparative Results

Experiments focus on robotic control in simulated and physical domains:

- **Benchmarks**:
  - ErgoReacher: 4 DoF robotic arm with goal-reaching.
  - ErgoPusher: 3 DoF robotic arm pushing a puck to a goal.

- **Domain Randomization**:
  - Pusher: puck friction in $[0.01, 1.0]$.
  - Reacher: 8 simulator dynamics parameters randomized.

- **Baselines**:
  - Uniform Domain Randomization (UDR): $\theta \sim \mathrm{Uniform}(\Xi)$.
  - Unsupervised-Default: self-play over goals only, $\theta = \theta_{\rm ref}$.

- **Metrics**:
  - Final distance to goal (lower is better).
  - Success rate on held-out "hard" environments.
  - Sample efficiency (performance versus simulator steps).
  - Variance across 4 seeds.

**Simulation Findings**:

| Method                 | In-domain performance | Hard env. generalization | Variance    |
|------------------------|----------------------|--------------------------|-------------|
| SS-ADR                 | Fast, stable         | Robust convergence       | Low         |
| UDR                    | Unstable             | Often fails              | High        |
| Unsupervised-Default   | Stable               | Poor generalization      | High        |

- Self-play alone and SS-ADR perform similarly in-domain.
- Only SS-ADR reliably reaches low error in unseen hard environments.

**Sim-to-Real Transfer**:

- Transfer to real Poppy Ergo Jr. robots in three friction/torque regimes shows SS-ADR outperforms both baselines, with lower and more consistent final distances on target.
- Error reduction is often 20–50% over baselines on hardest real settings.

**Self-Calibration Study**:

- UDR can collapse if $\Xi$ includes "impossible" or unsolvable $\theta$ parameters.
- SS-ADR’s SVPG sampler avoids unsolvable extremes, concentrating on a viable "band" of environments.

## 6. Mechanistic Insights and Limitations

The coupling of goal and environmental curriculum in SS-ADR avoids degenerate scenarios where goals or environment variations become unlearnably hard. The single self-play reward signal $r_a$ unifies goal and environment adaptation, aligning the difficulty levels adaptively.

Empirically, SS-ADR achieves lower performance variance than either ADR or goal-space self-play isolated, a property important for safety and reliability in zero-shot sim-to-real transfer.

**Limitations include**:

- Necessity for a realistic bounding range $\Xi$; excessively broad or unrealistic domains slow or stall progress.
- Reliance on a time-delayed copy of Bob for Alice’s acting policy may introduce goal proposal bias; improved synchronization or joint optimization could enhance performance.
- Extensions to tasks beyond reachability (e.g., deformable objects, multi-object manipulation) require new definitions for the goal-stopping criterion and possibly for the reward structure.

*These factors underscore the importance of ongoing calibration, careful domain space specification, and further architectural innovation for broader task classes* [2002.07911].

Source: https://www.emergentmind.com/topics/self-supervised-active-domain-randomization-ss-adr