---
title: 'SuperMimic: Scalable Humanoid Motion Control'
url: https://www.emergentmind.com/topics/supermimic
type: topic
---

# SuperMimic: Scalable Humanoid Motion Control

SuperMimic is a framework for scaling humanoid whole-body motion control through a single differentiable world model of shared rigid-body dynamics. Rather than training motion-specific experts or optimizing a generalist exclusively with model-free reinforcement learning, it fits a differentiable dynamics model from transitions generated by a standard rigid-body simulator and uses short imagined rollouts with backpropagation-through-time (BPTT) to optimize a goal-conditioned tracking policy. The method is evaluated on large human-motion libraries, including 129,514 filtered benchmark motions, and transfers to a physical Unitree G1 across locomotion and acrobatic behaviors [2610.28480].

## 1. Motivation and problem formulation

Physics-based humanoid imitation commonly uses model-free reinforcement learning, particularly PPO, to track motion-capture references. A typical pipeline samples a reference motion, executes a policy in many simulator environments, estimates advantages from sampled trajectories, and updates the policy repeatedly. This approach does not explicitly exploit the fact that substantially different behaviors share the same humanoid body, contacts, actuators, and rigid-body dynamics.

Large motion libraries therefore create a computational scaling problem. Existing approaches address this problem either by training motion-specific experts and distilling them into a generalist, or by training one generalist directly with cluster-scale PPO. The paper uses SONIC as the closest published comparison; SONIC trains for approximately 168 hours on 128 NVIDIA L40 GPUs, equivalent to roughly 21,000 GPU-hours.

SuperMimic instead amortizes the shared dynamics across all motions. The simulator remains responsible for producing grounded transitions, while the learned world model supplies fast differentiable rollouts for policy optimization. The policy is represented as a deterministic mapping

$$
\mathbf{a}_t=\pi_\theta(\mathbf{o}_t,\mathbf{g}_t),
$$

where $\mathbf{o}_t$ is a proprioceptive observation and $\mathbf{g}_t$ is a motion or task goal.

The formulation is a goal-conditioned partially observable Markov decision process,

$$
\mathcal{M}=\langle \mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},\gamma\rangle.
$$

The simulator state $\mathbf{s}_t$ includes the physical state, while the observation, goal, action, reward, termination indicator, and randomized simulator context are denoted by $\mathbf{o}_t$, $\mathbf{g}_t$, $\mathbf{a}_t$, $r_{t+1}$, $d_t$, and $\boldsymbol{\xi}$, respectively. During simulator collection, action noise is added:

$$
\mathbf{a}^{\mathrm{exec}}_t=\mathbf{a}_t+\boldsymbol{\epsilon}_t.
$$

The noisy executed action is stored in the replay buffer, broadening the action distribution used to train the world model. The simulator return uses a survival-weighting variable,

$$
c_0=1,\qquad c_{t+1}=c_t(1-d_{t+1}),
$$

so that terminated trajectories contribute no further reward:

$$
J(\theta)=\mathbb{E}\left[\sum_{t\geq 0}\gamma^t c_t r_{t+1}\right].
$$

In imagined rollouts, binary termination is replaced by a predicted termination probability, making the survival weight differentiable.

## 2. Policy inputs, motion goals, and control representation

The target platform is a 29-degree-of-freedom humanoid. The proprioceptive observation contains 29 joint positions relative to the default pose, 29 joint velocities, 3-dimensional base angular velocity from an IMU, 3-dimensional projected gravity, and the previous 29-dimensional action. The policy receives the last 10 observations, enabling short-term temporal information despite partial observability.

Typical sensor-noise ranges are:

- Joint positions relative to the default pose: $\mathcal{U}[-0.01,0.01]$ radians.
- Joint velocities: $\mathcal{U}[-0.5,0.5]$ radians per second.
- Base angular velocity: $\mathcal{U}[-0.2,0.2]$ radians per second.
- Projected gravity: $\mathcal{U}[-0.05,0.05]$.

For motion tracking, the goal consists of $K=10$ future reference frames. A motion encoder maps this sequence into a 32-dimensional latent representation:

$$
\mathbf{z}=\operatorname{enc}_{\theta,\mathrm{motion}}(\mathbf{g}_t).
$$

A shared decoder combines the motion latent with the observation history:

$$
\mathbf{a}_t=\operatorname{dec}_\theta(\mathbf{z},\mathbf{o}_{t-9:t}).
$$

The action is a normalized joint setpoint converted to a physical setpoint by

$$
\mathbf{q}^{\mathrm{sp}}_t=\boldsymbol{\alpha}\odot\mathbf{a}_t+\mathbf{q}^0,
$$

where $\boldsymbol{\alpha}$ is an action scale, $\mathbf{q}^0$ is the default configuration, and $\odot$ denotes elementwise multiplication. Because the PD gains are deliberately low, the setpoint behaves approximately like a torque command rather than a stiff position command.

The motion-tracking reward follows BeyondMimic, with the self-collision penalty removed and the joint-limit penalty reduced. It includes position, orientation, linear-velocity, angular-velocity, root-position, and root-orientation terms for tracked bodies. Representative terms are exponential penalties on squared tracking errors, including

$$
r^{\mathrm{pos}}_t=
\exp\left(
-\frac{
\frac{1}{|\mathcal{T}|}
\sum_{b\in\mathcal{T}}
\|\mathbf{p}^{\mathrm{des}}_b-\mathbf{p}_b\|^2
}{0.3^2}
\right)
$$

and

$$
r^{\mathrm{ori}}_t=
\exp\left(
-\frac{
\frac{1}{|\mathcal{T}|}
\sum_{b\in\mathcal{T}}
\left\|
\log\left(R^{\mathrm{des}}_bR_b^\top\right)
\right\|^2
}{0.4^2}
\right).
$$

Regularization includes an action-smoothness term,

$$
r^{\mathrm{smooth}}_t=-0.1\|\mathbf{a}_t-\mathbf{a}_{t-1}\|^2,
$$

and a joint-limit penalty. The total reward is multiplied by the control period, $\Delta t=0.02$ seconds. Episodes terminate when the root deviates substantially from the reference, approximately when $|e_{p,z}|>0.25$ meters or $e_{\mathrm{tilt}}>0.8$.

## 3. Differentiable world-model architecture

SuperMimic contains three coupled components: a simulator that produces logged transitions, a learned differentiable world model, and a goal-conditioned tracking policy with a critic.

During data collection, the policy interacts with MuJoCo Warp. A transition contains the simulator state, executed action, reward, next state, noise-free observation, termination indicator, and randomized physical context. The replay buffer is fixed-size; new transitions replace old ones. The implementation uses 4096 simulator environments per GPU and 1000 steps per environment in the replay buffer. Action noise is sampled from

$$
\boldsymbol{\epsilon}_t\sim\mathcal{N}(0,0.1^2).
$$

Return-based reset sampling preferentially revisits difficult portions of the motion library. Physical randomization includes friction coefficients sampled from $\mathcal{U}[0.3,1.2]$, joint calibration offsets in $[-0.01,0.01]$ radians, and torso center-of-mass offsets of approximately 2.5–5 centimeters. The randomized context $\boldsymbol{\xi}$ is logged because a deterministic model cannot otherwise infer which randomized simulator instance produced a transition.

The world model predicts state transitions, rewards, observations, and termination probabilities:

$$
\hat{\mathbf{s}}_{t+1}
=f_\omega(\hat{\mathbf{s}}_t,\mathbf{a}_t,\boldsymbol{\xi}),
$$

$$
\hat r_{t+1}
=r_\omega(\hat{\mathbf{s}}_t,\mathbf{a}_t,\mathbf{g}_t),
$$

$$
\hat{\bar{\mathbf{o}}}_t
=g_\omega(\hat{\mathbf{s}}_t,\boldsymbol{\xi}),
$$

$$
\hat d_t
=d_\omega(\hat{\mathbf{s}}_t,\mathbf{g}_t).
$$

The principal dynamics network is a three-layer multilayer perceptron with hidden widths $(2048,2048,2048)$. Some observation and reward quantities can be computed analytically when the state representation contains sufficient information; these quantities need not require independent prediction heads.

The world model is trained with an autoregressive multi-step loss. A replay sample consists of a logged $k$-step window, with the default $k=8$. The model is initialized from the logged state and rolled forward using logged actions:

$$
\hat{\mathbf{s}}_0=\mathbf{s}_0,
$$

$$
\hat{\mathbf{s}}_{j+1}
=f_\omega(\hat{\mathbf{s}}_j,\mathbf{a}^{\mathrm{logged}}_j,\boldsymbol{\xi}).
$$

The loss combines state, reward, observation, and termination errors:

$$
\mathcal{L}_{\mathrm{WM}}(\omega)
=
\mathbb{E}_{\mathcal{B}}
\left[
\sum_{j=1}^{k}
\left(
\|\hat{\mathbf{s}}_j-\mathbf{s}_j\|^2
+
(\hat r_j-r_j)^2
+
\|\hat{\bar{\mathbf{o}}}_j-\bar{\mathbf{o}}_j\|^2
+
\operatorname{BCE}(\hat d_j,d_j)
\right)
\right].
$$

The autoregressive loss exposes the model to its own predictions and therefore addresses compounding error more directly than a one-step loss. World-model optimization uses Adam, global gradient clipping at norm 1.0, and batches of ten groups of 2048 windows per update.

## 4. Imagined rollouts and analytic policy gradients

During policy optimization, the world-model parameters are frozen, but the model remains differentiable with respect to its state and action inputs. Imagined rollouts begin from states sampled from the replay buffer:

$$
\hat{\mathbf{s}}_0=\mathbf{s}_0,\qquad \mathbf{s}_0\sim\rho_0.
$$

The world model produces a noise-free observation, sensor noise is reintroduced, and the policy generates an action:

$$
\hat{\bar{\mathbf{o}}}_t
=
g_\omega(\hat{\mathbf{s}}_t,\boldsymbol{\xi}),
$$

$$
\hat{\mathbf{o}}_t
=
\hat{\bar{\mathbf{o}}}_t+\boldsymbol{\eta}_t,
$$

$$
\mathbf{a}_t
=
\pi_\theta(\hat{\mathbf{o}}_t,\mathbf{g}_t).
$$

The model then predicts the next state, reward, and termination probability. The imagined survival weight is

$$
\hat c_0=1,\qquad
\hat c_{t+1}=\hat c_t(1-\hat d_{t+1}).
$$

The default imagination horizon is $H=32$ control steps, longer than the eight-step model-fitting horizon. A critic bootstraps the imagined return:

$$
\hat J(\theta)
=
\mathbb{E}
\left[
\sum_{t=0}^{H-1}
\gamma^t\hat c_t\hat r_{t+1}
+
\gamma^H\hat c_H
V_\psi(\hat{\mathbf{s}}_H,\mathbf{g}_H,\boldsymbol{\xi})
\right].
$$

The policy is updated by ascending the gradient of this objective,

$$
\theta\leftarrow\theta+\alpha_\theta\nabla_\theta\hat J.
$$

Because the imagined rollout is a differentiable computation graph, gradients propagate through actions, predicted states, future actions, rewards, and the terminal bootstrap. For each transition,

$$
\frac{\partial\hat{\mathbf{s}}_{t+1}}{\partial\theta}
=
\frac{\partial f_\omega}{\partial\hat{\mathbf{s}}_t}
\frac{\partial\hat{\mathbf{s}}_t}{\partial\theta}
+
\frac{\partial f_\omega}{\partial\mathbf{a}_t}
\frac{\partial\mathbf{a}_t}{\partial\theta}.
$$

This is the principal distinction from PPO. PPO treats the simulator transition as a black box and estimates policy gradients through sampled action log-probabilities and advantage estimates. SuperMimic instead uses the learned model Jacobians, including $\partial f_\omega/\partial\mathbf{a}_t$, to obtain analytic action sensitivities. The resulting gradient is biased with respect to the true simulator objective because it is exact only for the learned surrogate, but it has lower variance over short imagined horizons.

The critic is trained with generalized-advantage-estimation-style targets. With

$$
V_t=V_\psi(\hat{\mathbf{s}}_t,\mathbf{g}_t,\boldsymbol{\xi}),
$$

the target recursion is

$$
\hat R_t
=
\hat r_{t+1}
+
\gamma(1-\hat d_{t+1})
\left[
(1-\lambda)V_{t+1}
+
\lambda\hat R_{t+1}
\right],
$$

with $\hat R_H=V_H$ and $\lambda=0.95$. The critic loss is

$$
\mathcal{L}_V(\psi)
=
\mathbb{E}
\sum_{t=0}^{H-1}
(V_t-\hat R_t)^2.
$$

The critic permits short imagined rollouts while representing consequences beyond the imagination horizon. In the reported push-recovery experiment, critic-trained policies recovered from velocity pushes, whereas critic-free policies continued stumbling or fell.

## 5. Training loop and computational scaling

SuperMimic alternates among simulator data collection, world-model fitting, and policy improvement. The simulator executes the current policy with action noise and appends transitions to the replay buffer. The world model is then trained on autoregressive windows from the buffer. Finally, replay-buffer states seed imagined rollouts through the frozen differentiable model, and BPTT updates the policy while the critic is optimized from imagined targets.

This procedure is Dyna-like: real simulator states seed model-based imagination, while the replay buffer continually refreshes the distribution on which the model is trained. Earlier-policy transitions remain useful because the physical dynamics do not depend on which policy collected them.

The computational advantage has three components. First, learned rollouts are faster than articulated rigid-body simulation. On an H100, the learned model is reported to be approximately 11 times faster than MuJoCo Warp at 2048 parallel environments, 26 times faster at 16,384, and up to 62 times faster at 65,536. Second, BPTT supplies analytic action gradients rather than requiring extensive trial-and-error simulator interaction. In a mechanism ablation, SuperMimic reached the best return of PPO trained on world-model rollouts using approximately 64 times fewer imagined steps. Third, one world model amortizes the shared dynamics across all motions rather than learning separate models for individual clips.

The complete training process is therefore not a fixed model-pretraining stage followed by policy optimization. The world model, replay buffer, and policy are updated together. This joint loop reduces distribution shift between model training and policy improvement, although it does not eliminate model bias.

## 6. Evaluation, scaling behavior, and sim-to-real transfer

SuperMimic is compared primarily with SONIC and with PPO-Sim. SONIC uses 128 NVIDIA L40 GPUs for approximately seven days, or 168 hours, corresponding to roughly 21,000 GPU-hours. SuperMimic surpasses the released SONIC checkpoint’s normalized return of approximately 0.90 within about three hours on one H100. The paper reports a conservative reduction of more than $1000\times$ in GPU-hours, while noting that hardware differences prevent interpreting the ratio as a purely algorithmic speedup.

The large benchmark contains 129,514 filtered BONES-SEED motions. SONIC fails on 179 motions. SuperMimic fails on 25 motions after three hours on one H100 and reaches zero failures after 9.8 hours on two H100s. At the three-hour point, remaining failures include difficult side rolls, flips, fire-in-the-hole turns, lying, resting, and standing motions. Because the benchmark consists of training motions, these results measure coverage of the training library rather than generalization to unseen motion distributions.

On a controlled 40-clip, 2.45-hour LAFAN1 set, SuperMimic reaches normalized return 0.82 within approximately 1.2 hours on one GPU. Within an 18-hour budget, PPO-Sim reaches 0.60 on one GPU, 0.70 on four GPUs, and 0.72 on eight GPUs. The comparison indicates diminishing returns from simply adding GPUs to model-free PPO in this experiment.

The paper evaluates scaling across approximately 1, 7, 40, and 264 hours of motion. The normalized eight-step world-model prediction errors are approximately 0.00679, 0.00738, 0.01096, and 0.00993, respectively. Thus, prediction error remains in the approximate range 0.007–0.011 rather than increasing proportionally with motion-library duration. The compute required to reach a fixed target return grows slowly, with plotted normalized GPU-hour values of 0.539, 1.350, 1.908, and 1.892 for the respective subsets.

The method also supports multiple downstream goal interfaces. For interface $i$, a goal encoder produces a latent and a shared decoder maps that latent and proprioceptive observations to actions:

$$
\mathbf{z}=\operatorname{enc}_{\theta,i}(\mathbf{g}),
\qquad
\mathbf{a}=\operatorname{dec}_\theta(\mathbf{z},\mathbf{o}).
$$

Each interface can have its own goal space, reward, termination condition, and critic, while sharing the decoder and world model. A velocity-command interface uses $\mathbf{g}_t=(v_x,v_y,\omega_z)$. On 672 step responses, the RL-trained encoder has lower steady-state error on 650 commands, while the distillation encoder is better on 15. Median relative errors are 0.16 for RL and 0.49 for distillation; the distillation controller falls on seven fast-turn commands.

A single policy is transferred to a physical Unitree G1 without per-motion tuning. Demonstrated behaviors include a skipping jump, a side roll involving torso and limb contact with the floor, a $90^\circ$ flip, and an in-place $360^\circ$ flip. Transfer uses noisy proprioception, randomized friction, joint calibration offsets, torso center-of-mass perturbations, low-level PD control, and the removal of root-velocity perturbations from training because discontinuous state jumps are difficult for the deterministic world model to model. The physical evaluation is primarily qualitative rather than a broad statistical benchmark.

## 7. Limitations and significance

SuperMimic optimizes the gradient of the learned-model objective $\nabla_\theta\hat J$, not the exact gradient of the simulator return. If the learned dynamics function differs from the simulator dynamics, BPTT may optimize actions that perform well in imagination but poorly in the simulator. This risk is especially relevant near impacts, falls, contact-mode switches, joint limits, and unstable acrobatics.

The world model is trained on replay-buffer transitions, so rapid policy changes can drive imagined rollouts into poorly represented regions. The joint training loop mitigates but does not remove this distribution shift. Short-horizon optimization also depends on critic accuracy: the critic represents consequences beyond the imagination horizon, and inaccurate value estimates can produce myopic or unstable behavior. The model is differentiable even though true contact dynamics are nonsmooth; this provides usable gradients but may conceal discontinuities and mode changes.

Additional limitations concern scope. The world model is trained only from simulator transitions, although the paper suggests real-robot data could be used for system identification. Experiments use flat ground and do not cover general terrain, manipulation, or external-object interaction. The velocity-interface study is preliminary, and the paper does not establish that simultaneous training of many downstream interfaces is necessary. The benchmark results also evaluate training motions, so they do not by themselves establish generalization to unseen motion distributions.

The principal significance of SuperMimic is architectural and computational. It divides the role of simulation between grounded transition generation and scalable differentiable imagination. A single world model captures shared humanoid dynamics, while BPTT supplies analytic policy gradients through short rollouts and a critic extends their effective horizon. The reported results indicate that this combination can substantially reduce the compute required for large-scale humanoid motion tracking while retaining transfer across locomotion, contact-rich rolling, and acrobatic motions [2610.28480].

Source: https://www.emergentmind.com/topics/supermimic