---
title: 'MotionWAM: Real-Time Humanoid Control'
url: https://www.emergentmind.com/topics/motionwam
type: topic
---

# MotionWAM: Real-Time Humanoid Control

MotionWAM is a real-time world action model (WAM) designed for autonomous humanoid loco-manipulation, integrating egocentric vision, language goals, and fully unified motion control. It addresses key limitations of prior WAMs in real-time performance and embodiment coverage by introducing a fast dual-Diffusion Transformer architecture and a unified motion latent that encompasses the entire body, including feet, torso, and manipulation actions. MotionWAM achieves substantial performance improvements over state-of-the-art Vision-Language-Action (VLA) baselines on a comprehensive suite of real-world humanoid tasks [2606.09215].

## 1. Formulation and Problem Setting

MotionWAM models the joint distribution over future egocentric video observations and whole-body robot actions, inverting this distribution to select actions that causally steer predicted visual outcomes toward a goal. At decision time \( t \), the system receives:

- Current egocentric RGB frame \( o_t \)
- Proprioceptive state \( p_t \) (e.g., joint positions, velocities)
- Language goal prompt \( l \)

The generative process consists of:

1. Sampling an intermediate future-frame latent \( \mathbf{z}_{t+1}^{\tau_v} \) via a video diffusion prior.
2. Extracting hidden features \( \mathbf{h}_t^{\tau_f} \) from this latent.
3. Conditioning a motion diffusion transformer on these features (and proprioception) to sample a unified motion latent \( \mathbf{m}_t \).

The model is formalized as:
\[
\begin{aligned}
o_{t+1} &\sim p_{v}(\cdot \mid o_t, l) \\
m_t &\sim p_{a}(\cdot \mid o_t, p_t, \mathcal H(o_{t+1}^{\tau_v}))
\end{aligned}
\]

Training uses a flow-matching loss for both video (\( \mathcal{L}_{\rm video} \)) and motion (\( \mathcal{L}_{\rm motion} \)) diffusion transformers, with continuous-time noise injection for efficient learning and stable optimization.

## 2. Unified Motion Latent Representation

MotionWAM eliminates the traditional hierarchical split between upper-body manipulation and lower-body locomotion, building instead a unified action representation covering:

- Locomotion
- Torso and height regulation
- Foot–environment interaction
- Hand/gripper manipulation

The motion latent \( \mathbf{m}_t \) comprises:

- **Discrete SONIC tokens**: \( \mathbf{k}_t \in \left\{ -1, -\frac{15}{16}, \ldots, 1 \right\}^{64} \) quantized intent features, capturing coarse spatio-temporal action intent for locomotion, torso, height, and feet.
- **Continuous control channels**: \( \mathbf{m}_t^{\rm cont} \), handling dexterous manipulation and other fine-scale actuation not bottlenecked by quantization.

Inference involves diffusion regression of \( \mathbf{m}_t \), followed by nearest-neighbor rounding of \( \tilde{k}_t \) to obtain the action to be deployed via the SONIC controller.

## 3. Dual Diffusion Transformer Architecture

MotionWAM employs a dual-DiT (Diffusion Transformer) architecture:

### Video DiT

- Initialized from Cosmos-Predict2.5-2B.
- Spatio-temporal VAE encoder/decoder is combined with a flow-matching diffusion transformer.
- Processes two consecutive frames \((o_{t-1}, o_t)\) and language goal \( l \).
- Uses a one-shot regime (pure-noise latent at \( \tau_f \approx 1 \)), forgoing iterative denoising.

### Motion DiT

- DiT-B configuration interleaving self- and cross-attention.
- Inputs: video hidden features \( \mathbf{h}_t^{\tau_f} \), proprioceptive vector \( p_t \), noisy motion latent \( \mathbf{m}_t^{\tau_a} \), embodiment index \( e \).
- Outputs: velocity for motion latents, supporting continuous adaptation.
- Employs per-embodiment projectors in cross-robot pretraining.

Losses during training:
\[
\mathcal{L}_{\rm S2} = \mathcal{L}_{\rm video} + \mathcal{L}_{\rm motion}
\]
where individual losses correspond to velocity prediction objectives in video and motion latent space, as detailed above.

## 4. Three-Stage Progressive Learning Framework

MotionWAM's training involves progressive specialization over three stages:

1. **Egocentric Video Pretraining**
   - Data: 2,136 hours of human and humanoid-robot egocentric video (no action labels).
   - Optimizes \( \mathcal{L}_{\rm video} \), adapting Cosmos DiT to real-world visual dynamics.

2. **Cross-Embodiment Action Post-Training**
   - Data: Multiple humanoid datasets, including Unitree G1 with per-embodiment action masks.
   - Optimizes both video and motion DiTs under \( \mathcal{L}_{\rm S2} \), regularizing video model on original flow-matching loss.

3. **Unitree G1 Whole-Body Fine-Tuning**
   - Data: 9 real-world loco-manipulation tasks × 200 teleoperation episodes.
   - Continues \( \mathcal{L}_{\rm S2} \); the action decoder is now unified SONIC latent pipeline.
   - Policy learning is by supervised flow matching (no RL or value learning).

## 5. Real-Time Inference Pipeline

During execution, MotionWAM operates with low-latency by avoiding iterative video denoising. The sequence per control step is:

1. One forward Video DiT pass: encodes \( o_t \) to latent \( \mathbf{z}_t^0 \); injects pure-noise latent at \( \tau_f \approx 1 \); outputs hidden features \( \mathbf{h}_t^{\tau_f} \).
2. Motion DiT pass: conditioned on \( \mathbf{h}_t^{\tau_f} \) and proprioception, regresses velocity for \( \mathbf{m}_t \).
3. Decoding and rounding: \( \mathbf{m}_t \to \mathbf{a}_t \) via SONIC controller.

This system achieves approximately 4.9 Hz on a single NVIDIA A100 (2.5 B parameters), substantially faster than earlier WAMs requiring iterative denoising (typically below 1 Hz), although slightly slower than pure diffusive or matched-scale VLA baselines that lack video dynamics modeling.

### Inference Pseudocode

```python
# Input: current frame o_t, proprioceptive state p_t, language goal l
# Output: joint commands a_t

1. z_t^0 = encode_vae(o_t)
2. ε_v ~ N(0,I); τ_f ≈ 1
   z_{t+1}^{τ_f} = (1 - τ_f) * z_{t+1}^0 + τ_f * ε_v
3. h_t^{τ_f} = VideoDiT(z_{t+1}^{τ_f}, τ_f | z_t^0, l)
4. ε_m ~ N(0, I); select τ_a
   m_t^{τ_a} = (1 - τ_a) * m_t^0 + τ_a * ε_m
5. ḿ = MotionDiT(m_t^{τ_a}, τ_a | h_t^{τ_f}, p_t)
   integrate ḿ → ŷ m_t
6. ŷ k_t = round(ŷ \tilde{k}_t)
   ŷ m̂_t = (ŷ m_t^{cont}, ŷ k_t)
7. a_t = decode_sonic(ŷ m̂_t)
```

## 6. Experimental Evaluation

Evaluated on nine real-world loco-manipulation tasks with Unitree G1, all requiring coordinated whole-body actions:

- PnP Bottle
- Kick Soccer
- Retrieve Item
- Load Cart
- Toss Garbage
- Lift Basket
- Stock Shelves
- Wipe Board
- Do Laundry

### Comparative Performance

| Model         | Success Rate (%) | Notable Gains on Feet-Heavy Tasks |
|---------------|-----------------|-----------------------------------|
| MotionWAM     | 76.1            | +40–45% on several tasks          |
| Best VLA      | 43.9            |                                   |
| Pure Diff.    | <9 Hz, weaker   |                                   |

MotionWAM exceeds the best VLA baseline by +32% absolute on overall success. On tasks demanding active feet/torso (e.g., Kick Soccer, Load Cart, Retrieve Item, Wipe Board, Do Laundry), success rates improve by 30–45%. Qualitatively, the system successfully incorporates lower body for task-driven actions (pedal stepping, squatting) that VLA or pipelined approaches cannot cover.

## 7. Architectural Rationale, Generalization, and Limitations

MotionWAM's representational and architectural choices are motivated by the need for closed-loop, temporally coherent whole-body control:

- Conditioning policy on video dynamics prior provides temporally grounded physical context absent in static VLMs.
- Unified motion latent enables non-trivial whole-body synergies, permitting foot, torso, and hand/arm actuation within a single transformer block—precluding the upper-and-lower body policy decoupling seen in prior art.

### Limitations

- Demonstrated only on the Unitree G1 embodiment; cross-robot generalization remains open.
- Relies on a single head-mounted camera; performance may degrade with occlusions or adverse lighting.
- Evaluated tasks overlap with training-time visual distributions; out-of-distribution generalization, especially for novel objects, is not established.

*This suggests that further work is needed in multi-embodiment transfer and robust perception for visually unstructured environments.*

---

**References:**  
- "MotionWAM: Towards Foundation World Action Models for Real-Time Humanoid Loco-Manipulation" [2606.09215]

Source: https://www.emergentmind.com/topics/motionwam