MotionWAM: Real-Time Humanoid Control

• MotionWAM is a real-time world action model that unifies humanoid locomotion and manipulation by integrating egocentric vision and language prompts.
• It employs a fast dual-diffusion transformer architecture to generate a unified motion latent covering feet, torso, and hand actions for coordinated control.
• The model shows substantial performance improvements over existing VLA baselines, achieving higher success rates in diverse real-world loco-manipulation tasks.

MotionWAM is a real-time world action model (WAM) designed for autonomous humanoid loco-manipulation, integrating egocentric vision, language goals, and fully unified motion control. It addresses key limitations of prior WAMs in real-time performance and embodiment coverage by introducing a fast dual-Diffusion Transformer architecture and a unified motion latent that encompasses the entire body, including feet, torso, and manipulation actions. MotionWAM achieves substantial performance improvements over state-of-the-art Vision-Language-Action (VLA) baselines on a comprehensive suite of real-world humanoid tasks (Zheng et al., 8 Jun 2026).

1. Formulation and Problem Setting

MotionWAM models the joint distribution over future egocentric video observations and whole-body robot actions, inverting this distribution to select actions that causally steer predicted visual outcomes toward a goal. At decision time $t$, the system receives:

• Current egocentric RGB frame $o_t$
• Proprioceptive state $p_t$ (e.g., joint positions, velocities)
• Language goal prompt $l$

The generative process consists of:

1. Sampling an intermediate future-frame latent $\mathbf{z}_{t+1}^{\tau_v}$ via a video diffusion prior.
2. Extracting hidden features $\mathbf{h}_t^{\tau_f}$ from this latent.
3. Conditioning a motion diffusion transformer on these features (and proprioception) to sample a unified motion latent $\mathbf{m}_t$.

The model is formalized as: $$\begin{aligned} o_{t+1} &\sim p_{v}(\cdot \mid o_t, l) \ m_t &\sim p_{a}(\cdot \mid o_t, p_t, \mathcal H(o_{t+1}^{\tau_v})) \end{aligned}$$

Training uses a flow-matching loss for both video ($\mathcal{L}_{\rm video}$) and motion ($\mathcal{L}_{\rm motion}$) diffusion transformers, with continuous-time noise injection for efficient learning and stable optimization.

2. Unified Motion Latent Representation

MotionWAM eliminates the traditional hierarchical split between upper-body manipulation and lower-body locomotion, building instead a unified action representation covering:

• Locomotion
• Torso and height regulation
• Foot–environment interaction
• Hand/gripper manipulation

The motion latent $o_t$0 comprises:

• Discrete SONIC tokens: $o_t$1 quantized intent features, capturing coarse spatio-temporal action intent for locomotion, torso, height, and feet.
• Continuous control channels: $o_t$2, handling dexterous manipulation and other fine-scale actuation not bottlenecked by quantization.

Inference involves diffusion regression of $o_t$3, followed by nearest-neighbor rounding of $o_t$4 to obtain the action to be deployed via the SONIC controller.

3. Dual Diffusion Transformer Architecture

MotionWAM employs a dual-DiT (Diffusion Transformer) architecture:

Video DiT

• Initialized from Cosmos-Predict2.5-2B.
• Spatio-temporal VAE encoder/decoder is combined with a flow-matching diffusion transformer.
• Processes two consecutive frames $o_t$5 and language goal $o_t$6.
• Uses a one-shot regime (pure-noise latent at $o_t$7), forgoing iterative denoising.

Motion DiT

• DiT-B configuration interleaving self- and cross-attention.
• Inputs: video hidden features $o_t$8, proprioceptive vector $o_t$9, noisy motion latent $p_t$0, embodiment index $p_t$1.
• Outputs: velocity for motion latents, supporting continuous adaptation.
• Employs per-embodiment projectors in cross-robot pretraining.

Losses during training: $p_t$2 where individual losses correspond to velocity prediction objectives in video and motion latent space, as detailed above.

4. Three-Stage Progressive Learning Framework

MotionWAM's training involves progressive specialization over three stages:

1. Egocentric Video Pretraining
   • Data: 2,136 hours of human and humanoid-robot egocentric video (no action labels).
   • Optimizes $p_t$3, adapting Cosmos DiT to real-world visual dynamics.
2. Cross-Embodiment Action Post-Training
   • Data: Multiple humanoid datasets, including Unitree G1 with per-embodiment action masks.
   • Optimizes both video and motion DiTs under $p_t$4, regularizing video model on original flow-matching loss.
3. Unitree G1 Whole-Body Fine-Tuning
   • Data: 9 real-world loco-manipulation tasks × 200 teleoperation episodes.
   • Continues $p_t$5; the action decoder is now unified SONIC latent pipeline.
   • Policy learning is by supervised flow matching (no RL or value learning).

5. Real-Time Inference Pipeline

During execution, MotionWAM operates with low-latency by avoiding iterative video denoising. The sequence per control step is:

1. One forward Video DiT pass: encodes $p_t$6 to latent $p_t$7; injects pure-noise latent at $p_t$8; outputs hidden features $p_t$9.
2. Motion DiT pass: conditioned on $l$0 and proprioception, regresses velocity for $l$1.
3. Decoding and rounding: $l$2 via SONIC controller.

This system achieves approximately 4.9 Hz on a single NVIDIA A100 (2.5 B parameters), substantially faster than earlier WAMs requiring iterative denoising (typically below 1 Hz), although slightly slower than pure diffusive or matched-scale VLA baselines that lack video dynamics modeling.

Inference Pseudocode

$l$3

6. Experimental Evaluation

Evaluated on nine real-world loco-manipulation tasks with Unitree G1, all requiring coordinated whole-body actions:

• PnP Bottle
• Kick Soccer
• Retrieve Item
• Load Cart
• Toss Garbage
• Lift Basket
• Stock Shelves
• Wipe Board
• Do Laundry

Comparative Performance

Model	Success Rate (%)	Notable Gains on Feet-Heavy Tasks
MotionWAM	76.1	+40–45% on several tasks
Best VLA	43.9
Pure Diff.	<9 Hz, weaker

MotionWAM exceeds the best VLA baseline by +32% absolute on overall success. On tasks demanding active feet/torso (e.g., Kick Soccer, Load Cart, Retrieve Item, Wipe Board, Do Laundry), success rates improve by 30–45%. Qualitatively, the system successfully incorporates lower body for task-driven actions (pedal stepping, squatting) that VLA or pipelined approaches cannot cover.

7. Architectural Rationale, Generalization, and Limitations

MotionWAM's representational and architectural choices are motivated by the need for closed-loop, temporally coherent whole-body control:

• Conditioning policy on video dynamics prior provides temporally grounded physical context absent in static VLMs.
• Unified motion latent enables non-trivial whole-body synergies, permitting foot, torso, and hand/arm actuation within a single transformer block—precluding the upper-and-lower body policy decoupling seen in prior art.

Limitations

• Demonstrated only on the Unitree G1 embodiment; cross-robot generalization remains open.
• Relies on a single head-mounted camera; performance may degrade with occlusions or adverse lighting.
• Evaluated tasks overlap with training-time visual distributions; out-of-distribution generalization, especially for novel objects, is not established.

This suggests that further work is needed in multi-embodiment transfer and robust perception for visually unstructured environments.

---

References:

• "MotionWAM: Towards Foundation World Action Models for Real-Time Humanoid Loco-Manipulation" (Zheng et al., 8 Jun 2026)