DreamZero: Diffusion-Driven Robot Skills

• DreamZero is a world action model that jointly predicts future video frames and robot actions from language, observations, and proprioceptive data, enabling efficient skill acquisition.
• Its autoregressive diffusion framework leverages spatiotemporal priors for long-horizon planning, achieving zero-shot generalization and real-time low-latency control.
• Optimization techniques like classifier-free guidance and caching deliver a 38× speedup, facilitating robust cross-embodiment transfer and improved performance on novel tasks.

DreamZero is a World Action Model (WAM) designed to advance the generalization and efficiency of robot skill acquisition by jointly predicting future video frames and robot actions conditioned on language instructions, observations, and proprioceptive state. In contrast to Vision-Language-Action (VLA) models that map static image–text representations to actions, DreamZero leverages spatiotemporal priors from large-scale video diffusion pretraining to model environment dynamics, resulting in state-of-the-art zero-shot generalization, real-time closed-loop control, and cross-embodiment transfer of skills (Ye et al., 17 Feb 2026).

1. Conceptual Foundations and Model Distinctions

DreamZero operates by aligning action inference with visual future prediction within an autoregressive diffusion framework. The model builds on a pretrained video diffusion transformer backbone, enabling it to utilize dense video representations for explicit learning of physical dynamics rather than relying solely on direct imitation from static perceptual embeddings. Key distinctions relative to VLAs include:

• Joint video–action prediction: Action generation is tightly coupled with visual futures, functioning as an implicit inverse-dynamics model rather than a pure state–action mapping.
• Autoregressive diffusion backbone: The model performs chunk-wise, temporally consistent prediction over both modalities, enabling long-horizon planning and closed-loop deployment without loss of native control frequency.
• Real-time, low-latency inference: System and model optimizations allow the 14-billion-parameter backbone to close the control loop at 7 Hz, reducing inference latency from approximately 5.7 s to 150 ms.

A plausible implication is that DreamZero’s approach may generalize to other domains requiring high-fidelity, temporally coupled multimodal prediction.

2. Mathematical and Algorithmic Formulation

Joint Generation Decomposition

DreamZero models the joint distribution over future video frames $\mathbf{o}_{l:l+H}$ and actions $\mathbf{a}_{l:l+H}$ as follows: $$\pi_0(\mathbf{o}_{l:l+H},\mathbf{a}_{l:l+H}\mid \mathbf{o}_{0:l},\mathbf{c},\mathbf{q}_l) = \pi_0(\mathbf{o}_{l:l+H}\mid \mathbf{o}_{0:l},\mathbf{c},\mathbf{q}_l)\: \pi_0(\mathbf{a}_{l:l+H}\mid \mathbf{o}_{0:l+H},\mathbf{q}_l)$$

Flow-Matching Objective

Let video latents $\mathbf{z}_1^k$ and normalized actions $\mathbf{a}_1^k$ define chunk $k$. For denoising timestep $t \in [0,1]$: $$\mathbf{z}_{t}^k = t\,\mathbf{z}_1^k + (1-t)\,\mathbf{z}_0^k,\quad \mathbf{a}_{t}^k = t\,\mathbf{a}_1^k + (1-t)\,\mathbf{a}_0^k,$$ using noise vectors sampled $$\mathbf{z}_0^k,\mathbf{a}_0^k \sim \mathcal{N}(0,I)$$. The model’s predictor $\mathbf{u}_\theta$ estimates the velocity $\mathbf{a}_{l:l+H}$0, and the loss is: $\mathbf{a}_{l:l+H}$1

Although DreamZero employs a unified flow-matching objective, modality-separated losses are conceptually possible: $\mathbf{a}_{l:l+H}$2 with weighted total loss $\mathbf{a}_{l:l+H}$3.

3. Model Architecture and Training Paradigm

The backbone of DreamZero is the Wan‐2.1 I2V‐14B‐480P, a 14-billion-parameter Video Diffusion Transformer trained via flow matching on web-scale video data. Fixed-length video frame chunks ($\mathbf{a}_{l:l+H}$4 latent frames) correspond to 1.6 s of 30 Hz video.

• KV-cached autoregressive attention mechanisms support arbitrary context lengths.
• Chunk-wise teacher-forcing ensures denoising of each predicted chunk conditioned on previously generated clean chunks, avoiding temporal drift.

Inputs include:

• Video latents $\mathbf{a}_{l:l+H}$5: Derived from a fixed VAE encoder.
• Language instructions: Encoded by a frozen text encoder.
• Proprioceptive state $\mathbf{a}_{l:l+H}$6: Encoded by a dedicated state encoder.
• Actions: Jointly denoised with video to enforce cross-modal alignment.

Training incorporates main and “–Flash” stages:

• Main stage: 100K steps, batch size 128, updates DiT layers/state/action encoders and decoders, while VAE and text encoder remain frozen.
• –Flash fine-tuning: Decoupled noise schedules (Beta-distributed high noise for video, uniform for actions) facilitate efficient single-step action denoising.

Data encompasses ~500 h AgiBot G1 teleoperation, ~340 h Franka/DROID data, and cross-embodiment video-only human/YAM demonstrations.

4. Performance Evaluation and Transfer Abilities

Zero-Shot and Cross-Embodiment Results

Setting	DreamZero (%)	Pretrained VLAs (%)	Relative Gain
Zero-shot, seen tasks/environments	62.2	27.4	≈2.3×
Zero-shot, unseen tasks (novel verbs/motions)	39.5	16.3	>2×
Cross-embodiment, video-only (YAM→AgiBot)	55.4	38.3 (baseline)	+44%
Cross-embodiment, video-only (Human→AgiBot)	54.3	38.3 (baseline)	+42%
Few-shot YAM adaptation (30 min play data)	Strong adaptation and retention of zero-shot generalization	—	—

DreamZero demonstrates substantially superior generalization to new physical motions and environments, achieving 2.3× average task progress compared to prior VLAs in challenging real-world scenarios. Cross-embodiment transfer via video-only demonstration yields over 42% relative improvement for unseen tasks, with successful adaptation to new robot embodiments in under 30 minutes.

Fine-tuning on task-specific datasets (e.g., shirt folding, fruit packing, table bussing) shows that DreamZero matches or exceeds prior VLA models while retaining environment generalization.

5. System Architecture and Latency Optimizations

System-level and model-level advancements enable real-time deployment:

• Asynchronous closed-loop execution: Model inference and action execution run in parallel, using the last available action chunk during generation.
• Classifier-Free Guidance (CFG) parallelism: Simultaneous conditional/unconditional DiT passes across dual GPUs reduce guidance overhead by ~47%.
• DiT caching: Reuses velocity predictions when cosine similarity is high between prediction steps, reducing effective steps from 16 to 4.
• Implementation techniques: torch.compile with CUDA Graphs, mixed-precision quantization (NVFP4/FP8/FP16 levels) on Blackwell GPUs, and CuDNN-accelerated attention remove CPU–GPU bottlenecks.
• –Flash: Decoupled video/action noise training enables single-step action denoising with only ~9% performance drop compared to 4-step denoising (74% vs 83% on table bussing), at 150 ms latency.

The aggregate result is a roughly 38× speedup (from baseline to cumulative optimizations) on GB200-class hardware.

6. Limitations, Open Questions, and Prospects

DreamZero’s performance and scalability motivate several research directions and limitations:

• Scaling laws: Systematic studies are required to understand the interplay of model size, data scale, and compute in WAM action quality.
• Human-to-robot transfer: Early experiments leveraging egocentric human video (e.g., Ego4D, Action100M) suggest significant potential for skill prior induction.
• Further acceleration: Research into smaller or distilled diffusion backbones could improve control frequencies to over 20 Hz on edge hardware.
• Long-horizon/high-precision control: Extending context windows for multi-minute action sequences and investigating deterministic control for sub-millimeter accuracy remain open challenges.
• Embodiment design: Trade-offs between degrees of freedom and human similarity for cross-embodiment and inverse-dynamics learning require deeper study (Ye et al., 17 Feb 2026).

7. Algorithmic Summary

DreamZero’s training and closed-loop inference pipelines are characterized by:

• Chunk-wise flow-matching training: Incorporating optional –Flash noise schedules.
• Closed-loop autoregressive inference: Employing KV cache updates and asynchronous execution.

Algorithm summaries and pseudo-code sketches are detailed in the appendix of the original source.

---

By tightly coupling diffusion-based video prediction and action inference, DreamZero advances the frontier in zero-shot robotics, heterogeneous data skill learning, data-efficient cross-embodiment transfer, and real-time visuomotor control (Ye et al., 17 Feb 2026).