GigaWorld-0-Video Synthesis

• GigaWorld-0-Video is a video generation module in a unified world model, producing texture-rich, temporally coherent embodied video sequences with fine-grained control over appearance, viewpoint, and actions.
• It integrates a continuous flow-matching diffusion approach with efficient transformer architectures and Mixture-of-Experts for scalable and data-efficient Vision-Language-Action learning.
• The system achieves high performance by outperforming larger models on video realism benchmarks and enabling rapid single-step sampling via distillation.

GigaWorld-0-Video is the video generation module within GigaWorld-0, a unified world model framework developed as a scalable, data-efficient data engine for Vision-Language-Action (VLA) learning in embodied AI. Its core innovation lies in generating large-scale, texture-rich, and temporally coherent embodied video sequences with fine-grained controllability over appearance, camera viewpoint, and action semantics. By integrating a continuous-diffusion (flow-matching) approach with efficient transformer architectures and flexible post-training control branches, GigaWorld-0-Video synthesizes diverse datasets, facilitating robust training of embodied VLA models and enabling real-world robot generalization without requiring real data during training (Team et al., 25 Nov 2025).

1. Architecture and Data Processing Pipeline

GigaWorld-0-Video's architecture is purpose-built for multimodal conditional video generation with a focus on efficiency and control. Inputs consist of:

• Text Prompt ($c_\text{text}$): Natural language instructions (e.g., "Pick up the red block and place it on green block") are tokenized and encoded via a T5 encoder, producing embeddings $E_\text{text} \in \mathbb{R}^{L \times d}$.
• Visual "Seed" Frames ($v_1$): Optionally, keyframes are passed through a pretrained 3D Variational Autoencoder (3D-VAE) encoder, generating conditional latents $$z_\text{cond} \in \mathbb{R}^{T_0 \times H' \times W' \times C}$$. For typical 61-frame $480 \times 768$ videos, a compression ratio of $(T_0, H', W') = (4, 8, 8)$ produces $16$-channel latents.
• Patchification and Projection: Non-overlapping $1 \times 2 \times 2$ patchification further reduces spatial tokens, after which a linear projection maps latents to the DiT (diffusion transformer) token space (e.g., $d=1024$).

The core backbone is a DiT employing sparse neighborhood attention (NATTEN) and a 4-expert Mixture-of-Experts (MoE) position-wise FFN per block, with Rotary 3D Positional Embeddings (3D-RoPE) encoding spatio-temporal indices. Generation proceeds by parameterizing the evolution of latent variables as a "flow-matching" SDE/ODE, with sampling performed via backward integration starting from Gaussian noise, and distilled for single-step efficiency via Denoising-Step Distillation.

2. Continuous-Time Flow-Matching Generative Framework

GigaWorld-0-Video-Dreamer is grounded in a continuous-time flow-matching generative modeling strategy, akin to score-based diffusion models but employing a deterministic flow.

• Forward (Noising) ODE: For clean latent $z_0 \sim q(z_0|c)$ and $E_\text{text} \in \mathbb{R}^{L \times d}$0,

$E_\text{text} \in \mathbb{R}^{L \times d}$1

• Reverse (Generative) ODE: The model predicts the flow:

$E_\text{text} \in \mathbb{R}^{L \times d}$2

• Training Objective: The flow-matching loss is

$E_\text{text} \in \mathbb{R}^{L \times d}$3

• Denoising-Step Distillation: To enable rapid inference, a student network $E_\text{text} \in \mathbb{R}^{L \times d}$4 is distilled such that

$E_\text{text} \in \mathbb{R}^{L \times d}$5

No adversarial (GAN) objectives are used; squared-error flow loss directly supervises generation fidelity.

3. Modular Fine-Grained Control

Post-training, GigaWorld-0-Video supports specialized adaptation branches for appearance, viewpoint, and skill mimicking:

• Control Injection Mechanism: Each control channel processes auxiliary cues (e.g., depth, normals, motion) through a shared 3D-VAE encoder, producing control latents concatenated with the generative noise latents at each diffusion step. A two-layer MLP (with GeLU activation) fuses this modality-concatenated tensor into the transformer token dimension.
• Branch-Specific Control Modes:
  • AppearanceTransfer: Utilizes depth and normal maps (from VideoDepthAnything and LOTUS) plus textual appearance cues; the loss augments $E_\text{text} \in \mathbb{R}^{L \times d}$6 with

  $E_\text{text} \in \mathbb{R}^{L \times d}$7 - ViewTransfer: Enables background geometry editing (depth from MoGe and reprojection consistency) and end-effector pose transformations (rendered with Sapien), with supervision

  $E_\text{text} \in \mathbb{R}^{L \times d}$8 - MimicTransfer: Masks out robot arm/background and transfers human hand motion via inverse kinematics; supervision is

  $E_\text{text} \in \mathbb{R}^{L \times d}$9

The DiT backbone weights are frozen; only small adapters and control MLPs are fine-tuned for each branch.

4. Training Regime and Loss Formulation

Data and Augmentation

• Pretraining Data: Draws from public sources (AgiBotWorld, RoboMind) and proprietary datasets covering approximately $v_1$0 of real robot workspace across five domains and fourteen scenarios.
• Resolution: Each sample is a $v_1$1-frame $v_1$2 video with corresponding action/object/environment text captions.
• Augmentation: Includes in-context multi-view (stacked panoramas), random cropping, color, and viewpoint jitter.

Compute Efficiency

• Precision: Mixed-precision FP8 (9C27B0) for parameters, activations, and gradients.
• Sparse Attention: NATTEN prunes memory/compute by approximately $v_1$3.
• MoE: Four experts per FFN layer, two active per token ($v_1$4, $v_1$5).
• Distributed Training: FSDP-2 or DeepSpeed-ZeRO2.
• Activation Checkpointing: Applied to FFN layers when MoE is enabled.

Optimization Objective

The overall (pretraining + fine-tuning) loss is:

$v_1$6

where $v_1$7 regularizes MoE expert assignment and activity; $v_1$8 values balance reconstruction. Hyperparameters include $v_1$9, $$z_\text{cond} \in \mathbb{R}^{T_0 \times H' \times W' \times C}$$0.

5. Evaluation Protocols and Results

Quantitative Benchmarks

PBench (Robot Set): Assesses video realism across eight metrics (range: 0–100)—

• i2v-bg (background consistency)
• i2v-s (static consistency)
• aes (aesthetic)
• img (image fidelity)
• bg-con (background consistency)
• mot (motion quality)
• sub-con (scene-subject consistency)
• o-con (object consistency)

The mean of these metrics constitutes the overall quality score.

Model	Params	Score
Cosmos-Predict2 (14B)	14B	79.88
Wan2.2 (14B)	14B	78.85
Cosmos-Predict2.5 (2B)	2B	79.95
GigaWorld-0-Video-Dreamer	2B	82.07

DreamGenBench (GR1 subset): Measures instruction following (Qwen-IF, GPT-IF) and physical alignment (PA).

Method	Qwen-IF	GPT-IF	PA
Cosmos-Predict2.5 (2B)	0.930	0.480	0.471
GigaWorld-0-Video-Dreamer	0.966	0.586	0.446

Qualitative Analysis

• Diverse futures: Fixing seed frame but varying text prompts leads to distinct, plausible object-interaction sequences.
• AppearanceTransfer: Enables scene-consistent modification of textures, materials, and lighting.
• ViewTransfer: Generates novel egocentric perspectives with geometric and pose consistency.
• MimicTransfer: Facilitates transfer from human demonstration to robot arm, preserving semantic motion.
• Multi-view: Produces spatio-temporally coherent 3-view panoramas.
• Distillation: Single-step sampling yields a $$z_\text{cond} \in \mathbb{R}^{T_0 \times H' \times W' \times C}$$1 generation speedup with negligible quality loss ($$z_\text{cond} \in \mathbb{R}^{T_0 \times H' \times W' \times C}$$20.5 points).

6. Ablation Studies and Comparative Analysis

Ablation and comparative results underscore the parameter and compute efficiency:

• Parameter Efficiency: Dreamer with 2B activated parameters outperforms 14B-parameter competitors (Cosmos, Wan2.2) across benchmarks.
• FP8 and Sparse Attention: Yields a $$z_\text{cond} \in \mathbb{R}^{T_0 \times H' \times W' \times C}$$3 reduction in peak memory and $$z_\text{cond} \in \mathbb{R}^{T_0 \times H' \times W' \times C}$$4 faster training (on 8$$z_\text{cond} \in \mathbb{R}^{T_0 \times H' \times W' \times C}$$5H20 GPUs).
• MoE Effectiveness: Addition of 4-expert MoE (a $$z_\text{cond} \in \mathbb{R}^{T_0 \times H' \times W' \times C}$$6 parameter increase) improves motion fidelity by approximately 1.5 points.
• Single-Step Distillation: Results in a $$z_\text{cond} \in \mathbb{R}^{T_0 \times H' \times W' \times C}$$7 sampling speedup with a minimal decline ($$z_\text{cond} \in \mathbb{R}^{T_0 \times H' \times W' \times C}$$80.5 points) in overall quality.
• Control Branch Fine-Tuning: Specialized branches, trained on under $$z_\text{cond} \in \mathbb{R}^{T_0 \times H' \times W' \times C}$$9 pairs, increase controllability metrics (sub-con, o-con) to $480 \times 768$0 from a $480 \times 768$1 baseline.

This suggests the flow-matching backbone, together with modular control injections and efficient transformer design, enables GigaWorld-0-Video to serve as an effective and practical data engine for the diverse, controllable, and high-quality video synthesis required for large-scale training of embodied VLA models (Team et al., 25 Nov 2025).