GigaWorld-0: Unified Vision-Language-Action Model

• GigaWorld-0 is a unified world model framework that integrates video synthesis and geometry-aware 3D modeling to generate scalable, instruction-aligned data for Vision-Language-Action learning.
• It employs a dual-branch pipeline that conditions photorealistic video generation and 3D asset reconstruction using techniques like Gaussian Splatting and differentiable system identification.
• GigaTrain, a distributed FP8 training engine with sparse attention, enables efficient optimization of large-scale models, driving strong sim2real generalization in complex robotic tasks.

GigaWorld-0 is a unified world model framework designed as a "world-model-as-data-engine" for scalable, data-efficient Vision-Language-Action (VLA) learning in embodied AI. It integrates synergistic video generation and geometry-aware 3D modeling modules to synthesize visually compelling, physically plausible, spatially coherent, and instruction-aligned embodied interaction data. This synthetic data underpins VLA model training, enabling strong real-world generalization and policy deployment for complex robotics tasks, even in the absence of real-world training interaction. GigaTrain, a distributed training engine leveraging FP8 precision and sparse attention, enables efficient large-scale training that makes this paradigm tractable for foundation models (Team et al., 25 Nov 2025).

1. Architectural Overview

GigaWorld-0 decomposes into three primary components:

• GigaWorld-0-Video synthesizes photorealistic, temporally coherent video sequences of embodied interaction. It supports fine-grained conditioning on text, images, camera viewpoint, appearance, and action semantics.
• GigaWorld-0-3D generates and reconstructs 3D assets using generative modeling, 3D Gaussian Splatting (3DGS), physically differentiable system identification (PINN), and executable motion planning to guarantee geometric and physical fidelity.
• GigaTrain implements massively distributed model optimization using FP8 arithmetic and neighborhood-based sparse attention to achieve significant reductions in memory and compute consumption, enabling training of large-scale models and complex pipelines.

System Pipeline

Input modalities—combinations of text prompts, example images, short videos, and optionally action sequences—are processed via two major branches:

• Video branch: Inputs propagate through IT2V foundation models into controllable branches (AppearanceTransfer, ViewTransfer, MimicTransfer) to yield generated videos and inferred robot action traces using an Inverse Dynamics Model.
• 3D branch: Foreground actors are generated (image-to-3D), and backgrounds are reconstructed via 3DGS. Physical parameters are assigned through differentiable system identification, followed by physically grounded motion planning.
• Output integration: Resulting data streams are fed into downstream VLA model training (e.g., GigaBrain-0) or used directly as a policy-environment simulation.

2. GigaWorld-0-Video: Controllable Video Generation

Probabilistic Generative Formulation

Video sequence $\mathbf{x}_{1:T}$ generation, given action sequence $\mathbf{a}_{1:T}$, camera viewpoint $v$, and appearance controls $c$, is modeled as a conditional distribution: \begin{equation} p_\theta\bigl(\mathbf{x}{1:T}\mid \mathbf{a}{1:T},\,v,\,c\bigr)\,. \end{equation}

The training objective is to minimize the composite negative log-likelihood: \begin{equation} \mathcal{L}\mathrm{Video} = \mathbb{E}{q}\bigl[-\log p_\theta(\mathbf{x}{1:T}\mid \mathbf{a}{1:T},v,c)\bigr] = \lambda_{r}\,\mathcal{L}{\mathrm{rec}} + \lambda{t}\,\mathcal{L}{\mathrm{temp}} + \lambda{s}\,\mathcal{L}_{\mathrm{sem}}, \end{equation} where:

• $$\mathcal{L}_\mathrm{rec} = \sum_{t=1}^T \lVert x_t - \hat{x}_t\rVert_2^2$$ (reconstruction loss),
• $$\mathcal{L}_\mathrm{temp} = \sum_{t=2}^T \lVert (x_t - x_{t-1}) - (\hat{x}_t - \hat{x}_{t-1})\rVert_2^2$$ (temporal coherence),
• $$\mathcal{L}_\mathrm{sem} = 1 - \mathrm{SIM}(\mathrm{CLIP}(x_{1:T}),\,\mathrm{CLIP}(\text{prompt}))$$ (semantic alignment).

Foundation Model Architecture

GigaWorld-0-Video-Dreamer employs a diffusion architecture based on flow matching: \begin{equation} \frac{d\mathbf{z}t}{dt} = \mathbf{v}\theta(\mathbf{z}t,\,t,\,\mathbf{c}), \end{equation} with latents $\mathbf{z}_t$ derived using a 3D-VAE and 3D rotary position embeddings. The backbone leverages sparse attention (NATTEN) and a 4-expert Mixture-of-Experts (MoE), routing two experts per token, with load-balancing regularized by: \begin{equation} \mathcal{L}{\mathrm{Load}} = \alpha\sum_{i=1}^{N_r}f_i\,P_i, \end{equation} controlling expert assignment frequency and capacity.

Fine-Grained Control Branches

Branches enable targeted data generation:

• AppearanceTransfer: Combines control latents (e.g., depth, normals) with diffusion noise using MLPs.
• ViewTransfer: Dual-conditions generation on warped backgrounds and simulated arm renderings, adapting end-effector pose $K_t$ using: \begin{equation} K_t = (T^{\mathrm{base}\to\mathcal{W}_B})^{-1}\,T^{\mathrm{base}\to\mathcal{W}_A}\,T_t^{\mathrm{ee}\to\mathrm{base}}, \end{equation} ensuring geometric consistency under changing viewpoints.
• MimicTransfer: For human–robot transfer, reconstructs robot arm sequences from human first-person video, using IK joint angle inference.

3. GigaWorld-0-3D: Geometry, Physics, Planning

3D Gaussian Splatting (3DGS) Background Reconstruction

Scene backgrounds are modeled with parameterized Gaussian splats (weight $w_i$, mean $\mathbf{a}_{1:T}$0, covariance $\mathbf{a}_{1:T}$1), contributing to the radiance field: \begin{equation} f(\theta) = \sum_{i} w_i \exp!\Bigl(-\frac{1}{2}(x-\mu_i)^\top\Sigma_i^{-1}(x-\mu_i)\Bigr). \end{equation} Sparse-view inputs are completed by a view-restoration network followed by dense 3DGS refinement.

Physically Differentiable System Identification

Robot and object dynamics are modeled using differentiable PINN surrogates. Let $\mathbf{a}_{1:T}$2 be trajectory data and $\mathbf{a}_{1:T}$3 the parameters (friction, stiffness, damping). The surrogate dynamics $\mathbf{a}_{1:T}$4 is fit by: \begin{equation} \min_{\phi}\sum_t\lVert \mathcal{M}{\phi}(\mathbf{s}{t-1},\mathbf{a}{t-1}) - \mathbf{s}_t^{\mathrm{sim}}\rVert^2 \quad\longrightarrow\quad \min{f,p,d}\sum_t\lVert \mathcal{M}{\phi}(\mathbf{s}{t-1},\mathbf{a}_{t-1};f,p,d) - \mathbf{s}_t^{\mathrm{real}}\rVert^2. \end{equation} Physical property inference uses a Vision-Language expert (Qwen3-VL); for deformables, PhysTwin-style spring-mass modeling is incorporated.

Executable Motion Planning

Given states $\mathbf{a}_{1:T}$5 and controls $\mathbf{a}_{1:T}$6, the following optimization is solved: \begin{align} &\min_{{\mathbf{u}t}{t=1}^T}\sum_{t=1}^T c(\mathbf{x}t,\mathbf{u}_t), \ &\text{s.t. } \mathbf{x}{t+1}=f_{\mathrm{phys}}(\mathbf{x}_t,\mathbf{u}_t),\quad g(\mathbf{x}_t,\mathbf{u}_t)\le0, \end{align} where the cost $\mathbf{a}_{1:T}$7 encodes task objectives and physical constraints. MimicGen is used for cloning simple tasks; complex behaviors utilize rapid RL-PD policy refinement.

4. Joint Optimization and Synthetic Data Pipeline

When applicable, the Video and 3D modules are jointly optimized with a total loss: \begin{equation} \mathcal{L}{\mathrm{total}} = \lambda{\mathrm{vid}}\mathcal{L}{\mathrm{Video}} + \lambda{\mathrm{3D}}\mathcal{L}{\mathrm{3DRecon}} + \lambda{\mathrm{phys}}\mathcal{L}{\mathrm{phy}} + \lambda{\mathrm{plan}}\mathcal{L}_{\mathrm{plan}}, \end{equation} where $\mathbf{a}_{1:T}$8 arises from 3DGS reprojection errors, $\mathbf{a}_{1:T}$9 from system identification, and $v$0 from trajectory feasibility constraints.

Evaluation metrics include:

• Data diversity (embedding-space variance)
• Temporal coherence (frame-difference consistency)
• Physical plausibility (VideoPhy PA score)
• Geometric consistency (multi-view reprojection error)

5. GigaTrain: Efficient Distributed Training

GigaTrain employs multiple optimizations for tractability at scale:

• FP8 Arithmetic: All GEMMs and attention layers operate in 8-bit floating point (1 sign, 5 exponent, 2 mantissa), with master copies in FP16/BF16 for stability.
• Sparse Attention: Neighborhood-attention (NATTEN) policies reduce compute and memory from $v$1 to $v$2, attending within spatial windows and selected long-range dependencies.
• Distributed Scheduling: Batch and gradient aggregation, activation checkpointing (especially in MoE feedforward networks), and backends like DeepSpeed-ZeRO or FSDP.
• Empirical Results: Up to 50% memory savings and step times of approximately 25 s per step on 8 × H20 GPUs with FP8 + sparse attention + ZeRO-2. Training schedulers operate as illustrated: $v$3

6. Empirical Results and Applications

Benchmarks and Metrics

GigaWorld-0-Video-Dreamer (2B active parameters) achieves an overall score of 82.07 on the PBench Robot Set, surpassing Cosmos-Predict2.5 (79.95), with consistent advances across scene fidelity, motion accuracy, and object consistency. On the DreamGen Bench, the model outperforms Wan2.2 and Cosmos variants in instruction fidelity and physical alignment.

Evaluation includes both quantitative scores and qualitative demonstrations, such as multi-view coherence, human-to-robot mimicry, and appearance edits, all exhibiting semantic correctness and temporal stability.

Real-World Transfer

VLA models (e.g., GigaBrain-0) trained solely on GigaWorld-0 synthetic data enable real robot policies with high success rates: laundry folding (>90%), paper-towel preparation (≈88%), table bussing, juice/box/basket moving (all >85%), with zero real-world interaction during training.

Application Domains

• Household robotics (e.g., cleaning, folding, kitchen tasks)
• Industrial assembly and warehouse logistics
• Policy pre-training for zero-shot sim2real robotic deployment

7. Limitations and Open Problems

Open challenges include:

• Diffusion artifacts and hallucinations occasionally degrade downstream policy deployment; mitigation utilizes composite quality scoring and automated rejection.
• Physical and geometric realism is bounded by the performance of the PINN surrogate system identification and Gaussian Splatting accuracy.
• Transitioning from a data-engine paradigm to a fully interactive, closed-loop model-based RL environment remains a future direction.

A plausible implication is that further advances in physically grounded modeling and interactive world models could enhance robustness and generalization in embodied AI policy learning.