GigaWorld-0-3D: Modular 3D World Synthesis

• GigaWorld-0-3D is an advanced, modular infrastructure that synthesizes infinite, photorealistic, and physically plausible 3D environments via hierarchical block-based modeling and inpainting.
• It employs a multi-stage process—from coarse layout design to fine structure refinement and latent decoding—to generate spatially coherent and render-ready scene assets.
• Its integration of differentiable system identification and motion planning makes it a comprehensive data engine for embodied Vision-Language-Action learning.

GigaWorld-0-3D is an advanced, modular infrastructure for the infinite synthesis of 3D worlds, designed as a world model data engine for Vision-Language-Action (VLA) learning and scalable embodied AI. Integrating hierarchical 3D generative modeling, Gaussian splatting-based scene reconstruction, differentiable system identification, and executable motion planning, GigaWorld-0-3D enables the synthesis of photorealistic, spatially coherent, and physically plausible 3D environments at giga-scale. Its architecture builds on and extends prior block-wise scene inpainting frameworks such as WorldGrow, establishing a unified pipeline that guarantees both visual realism and real-world actionable outputs for agents and robots (Li et al., 24 Oct 2025, Team et al., 25 Nov 2025).

1. Hierarchical Generative Framework

GigaWorld-0-3D synthesizes environments using a hierarchical, block-based approach with a multi-stage architecture:

1. Coarse Global Layout: The process commences from a seed structured 3D latent (SLAT) block, representing room-scale geometry at a coarse resolution. A structure generator $\mathcal G_s^c$ (implemented as a flow Transformer) proposes new sparse voxel centers $\{\mathbf p_i^c\}_{i=1}^{L^c}$ in the block grid. Overlapping context masks $m_s$ ensure information from adjacent regions is leveraged, guiding the expansion with geometric coherence.
2. Fine Structure Refinement: Coarse-level centers are trilinearly upsampled to fine resolution grids $\mathbf p_w^{c\uparrow f}$, partitioned into finer blocks. Each is encoded into an initial latent $\ell^{(0)}$, perturbed with partial noise $t'$ to obtain $\ell^{(t')}$, which a fine structure generator $\mathcal G_s^f$ denoises to recover detailed geometry.
3. Appearance (Latent) Generation: Fine-structure SLAT masks $m_l$ and context latents condition a latent generator $\mathcal G_l^f$, yielding refined latent features $\{(\mathbf z_i, \mathbf p_i)\}$ that encode local photometric and semantic properties.
4. Decoding: The complete SLAT, $\mathbf z_w$, is decoded by a scene-friendly VAE decoder $\mathcal D$ into explicit scene assets: triangle meshes or 3D Gaussian splats (3DGS) with PBR textures. These outputs are compatible with physically based rendering and physics simulation (Li et al., 24 Oct 2025).

The pipeline generalizes to unbounded worlds via iterative inpainting and spatial overlap constraints, supporting arbitrary expansion directions, parallel block generation, and continuity enforcement across edges using explicit overlap loss terms.

2. Data Curation and Structured Latent Representation

GigaWorld-0-3D relies on a large-scale data curation pipeline:

• Scene Slicing: 3D spaces (e.g., from 3D-FRONT or UrbanScene3D) are covered with randomly placed, axis-aligned cuboids at two scales: coarse ($2h\times2h\times h$) and fine ($h\times h\times h$). Cuboids must meet a top-down occupancy threshold ($>95\%$) to enter the block pool.
• SLAT Encoding: For each cuboid, surface voxels $\{\mathbf p_i\}$ are determined using occupancy or TSDF zero-crossings. DINOv2 features, masked for occlusion, are aggregated into vectors $\mathbf f_i\in \mathbb R^{C_f}$ and encoded by a VAE into structured latents $\mathbf z = \{(\mathbf z_i, \mathbf p_i)\}_{i=1}^L$.

The structured 3D latent (SLAT) $\mathbf z$ thus encodes geometry via voxel coordinates $\{\mathbf p_i\}$ and appearance via feature vectors $\{\mathbf z_i\}$ with total dimension $D=L\times C$ (e.g., $L=1{,}500$, $C=256$, $D\approx 384{,}000$). This compact, spatially grounded representation is optimized for both generative modeling and downstream physical simulation (Li et al., 24 Oct 2025).

3. Scene Inpainting and Loss Objectives

Block-wise inpainting enables seamless scene extension and repair:

• Conditioning is provided by structure masks $m_s\in \{0,1\}^{N^3}$ and latent masks $m_l = \{(m_i, \mathbf p_i)\}$, indicating regions to be synthesized.
• For each block, the flow network $\mathcal G$ receives the triplet $[\ell^{(t)},\,m,\,\ell^{(0)} \odot (1-m)]$ as input.

Total inpainting loss is:

$$L = L_\mathrm{flow} + \alpha L_\mathrm{geo} + \beta L_\mathrm{app}$$

with

• Flow-matching: $L_\mathrm{flow}$ matches the denoising trajectory in feature space,
• Geometry reconstruction: $$L_\mathrm{geo} = \|S_{\rm pred} - S_{\rm gt}\|_1 + \lambda_{\rm norm} \|n_{\rm pred}-n_{\rm gt}\|_2^2$$,
• Appearance consistency: $L_\mathrm{app}$ is computed via adversarial or diffusion-based patch discrimination over rendered textures.

This multi-objective approach ensures geometric, photometric, and diffusion consistency within and across block boundaries.

4. 3D Gaussian Splatting and Scene Reconstruction

GigaWorld-0-3D extensively uses 3D Gaussian Splatting (3DGS) to represent both foreground assets and background environments:

• Representation: Each Gaussian $g_i = (\mu_i, \Sigma_i, c_i, \alpha_i)$ parameterizes spatial center, covariance, color, and opacity.
• Initialization: Back-projecting pixels from sparse images yields initial point clouds, clustered into Gaussians.
• Rendering: The elliptical weighted average (EWA) rasterization sums occlusion-sorted splats for each pixel, using per-splat opacity:

$$\alpha_i = 1-\exp\bigl(-\tau\, \exp(-\tfrac{1}{2} (u-\pi(\mu_i))^\top W_i^{-1}(u-\pi(\mu_i)))\bigr)$$

where $u$ is the image-plane coordinate, $\pi$ is projection, and $W_i$ derives from $\Sigma_i$.

• Optimization: Multi-view photometric consistency is enforced by minimizing

$$\mathcal L_\mathrm{recon} = \sum_{v}\sum_{p\in\text{pixels}} \|I_v(p)-\hat I_v(p)\|^2 + \lambda_\mathrm{TV} \sum_i \|\nabla\Sigma_i\|_F^2$$

A plausible implication is that this explicit, render-based differentiable representation supports not only visual consistency but also gradient-based optimization for downstream tasks such as policy learning and physics-driven planning (Team et al., 25 Nov 2025).

5. Physical System Identification and Executable Planning

To endow generated worlds with actionable semantics, GigaWorld-0-3D integrates physically differentiable system identification and motion planning:

• System Identification: Using real or simulated trajectories, physical parameters (mass, friction, stiffness, damping) are optimized via gradient descent. The robot arm is parameterized with standard rigid-body Euler–Lagrange dynamics, and a neural surrogate $f_\phi$ is trained to match simulator steps:

$\hat s_{t+1}=f_\phi(s_t, a_t; \theta_\mathrm{phy})$

Losses combine simulation matching and real trajectory fitting.

• Motion Planning: Collision-free, dynamically feasible joint-space trajectories $\{q_k, \tau_k\}_{k=0}^K$ are computed via direct collocation, subject to kinematic, dynamic, and collision constraints. MimicGen-based perturbations and RL policy refinement are supported for complex scenarios.

This suggests GigaWorld-0-3D is not limited to world synthesis, but is also a fully executable environment generator for embodied VLA agents—generating data that remains consistent under both visual rendering and physically plausible simulation (Team et al., 25 Nov 2025).

6. Scalability, Scheduling, and Training Protocol

The system achieves geometric and photometric continuity at arbitrary scale via explicit overlap-and-inpaint scheduling:

• Block Sizes: Coarse blocks $w^c = 8\,\mathrm m$ for room layouts, fine blocks $w^f=4\,\mathrm m$ for object-level detail.
• Overlap and Stitching: Adjacent blocks share overlap margins $[1/2 w, 7/8 w)$, and continuity is enforced by minimizing both geometry and latent feature discrepancies in overlap regions.
• Growth Direction: Worlds can be synthesized in spiral or quadrant order, always expanding from current frontiers. Parallelization exploits non-overlapping contexts.
• Training Protocol: Includes pretraining VAEs on billions of voxels, curriculum expansion (from $3\times 3$ to $20 \times 20$ grids), adversarial edge discrimination, mixed-precision, and distributed model parallelism for giga-scale latent processing.

Joint optimization combines all loss stages:

$$\mathcal L_\mathrm{total} = \lambda_\mathrm{diff} \mathcal L_\mathrm{diff} + \lambda_\mathrm{recon} \mathcal L_\mathrm{recon} + \lambda_\mathrm{phy} \mathcal L_\mathrm{phy} + \lambda_\mathrm{plan} \mathcal L_\mathrm{plan}$$

This aligns object shapes, scene layout, appearance, dynamics, and motion feasibility in a single framework.

7. Applications and Significance for Embodied AI

GigaWorld-0-3D’s unified pipeline allows for the scalable generation of visually realistic, physically valid, and instruction-aligned interaction data. It directly supports downstream Vision-Language-Action (VLA) models such as GigaBrain-0, enabling training in open-ended environments without any real-world sample collection. Evaluations indicate high data diversity, spatial and photometric coherence, physical plausibility, and controllable semantics; models trained on GigaWorld-0-3D data demonstrate superior generalization and task performance on physical robots, significantly advancing the applicability of world models as comprehensive simulators and data engines for embodied AI (Team et al., 25 Nov 2025, Li et al., 24 Oct 2025).