World Volume Diffusion for Driving Scenes

• World volume diffusion is a 4D voxel-grid based framework that reconstructs and forecasts dense spatiotemporal representations for driving scenes.
• It employs a dual-phase, hierarchical diffusion pipeline with forward noising and reverse denoising, ensuring intra-world semantic consistency and inter-view coherence.
• The method integrates explicit geometric conditioning and multi-camera synthesis using advanced UNets, enhancing simulation quality for autonomous driving.

World volume diffusion is a generative modeling framework wherein a temporally-evolving, spatially explicit 4D voxel-grid—termed the "world volume"—serves as a central latent representation for diffusion-based synthesis of multi-camera videos, particularly within autonomous driving simulation contexts. Unlike prior approaches relying purely on per-image latent diffusion, world volume diffusion reconstructs and forecasts a dense spatiotemporal world model, ensuring intra-world semantic consistency, inter-view coherence, and robust temporal dynamics. This methodology was introduced by WoVoGen (“World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generation”), which leverages a dual-phase, hierarchical diffusion scheme incorporating explicit 4D geometric conditioning (Lu et al., 2023).

1. 4D World Volume Representation

At the core of world volume diffusion is the explicit representation of the environment as a sequence of 3D voxel grids over time, aggregating semantics, high-definition map (HD-map) data, and occupancy. Formally, the world volume over $T$ timesteps is defined:

$$V = \{W_1, W_2, ..., W_T\} \in \mathbb{R}^{T \times Z \times H \times W \times C}$$

For each $t$:

• $$W_t = \text{concat}(O_t, M_t) \in \mathbb{R}^{Z \times H \times W \times (C_\mathrm{occ} + C_\mathrm{map})}$$
  • $$O_t \in \mathbb{R}^{Z \times H \times W \times C_\mathrm{occ}}$$: 3D semantic occupancy (semantic classes)
  • $M_t \in \mathbb{R}^{1 \times H \times W \times 3}$: HD-map, zero-padded to $Z$ in height

To accommodate the high dimensionality for efficient modeling in the diffusion process, an autoencoder compresses $W_t$ into a lower-resolution latent $z_w^t$:

$$z_w^t = E_W(W_t) \in \mathbb{R}^{Z/s \times H/s \times W/s \times C_z}$$

with $$V = \{W_1, W_2, ..., W_T\} \in \mathbb{R}^{T \times Z \times H \times W \times C}$$0 as the downsampling factor and typically $$V = \{W_1, W_2, ..., W_T\} \in \mathbb{R}^{T \times Z \times H \times W \times C}$$1. Stacking over $$V = \{W_1, W_2, ..., W_T\} \in \mathbb{R}^{T \times Z \times H \times W \times C}$$2 gives:

$$V = \{W_1, W_2, ..., W_T\} \in \mathbb{R}^{T \times Z \times H \times W \times C}$$3

This 4D representation preserves both spatial environmental structure and temporal development, forming the substrate for latent diffusion.

2. Diffusion over the World Volume Latent

World volume diffusion follows the latent diffusion paradigm, introducing noise and learning to denoise in the compressed latent space.

2.1 Forward (Noising) Process

Each future-frame latent $$V = \{W_1, W_2, ..., W_T\} \in \mathbb{R}^{T \times Z \times H \times W \times C}$$4 undergoes additive Gaussian noise at diffusion steps $$V = \{W_1, W_2, ..., W_T\} \in \mathbb{R}^{T \times Z \times H \times W \times C}$$5:

$$V = \{W_1, W_2, ..., W_T\} \in \mathbb{R}^{T \times Z \times H \times W \times C}$$6

with fixed variance schedule $$V = \{W_1, W_2, ..., W_T\} \in \mathbb{R}^{T \times Z \times H \times W \times C}$$7. For jointly predicting $$V = \{W_1, W_2, ..., W_T\} \in \mathbb{R}^{T \times Z \times H \times W \times C}$$8 frames:

$$V = \{W_1, W_2, ..., W_T\} \in \mathbb{R}^{T \times Z \times H \times W \times C}$$9

2.2 Reverse (Denoising) Process

A parameterized reverse model (denoising UNet) $t$0 predicts and denoises latents, conditioned on past latents and vehicle control actions:

$t$1

where $t$2 aggregates the encoded past world-latent sequence and action tokens.

2.3 Score-Matching Objective

Model training minimizes the $t$3 difference between predicted and actual noise:

$t$4

Maximizing this objective is equivalent to maximizing a variational lower bound on $t$5.

3. Two-Phase Hierarchical Generation Pipeline

The world volume diffusion process is integrated into a two-stage generative architecture.

Phase I: World Volume Forecasting

Inputs: recent world volumes $t$6 and vehicle control token sequence $t$7, with $t$8 (velocity) and $t$9 (steering). Each $$W_t = \text{concat}(O_t, M_t) \in \mathbb{R}^{Z \times H \times W \times (C_\mathrm{occ} + C_\mathrm{map})}$$0 is encoded by $$W_t = \text{concat}(O_t, M_t) \in \mathbb{R}^{Z \times H \times W \times (C_\mathrm{occ} + C_\mathrm{map})}$$1; past latents are concatenated for temporal context. Denoising is conducted by a diffusion UNet comprising:

• Spatial MHSA (Multi-Head Self-Attention) over $$W_t = \text{concat}(O_t, M_t) \in \mathbb{R}^{Z \times H \times W \times (C_\mathrm{occ} + C_\mathrm{map})}$$2
• Temporal MHSA over $$W_t = \text{concat}(O_t, M_t) \in \mathbb{R}^{Z \times H \times W \times (C_\mathrm{occ} + C_\mathrm{map})}$$3 frames
• Cross-attention with action tokens (Fourier-embedded via a small Transformer)
• Per-block sequence: group norm—activation—conv/attention—FFN

The output is decoded to predicted future world volumes $$W_t = \text{concat}(O_t, M_t) \in \mathbb{R}^{Z \times H \times W \times (C_\mathrm{occ} + C_\mathrm{map})}$$4.

Losses include:

• $$W_t = \text{concat}(O_t, M_t) \in \mathbb{R}^{Z \times H \times W \times (C_\mathrm{occ} + C_\mathrm{map})}$$5 VQ regularization
• $$W_t = \text{concat}(O_t, M_t) \in \mathbb{R}^{Z \times H \times W \times (C_\mathrm{occ} + C_\mathrm{map})}$$6 from score-matching as above

Phase II: Multi-Camera Video Synthesis

Given predicted $$W_t = \text{concat}(O_t, M_t) \in \mathbb{R}^{Z \times H \times W \times (C_\mathrm{occ} + C_\mathrm{map})}$$7, the pipeline proceeds as follows:

• World-volume encoding: $$W_t = \text{concat}(O_t, M_t) \in \mathbb{R}^{Z \times H \times W \times (C_\mathrm{occ} + C_\mathrm{map})}$$8 yielding a sparse 4D feature grid
• Camera-frustum sampling: Each camera $$W_t = \text{concat}(O_t, M_t) \in \mathbb{R}^{Z \times H \times W \times (C_\mathrm{occ} + C_\mathrm{map})}$$9, time $$O_t \in \mathbb{R}^{Z \times H \times W \times C_\mathrm{occ}}$$0 uses the grid $$O_t \in \mathbb{R}^{Z \times H \times W \times C_\mathrm{occ}}$$1 to interpolate $$O_t \in \mathbb{R}^{Z \times H \times W \times C_\mathrm{occ}}$$2 into per-camera 3D grids $$O_t \in \mathbb{R}^{Z \times H \times W \times C_\mathrm{occ}}$$3 and, after squeeze-and-excitation and depth summation, $$O_t \in \mathbb{R}^{Z \times H \times W \times C_\mathrm{occ}}$$4 (2D features)
• Panoptic concatenation: Six surround-camera $$O_t \in \mathbb{R}^{Z \times H \times W \times C_\mathrm{occ}}$$5 are concatenated into a meta-image $$O_t \in \mathbb{R}^{Z \times H \times W \times C_\mathrm{occ}}$$6
• Latent diffusion: A ControlNet-style UNet denoises Gaussian latents $$O_t \in \mathbb{R}^{Z \times H \times W \times C_\mathrm{occ}}$$7, conditioned on $$O_t \in \mathbb{R}^{Z \times H \times W \times C_\mathrm{occ}}$$8, CLIP text prompt, and per-pixel object masks (provided by occupancy projections)
• Temporal fine-tuning: Video generation is improved via an additional temporal MHSA block trained on multi-frame losses

Overall, this pipeline enforces consistency across views and time through the explicit shared world volume.

4. Network Architectures and Feature Fusion Strategies

World volume diffusion employs two main UNet backbones, each with architectural innovations for 4D latent and multi-camera context.

• World-volume diffusion UNet ($$O_t \in \mathbb{R}^{Z \times H \times W \times C_\mathrm{occ}}$$9): Derived from Stable Diffusion; ResBlocks and self-attention replaced by sequence: spatial MHSA, temporal MHSA, cross-attention (action keys/values), and FFN. Past/future latents are channel-wise concatenated before every noise-prediction layer.
• Image-latent diffusion UNet ($M_t \in \mathbb{R}^{1 \times H \times W \times 3}$0): Follows ControlNet; at each cross-attention layer, $M_t \in \mathbb{R}^{1 \times H \times W \times 3}$1 is injected via $M_t \in \mathbb{R}^{1 \times H \times W \times 3}$2 convolution. Scene guidance is provided by mapping CLIP text vectors into cross-attention queries; object guidance uses MHCA with per-class occupancy masks and their CLIP embeddings. For video, a temporal attention branch is introduced, mirroring the architecture of the world-volume UNet.

Feature fusion is systematically executed, notably by summing $M_t \in \mathbb{R}^{1 \times H \times W \times 3}$3 into early and middle UNet layers and using cross-attention for action conditioning.

5. Explicit Volumetric Conditioning and Consistency

• Intra-world consistency: By forecasting an explicit 4D world volume—spanning semantics, map structure, and occupancy—the model accurately preserves geometric and semantic relationships. This serves as a strong geometric prior, preventing per-camera hallucinations and semantic drift.
• Inter-sensor coherence: View-specific features are sampled consistently from the shared world volume, ensuring that all camera frusta observe mutually consistent projections of the underlying 3D scene. Concatenating all view features into a panoptic latent enforces optimization towards a single, tightly coupled multi-view result.
• Temporal consistency: Temporal MHSA blocks in both the world volume and image branches smooth changes across time. The synthesis of a temporally coherent 3D structure in the latent world-volume provides superior conditioning to downstream video UNets, surpassing pairwise or ad-hoc temporal regularizations seen in standard video diffusion architectures.

6. Training and Inference Procedures

The learning and sampling workflow is organized as follows:

Algorithm A: Train World-Volume Diffusion

$M_t \in \mathbb{R}^{1 \times H \times W \times 3}$4

Algorithm B: Train Image-Latent Diffusion

$M_t \in \mathbb{R}^{1 \times H \times W \times 3}$5

Algorithm C: Inference

$M_t \in \mathbb{R}^{1 \times H \times W \times 3}$6

7. Significance and Application Domains

World volume diffusion, as instantiated in WoVoGen, establishes a foundation for controlled multi-camera driving scene generation that augments autonomous driving datasets with realistic, coherent, and controllable simulated sensor data. The explicit modeling of the world as a latent 4D voxel grid supports advanced simulation, dataset synthesis, and scene editing via action-conditioned generation, robustly addressing limitations of previous rendering- or image-based approaches in terms of spatial-temporal consistency and cross-sensor coherence (Lu et al., 2023).