WorldSplat: 4D Driving Scene Generation

• WorldSplat is a Gaussian-centric framework that explicitly represents dynamic 4D driving scenes using a multi-track, multi-view synthesis approach.
• It combines a 4D-aware latent diffusion model with a latent Gaussian decoder and enhanced video diffusion to achieve efficient, feed-forward scene generation.
• The system outperforms traditional methods in novel-view synthesis and supports scalable data generation for autonomous driving simulations.

Searching arXiv for WorldSplat and closely related papers to ground the article. WorldSplat is a Gaussian-centric feed-forward framework for 4D driving-scene generation that produces an explicit dynamic 4D Gaussian scene representation and then uses that representation for multi-track, multi-view novel-view synthesis; in broader splat literature, the term can also denote a splat-based world/scene representation system in which geometry, radiance, dynamics, and related fields are encoded as mixtures of spatial splats (Zhu et al., 27 Sep 2025, Daniels et al., 18 Nov 2025). The concrete WorldSplat system addresses a specific gap between generative video models, which work purely in the 2D image domain and have limited 3D consistency and sparse viewpoint coverage, and reconstruction-centric 3D/4D methods, which achieve strong novel-view synthesis from real driving logs but inherently lack generative capabilities (Zhu et al., 27 Sep 2025).

1. Problem setting and scope

WorldSplat targets a central tension in driving world models. Existing generation methods synthesize diverse and high-fidelity driving videos, often conditioned on BEV layouts, trajectories, and text, but they work purely in the 2D image domain and therefore struggle to support convenient and high-quality novel-view synthesis. Recent 3D/4D reconstruction approaches substantially improve novel-view synthesis for real driving scenes, yet they are reconstruction-centric, require per-scene optimization, and are expensive to run for large-scale data generation (Zhu et al., 27 Sep 2025).

The framework is designed for a setting in which scalable, controllable data are needed for perception and planning, while maintaining high-quality novel-view synthesis across time. In the formulation used by WorldSplat, “multi-track” refers to rendering the same scene along multiple perturbed ego trajectories, while “multi-view” refers to simultaneous consistency across multiple camera views. The decisive distinction is that the system does not stop at video generation: the generative model outputs a full 4D Gaussian field in a feed-forward way, rather than only 2D video frames (Zhu et al., 27 Sep 2025).

A common misconception is that WorldSplat is simply a diffusion-based driving-video generator. Its explicit target is instead a dynamic 4D Gaussian world that can be queried at arbitrary camera poses and times, making it closer to a simulator-oriented world representation than to a purely image-domain generator. This suggests that its primary novelty lies in coupling controllable generation with explicit scene structure, rather than in video synthesis alone.

2. System architecture and explicit 4D representation

WorldSplat consists of three independently trained modules: a 4D-aware latent diffusion model, a latent 4D Gaussians decoder, and an enhanced video diffusion model (Zhu et al., 27 Sep 2025).

Module	Inputs	Function
4D-aware latent diffusion model	noise latent, BEV layout, 3D boxes, ego trajectory, text	produces a multi-modal latent tensor
Latent 4D Gaussians decoder	denoised latent, ray map	predicts pixel-aligned 3D Gaussians and aggregates them into a 4D representation
Enhanced video diffusion model	rendered RGB videos, same conditions	refines rendered videos in latent space

The first module outputs a multi-view, multi-frame latent containing an RGB latent, a metric depth latent, and a semantic/dynamic mask latent. The second module converts this latent into pixel-aligned 3D Gaussians for each frame and view, separated into static background Gaussians and dynamic object Gaussians, and aggregates them into a unified 4D Gaussian representation. The third module takes rendered RGB videos from these Gaussians together with the same control signals and refines them to address artifacts from Gaussian splatting, including unseen regions and motion blur for large pose shifts (Zhu et al., 27 Sep 2025).

The 4D Gaussian representation is written as

$$\mathbf{G}_{4D} = \bigl\{\, (\mathbf{G}_t \odot \mathbf{M}_t) \;\cup\; \bigcup_{i=1}^{T}\bigl(\mathbf{G}_i \odot (1 - \mathbf{M}_i)\bigr) \bigr\}_{t=1}^{T}.$$

Here, the background is effectively static, accumulated across all frames, while the foreground remains time-varying through the current-frame dynamic component. This decomposition is central to the system’s ability to support novel-track rendering from arbitrary user-defined trajectories (Zhu et al., 27 Sep 2025).

“Feed-forward” in this context means that, given conditions and noise, the system produces 4D Gaussians and rendered videos in one pass per sample, with a small number of diffusion time steps under rectified flow, and without iterative optimization or per-scene fitting. That property distinguishes WorldSplat from scene-specific reconstruction pipelines (Zhu et al., 27 Sep 2025).

3. 4D-aware latent diffusion and multi-modal control

Given a $K$-view driving clip with $T$ frames $\mathcal{I} = \{\mathbf{I}^v_t\}$, WorldSplat first constructs a multi-modal latent. The image latent is encoded using a pretrained video VAE: $$\mathbf{L}_{\mathrm{img}} = \mathcal{E}(\mathcal{I}).$$ Metric depth maps are predicted by Metric3D-v2, normalized to $[-1,1]$, broadcast to 3 channels, and VAE-encoded into $\mathbf{L}_{\mathrm{depth}}$. Dynamic masks are produced by SegFormer and encoded into $\mathbf{L}_{\mathrm{seg}}$. These components are concatenated channel-wise: $$\mathbf{L} = \text{concate}\{\mathbf{L}_{\mathrm{img}}, \mathbf{L}_{\mathrm{depth}}, \mathbf{L}_{\mathrm{seg}}\}.$$ The inclusion of depth and segmentation latents means that the diffusion target carries not only appearance but also approximate geometry and dynamic/static structure (Zhu et al., 27 Sep 2025).

The control signal set is

$$\mathcal{C} = \{\mathcal{S}, \mathcal{B}, \mathcal{T}, \mathcal{D}\},$$

where $K$0 denotes BEV layout sketches, $K$1 3D instance boxes, $K$2 ego trajectory, and $K$3 a structured textual description generated by a DataCrafter pipeline using Qwen2-VL. These controls are encoded respectively by a sketch VAE, small CNN/MLP box encoders, an MLP over the ego-pose sequence, and a T5 encoder, then fused through cross-attention into a unified control representation injected through ControlNet blocks (Zhu et al., 27 Sep 2025).

The backbone is a Diffusion Transformer similar to OpenSora, with a main branch operating on the multi-view, multi-frame latent and a control branch processing the conditioning signals. Multi-view coherence is enforced through cross-view attention: an input tensor of shape

$K$4

is reshaped to

$K$5

so features from different views attend to one another at each time step. Temporal attention across frames provides fully 4D interactions (Zhu et al., 27 Sep 2025).

Instead of DDPM, the system uses Rectified Flow. With clean latent $K$6, Gaussian noise $K$7, and continuous mixing parameter $K$8, the interpolated state is

$K$9

The network $T$0 is trained to predict $T$1 via

$T$2

At inference, deterministic integration is performed through

$T$3

The final $T$4 is the denoised latent $T$5, which is sent directly to the Gaussian decoder rather than being decoded back to pixels by the VAE (Zhu et al., 27 Sep 2025).

4. Pixel-aligned Gaussian decoding, aggregation, and supervision

A single 3D Gaussian primitive in WorldSplat is parameterized as

$T$6

with center position $T$7, quaternion rotation $T$8, scale $T$9, opacity $\mathcal{I} = \{\mathbf{I}^v_t\}$0, and RGB color $\mathcal{I} = \{\mathbf{I}^v_t\}$1 (Zhu et al., 27 Sep 2025).

The decoder is pixel-aligned. For each pixel in each view and frame, it predicts a Gaussian placed near the corresponding camera ray using a Plücker ray map $\mathcal{I} = \{\mathbf{I}^v_t\}$2 that encodes ray origin $\mathcal{I} = \{\mathbf{I}^v_t\}$3 and direction $\mathcal{I} = \{\mathbf{I}^v_t\}$4. The predicted outputs are per-pixel depth $\mathcal{I} = \{\mathbf{I}^v_t\}$5, offset $\mathcal{I} = \{\mathbf{I}^v_t\}$6, rotation $\mathcal{I} = \{\mathbf{I}^v_t\}$7, scale $\mathcal{I} = \{\mathbf{I}^v_t\}$8, opacity $\mathcal{I} = \{\mathbf{I}^v_t\}$9, color $$\mathbf{L}_{\mathrm{img}} = \mathcal{E}(\mathcal{I}).$$0, and dynamic/static mask logits $$\mathbf{L}_{\mathrm{img}} = \mathcal{E}(\mathcal{I}).$$1. The Gaussian center is

$$\mathbf{L}_{\mathrm{img}} = \mathcal{E}(\mathcal{I}).$$2

This construction enforces tight alignment with the rendered image while still allowing learned geometric correction through the offset term (Zhu et al., 27 Sep 2025).

Architecturally, the decoder is transformer-based, inspired by ViT, STORM, and GS-LRM, with cross-view attention, temporal attention, and hierarchical upsampling from latent resolution to pixel-level predictions. Its output per time step is

$$\mathbf{L}_{\mathrm{img}} = \mathcal{E}(\mathcal{I}).$$3

The decoder supports up to 48 views simultaneously (Zhu et al., 27 Sep 2025).

Training uses real nuScenes sequences with RGB multi-view images, metric depth from Metric3D-v2, and dynamic masks from SegFormer. For supervision, the model selects a base timestep, samples $$\mathbf{L}_{\mathrm{img}} = \mathcal{E}(\mathcal{I}).$$4 target timesteps, encodes the corresponding clean latent, predicts a 4D Gaussian set, renders RGB and depth images by Gaussian splatting, and applies RGB $$\mathbf{L}_{\mathrm{img}} = \mathcal{E}(\mathcal{I}).$$5, LPIPS, metric-depth $$\mathbf{L}_{\mathrm{img}} = \mathcal{E}(\mathcal{I}).$$6, and segmentation BCE losses. The total loss is

$$\mathbf{L}_{\mathrm{img}} = \mathcal{E}(\mathcal{I}).$$7

The primary dataset is nuScenes, with 1,000 urban driving scenes captured by 6 cameras, 700 scenes for training and 150 for validation, using annotations upsampled from 2 Hz to 12 Hz following ASAP. Diffusion training is performed on 32 NVIDIA H20 GPUs in a four-stage schedule, beginning from OpenSora v1.2 checkpoints and ending with rectified-flow training up to 480p and full scale (Zhu et al., 27 Sep 2025).

5. Rendering, empirical performance, and computational profile

Rendering follows the standard Gaussian splatting pipeline: Gaussians are projected into the image plane, anisotropic covariance determines their footprint, and colors and opacities are composited front-to-back. Because the scene is explicit in world coordinates, any ego trajectory $$\mathbf{L}_{\mathrm{img}} = \mathcal{E}(\mathcal{I}).$$8 can be used to render multi-view images at each time step. The evaluation protocol includes lateral trajectory shifts

$$\mathbf{L}_{\mathrm{img}} = \mathcal{E}(\mathcal{I}).$$9

following FreeVS (Zhu et al., 27 Sep 2025).

On original-view video generation, the reported metrics are Fréchet Video Distance and Fréchet Inception Distance over multi-view clips. Without first-frame guidance, WorldSplat reports $[-1,1]$0 and $[-1,1]$1 on 5,369 samples, outperforming DriveDreamer-2, MagicDrive-V2, and Panacea in the cited comparison. With first-frame guidance, it reports FVD $[-1,1]$2 and FID $[-1,1]$3. Under noisy latent initialization on 6,019 samples, it reports FVD $[-1,1]$4 and FID $[-1,1]$5, giving the best FVD and near-best FID against UniScene in that setting (Zhu et al., 27 Sep 2025).

For novel-view synthesis, WorldSplat is evaluated against DiST-4D and several reconstruction baselines. At $[-1,1]$6 m shift it reports FID $[-1,1]$7 and FVD $[-1,1]$8, versus DiST-4D at FID $[-1,1]$9 and FVD $\mathbf{L}_{\mathrm{depth}}$0. At $\mathbf{L}_{\mathrm{depth}}$1 m, it reports FID $\mathbf{L}_{\mathrm{depth}}$2 and FVD $\mathbf{L}_{\mathrm{depth}}$3, versus DiST-4D at FID $\mathbf{L}_{\mathrm{depth}}$4 and FVD $\mathbf{L}_{\mathrm{depth}}$5. At $\mathbf{L}_{\mathrm{depth}}$6 m, it reports FID $\mathbf{L}_{\mathrm{depth}}$7 and FVD $\mathbf{L}_{\mathrm{depth}}$8, versus DiST-4D at FID $\mathbf{L}_{\mathrm{depth}}$9 and FVD $\mathbf{L}_{\mathrm{seg}}$0. The paper attributes the margin, especially at larger shifts, to the robustness of the explicit 4D Gaussian representation (Zhu et al., 27 Sep 2025).

Ablation results isolate the roles of 4D aggregation and diffusion refinement. Version A, which uses only reprojected boxes and sketches and no Gaussian rendering, gives FVD $\mathbf{L}_{\mathrm{seg}}$1 and FID $\mathbf{L}_{\mathrm{seg}}$2. Version B, using 3D Gaussians per frame only, gives FVD $\mathbf{L}_{\mathrm{seg}}$3 and FID $\mathbf{L}_{\mathrm{seg}}$4. Version C, adding 4D aggregation but no enhanced diffusion, gives FVD $\mathbf{L}_{\mathrm{seg}}$5 and FID $\mathbf{L}_{\mathrm{seg}}$6. Version D, adding enhanced diffusion without mixed augmentation, degrades to FVD $\mathbf{L}_{\mathrm{seg}}$7 and FID $\mathbf{L}_{\mathrm{seg}}$8. Version E, the full system with 4D Gaussians, enhanced diffusion, and mixed augmentation, achieves FVD $\mathbf{L}_{\mathrm{seg}}$9 and FID $$\mathbf{L} = \text{concate}\{\mathbf{L}_{\mathrm{img}}, \mathbf{L}_{\mathrm{depth}}, \mathbf{L}_{\mathrm{seg}}\}.$$0 (Zhu et al., 27 Sep 2025).

The framework is also evaluated on downstream autonomous-driving tasks. Using generated videos as inputs to a pretrained BEVFormer, WorldSplat yields mIoU $$\mathbf{L} = \text{concate}\{\mathbf{L}_{\mathrm{img}}, \mathbf{L}_{\mathrm{depth}}, \mathbf{L}_{\mathrm{seg}}\}.$$1 and mAP $$\mathbf{L} = \text{concate}\{\mathbf{L}_{\mathrm{img}}, \mathbf{L}_{\mathrm{depth}}, \mathbf{L}_{\mathrm{seg}}\}.$$2, compared with MagicDrive at mIoU $$\mathbf{L} = \text{concate}\{\mathbf{L}_{\mathrm{img}}, \mathbf{L}_{\mathrm{depth}}, \mathbf{L}_{\mathrm{seg}}\}.$$3, mAP $$\mathbf{L} = \text{concate}\{\mathbf{L}_{\mathrm{img}}, \mathbf{L}_{\mathrm{depth}}, \mathbf{L}_{\mathrm{seg}}\}.$$4, MagicDrive-V2 at mIoU $$\mathbf{L} = \text{concate}\{\mathbf{L}_{\mathrm{img}}, \mathbf{L}_{\mathrm{depth}}, \mathbf{L}_{\mathrm{seg}}\}.$$5, mAP $$\mathbf{L} = \text{concate}\{\mathbf{L}_{\mathrm{img}}, \mathbf{L}_{\mathrm{depth}}, \mathbf{L}_{\mathrm{seg}}\}.$$6, and DiVE at mIoU $$\mathbf{L} = \text{concate}\{\mathbf{L}_{\mathrm{img}}, \mathbf{L}_{\mathrm{depth}}, \mathbf{L}_{\mathrm{seg}}\}.$$7, mAP $$\mathbf{L} = \text{concate}\{\mathbf{L}_{\mathrm{img}}, \mathbf{L}_{\mathrm{depth}}, \mathbf{L}_{\mathrm{seg}}\}.$$8. In data augmentation for StreamPETR, training on real data plus WorldSplat-generated samples improves performance from mAP $$\mathbf{L} = \text{concate}\{\mathbf{L}_{\mathrm{img}}, \mathbf{L}_{\mathrm{depth}}, \mathbf{L}_{\mathrm{seg}}\}.$$9, NDS $$\mathcal{C} = \{\mathcal{S}, \mathcal{B}, \mathcal{T}, \mathcal{D}\},$$0 to mAP $$\mathcal{C} = \{\mathcal{S}, \mathcal{B}, \mathcal{T}, \mathcal{D}\},$$1, NDS $$\mathcal{C} = \{\mathcal{S}, \mathcal{B}, \mathcal{T}, \mathcal{D}\},$$2, exceeding the gains reported for Panacea-generated data (Zhu et al., 27 Sep 2025).

In computational terms, for 17 frames, 6 views, and $$\mathcal{C} = \{\mathcal{S}, \mathcal{B}, \mathcal{T}, \mathcal{D}\},$$3 resolution, the reported timings are 66.56 s for the 4D-aware diffusion, 0.84 s for the Gaussian decoder, 66.35 s for the enhanced diffusion, and 16.10 s for VAE decoding, for a total of 2.50 min using 22 GB GPU memory. The decoder alone therefore contributes a comparatively small fraction of the inference cost, and the diffusion models use rectified flow with 8 sampling steps rather than the 30+ steps typical of DDPM or VDM (Zhu et al., 27 Sep 2025).

6. Ecosystem position, theoretical framing, and limitations

WorldSplat is situated inside the broader 3D Gaussian Splatting ecosystem of explicit Gaussian primitives, standardized binary formats, fast GPU rasterizers, and scalable asset management. This broader positioning is made explicit by "Splats in Splats++," which is designed to sit inside exactly that ecosystem and treats a trained 3DGS model as the canonical asset while preserving usability in any standard 3DGS pipeline, including WorldSplat, GSplat, and the original Kerbl implementation (Guo et al., 17 Apr 2026). In that sense, WorldSplat is not only a generative method for autonomous driving but also an example of a WorldSplat-like system: a portable Gaussian asset pipeline in which splat representations can be serialized, streamed, pruned, compressed, and reused by downstream tools.

The theoretical generalization of this viewpoint is provided by "Splat Regression Models." There, a world or scene is represented through a mixing measure $$\mathcal{C} = \{\mathcal{S}, \mathcal{B}, \mathcal{T}, \mathcal{D}\},$$4 over output vectors and affine images of a mother splat $$\mathcal{C} = \{\mathcal{S}, \mathcal{B}, \mathcal{T}, \mathcal{D}\},$$5, yielding the function

$$\mathcal{C} = \{\mathcal{S}, \mathcal{B}, \mathcal{T}, \mathcal{D}\},$$6

For finitely supported measures,

$$\mathcal{C} = \{\mathcal{S}, \mathcal{B}, \mathcal{T}, \mathcal{D}\},$$7

With a Gaussian mother splat, this recovers Gaussian Splatting as a special case and separates model, inverse problem, and optimization algorithm through a Wasserstein–Fisher–Rao formulation (Daniels et al., 18 Nov 2025). A plausible implication is that WorldSplat’s explicit 4D Gaussian world can be interpreted not merely as a graphics data structure, but as a specific instance of a broader class of splat-based world models.

The current limitations of the concrete WorldSplat system remain those identified in its driving-scene formulation. Reliance on 2D Gaussian blending can still cause blur or artifacts in extreme pose changes or very sparse observations. Complex geometry, including thin structures and reflective surfaces, remains challenging. The method depends on external depth and segmentation estimators—Metric3D-v2 and SegFormer—so their errors propagate into the 4D Gaussians. Training is performed on nuScenes only, making generalization to different cities or conditions contingent on further data and domain adaptation. Although the representation is 4D, extremely long sequences or continual world evolution over minutes are not fully tackled (Zhu et al., 27 Sep 2025).

Taken together, these strands define WorldSplat at two levels. In the narrow sense, it is a feed-forward 4D driving-scene generation framework whose output is an explicit, controllable Gaussian world for novel-view rendering. In the broader sense suggested by adjacent literature, it exemplifies a splat-based world representation paradigm in which explicit Gaussian assets, differentiable rendering, control-conditioned generation, and even downstream protection mechanisms such as 3DGS steganography can coexist within the same native pipeline (Zhu et al., 27 Sep 2025, Guo et al., 17 Apr 2026).