Diff4Splat: Feed-forward 4D Scene Synthesis

• Diff4Splat is a feed-forward method that generates dynamic 4D scenes by predicting an explicit deformable 3D Gaussian field from a single image.
• It integrates camera trajectories and optional text prompts to control photorealistic rendering without costly test-time optimization.
• The approach achieves efficient, high-quality scene synthesis with strong geometric consistency and a progressive training scheme that accelerates inference.

Diff4Splat is a feed-forward method for generating a controllable explicit 4D scene from a single image, where “4D” denotes 3D geometry plus time rather than an abstract latent index. Given a single RGB image, a camera trajectory, and an optional text prompt, it directly predicts a deformable 3D Gaussian field encoding appearance, geometry, and motion in a single forward pipeline, without test-time optimization or post-hoc refinement. Its stated goal is to bridge camera-controlled video diffusion models, which usually produce only 2D frame sequences with weak geometric grounding, and dynamic reconstruction methods, which recover explicit 4D structure but typically require expensive per-scene optimization (Pan et al., 1 Nov 2025).

1. Problem setting and formulation

Diff4Splat addresses single-image, controllable, photorealistic, dynamic scene generation with explicit geometry and motion. The method is motivated by the practical shortcomings of prior monocular dynamic scene synthesis: optimization-based systems such as CAT4D and Mosca reconstruct dynamic scenes by iteratively solving for geometry, deformation, and appearance, while some recent feed-forward alternatives output dynamic pointmaps rather than a photorealistic rendering representation. Diff4Splat is positioned as a unified feed-forward model that produces an explicit deformable 3D Gaussian scene in one forward generation process (Pan et al., 1 Nov 2025).

At inference time, the model takes three principal conditioning signals. The first is a single RGB image,

$\mathbf{I}_0 \in \mathbb{R}^{H \times W \times 3},$

which defines the scene appearance and initial content. The second is a camera trajectory over $T$ frames, encoded with Plücker embeddings,

$$\mathcal{P} \in \mathbb{R}^{T \times H \times W \times 6},$$

which gives explicit viewpoint control for trajectories such as spiral, forward, backward, upward, or downward motion. The third is an optional text prompt $\mathbf{C}_{\text{ctx}}$, used to steer appearance and content semantics. Internally, inference starts from latent noise and denoises toward a video latent tensor

$$\mathbf{z} \in \mathbb{R}^{n \times h \times w \times c}.$$

The output is an explicit deformable 3D Gaussian field rather than only a rendered video. A canonical static scene is represented as a set of Gaussian primitives

$\{ \boldsymbol{G}_p \}_{p=1}^{\mathbf{M}},$

where each primitive has mean position $\boldsymbol{\mu}_p \in \mathbb{R}^3$, scale $\boldsymbol{s}_p \in \mathbb{R}^3$, rotation quaternion $\boldsymbol{q}_p \in \mathbb{R}^4$, opacity $\alpha_p \in \mathbb{R}$, and color features $T$0. Dynamics are added through time-dependent deformations

$T$1

so that the scene is parameterized as a canonical Gaussian set plus temporal deformation (Pan et al., 1 Nov 2025).

2. Dynamic Gaussian representation

The central geometric construction in Diff4Splat is a canonical 3D Gaussian representation combined with explicit temporal warping. The spatial support of Gaussian $T$2 is

$T$3

where $T$4 is induced by the learned scale $T$5 and rotation $T$6. For time $T$7, the canonical parameters are updated as

$T$8

$T$9

$$\mathcal{P} \in \mathbb{R}^{T \times H \times W \times 6},$$0

with $$\mathcal{P} \in \mathbb{R}^{T \times H \times W \times 6},$$1 denoting quaternion multiplication. This decomposition is also the basis of the progressive training schedule: during static pretraining, the deformation branch learns approximately identity updates,

$$\mathcal{P} \in \mathbb{R}^{T \times H \times W \times 6},$$2

and only later learns nontrivial motion during dynamic fine-tuning (Pan et al., 1 Nov 2025).

The model predicts a Gaussian feature map

$$\mathcal{P} \in \mathbb{R}^{T \times H \times W \times 6},$$3

and a deformation map

$$\mathcal{P} \in \mathbb{R}^{T \times H \times W \times 6},$$4

with

$$\mathcal{P} \in \mathbb{R}^{T \times H \times W \times 6},$$5

corresponding exactly to $$\mathcal{P} \in \mathbb{R}^{T \times H \times W \times 6},$$6 mean offsets, $$\mathcal{P} \in \mathbb{R}^{T \times H \times W \times 6},$$7 quaternion channels, and $$\mathcal{P} \in \mathbb{R}^{T \times H \times W \times 6},$$8 scale offsets. A plausible implication is that Diff4Splat treats motion prediction as a low-dimensional structured correction to a canonical Gaussian state rather than as unconstrained per-frame reconstruction.

Rendered outputs are produced with a differentiable Gaussian rasterization pipeline. The paper does not print the standard alpha-compositing equations, but it specifies the usual 3DGS procedure: each deformed 3D Gaussian is projected into the target camera, converted to a screen-space ellipse using the camera transform and projected covariance, and alpha-composited in depth order to yield RGB and depth. The method’s novelty lies not in a new rasterizer, but in applying standard Gaussian splatting to time-deformed primitives so that novel-view synthesis and temporal rendering become deterministic functions of the predicted 4D field (Pan et al., 1 Nov 2025).

3. Architecture and controllability

Diff4Splat combines a pretrained latent video diffusion backbone with a reconstruction transformer called the Latent Dynamic Reconstruction Model (LDRM). The base generator is built on CogVideoX, a pretrained Video Diffusion Transformer in the latent space of a 3D Causal VAE with 32 channels and $$\mathcal{P} \in \mathbb{R}^{T \times H \times W \times 6},$$9 compression. The diffusion backbone has 32 blocks and hidden dimension 4096. Its role is to transform the single image, camera-path conditioning, text conditioning, and latent noise into a spatiotemporally coherent latent video tensor $\mathbf{C}_{\text{ctx}}$0 (Pan et al., 1 Nov 2025).

The LDRM then lifts those video latents into explicit 4D geometry. It receives the generated latent tensor and corresponding camera poses, patchifies both into token sequences of matched length, concatenates latent and pose tokens channel-wise, and processes them through 16 standard Transformer blocks. The latent features originally have channel dimension $\mathbf{C}_{\text{ctx}}$1 and are projected into a 64-dimensional embedding space before entering the Transformer. A decoding module regresses Gaussian attributes from the transformer outputs, and a 3D deconvolutional layer establishes pixel-level correspondence with source video frames. The paper also describes a separate lightweight spatiotemporal deformation network that takes a canonical latent representation and a time embedding, aggregates multi-resolution spatial features, and outputs the deformation map $\mathbf{C}_{\text{ctx}}$2, with a final $\mathbf{C}_{\text{ctx}}$3 convolution producing the $\mathbf{C}_{\text{ctx}}$4 deformation channels (Pan et al., 1 Nov 2025).

Text control is introduced through cross-attention. Each DiT block incorporates a cross-attention layer to integrate image embedding information from T5. The intended conditioning stack fuses image condition, text condition, and camera condition in the latent video generator, after which the LDRM combines the resulting video latents with camera tokens to decode canonical Gaussian geometry plus temporal deformation. This arrangement makes camera control a first-class input rather than a post-hoc rendering request. The paper emphasizes that once a coherent 4D scene exists, the final video is obtained by deterministic rendering along the requested camera trajectory rather than by asking a 2D generator to follow extrinsic controls (Pan et al., 1 Nov 2025).

4. Training objectives, schedule, and data

Training combines flow matching in video latent space with rendering-based supervision on the predicted 4D Gaussian field. Let $\mathbf{C}_{\text{ctx}}$5 be a clean latent sequence and $\mathbf{C}_{\text{ctx}}$6 Gaussian noise. Diff4Splat defines the interpolation path

$\mathbf{C}_{\text{ctx}}$7

with target vector field

$\mathbf{C}_{\text{ctx}}$8

The model $\mathbf{C}_{\text{ctx}}$9 is trained with the Flow Matching objective

$$\mathbf{z} \in \mathbb{R}^{n \times h \times w \times c}.$$0

On the rendering side, the photometric loss is

$$\mathbf{z} \in \mathbb{R}^{n \times h \times w \times c}.$$1

the geometric loss is the depth-correlation term

$$\mathbf{z} \in \mathbb{R}^{n \times h \times w \times c}.$$2

with a depth smoothness prior

$$\mathbf{z} \in \mathbb{R}^{n \times h \times w \times c}.$$3

and the motion loss uses tracked 3D points: $$\mathbf{z} \in \mathbb{R}^{n \times h \times w \times c}.$$4 The reported weights are $$\mathbf{z} \in \mathbb{R}^{n \times h \times w \times c}.$$5 and $$\mathbf{z} \in \mathbb{R}^{n \times h \times w \times c}.$$6 (Pan et al., 1 Nov 2025).

A defining aspect of Diff4Splat is its progressive schedule. Stage 1 is static geometry pretraining for 40K iterations on static scenes at $$\mathbf{z} \in \mathbb{R}^{n \times h \times w \times c}.$$7, using photometric and geometric losses only while freezing the deformation module. Stage 2 is high-resolution refinement for another 40K iterations on static scenes at $$\mathbf{z} \in \mathbb{R}^{n \times h \times w \times c}.$$8, again with deformation frozen. Stage 3 is dynamic scene fine-tuning for 20K iterations on dynamic datasets, with the full model unfrozen and the motion loss enabled. The authors state that removing this schedule destabilizes training, yields poor initialization of the static Gaussian field, and requires about 21 days instead of 7 to reach similar quality (Pan et al., 1 Nov 2025).

The training data are likewise central to the method. The authors curate about 130,000 candidate scenes and refine them to about 100,000 synchronized multi-view videos with appearance, metric point maps, and point motion trajectories. Synthetic sources are TartanAir, MatrixCity, PointOdyssey, DynamicReplica, Spring, VKITTI2, and MultiCamVideo; real-world sources are RealEstate10K and Stereo4D. For real videos, metric scale is recovered using Video Depth Anything and MegaSaM, with relative-to-metric depth alignment

$$\mathbf{z} \in \mathbb{R}^{n \times h \times w \times c}.$$9

followed by

$\{ \boldsymbol{G}_p \}_{p=1}^{\mathbf{M}},$0

This suggests that Diff4Splat’s feed-forward 4D reconstruction depends not only on architecture, but also on unusually rich supervision for geometry and motion (Pan et al., 1 Nov 2025).

5. Inference and empirical performance

Inference consists of feed-forward latent video generation followed by latent-to-Gaussian lifting. Starting from one image, a camera path, and optional text, the diffusion backbone samples a latent video sequence, and the LDRM converts those latents plus camera tokens into canonical Gaussian attributes and per-time deformation parameters. The paper reports that generating a complete 4D scene takes about 30 seconds, with interactive rendering latency of 6.7 ms once the field has been produced. In the cited comparison, AC3D + Mosca requires 45 minutes, and memory is reduced from about 80 GB to 25 GB, yielding an approximately $\{ \boldsymbol{G}_p \}_{p=1}^{\mathbf{M}},$1 speedup in the stated setting (Pan et al., 1 Nov 2025).

The reported quantitative results indicate that this efficiency is accompanied by strong generation and geometry metrics. For video generation and aesthetics, Diff4Splat achieves FVD $\{ \boldsymbol{G}_p \}_{p=1}^{\mathbf{M}},$2, KVD $\{ \boldsymbol{G}_p \}_{p=1}^{\mathbf{M}},$3, CLIP-Score $\{ \boldsymbol{G}_p \}_{p=1}^{\mathbf{M}},$4, CLIP-Aesthetic $\{ \boldsymbol{G}_p \}_{p=1}^{\mathbf{M}},$5, and QA-Quality $\{ \boldsymbol{G}_p \}_{p=1}^{\mathbf{M}},$6. Against AC3D + Mosca, the paper states that Diff4Splat improves FVD, CLIP score, and aesthetic score while reducing reconstruction time from 45 minutes to 30 seconds. For geometric integrity, it reports Avg. Matches $\{ \boldsymbol{G}_p \}_{p=1}^{\mathbf{M}},$7, Subject Consistency $\{ \boldsymbol{G}_p \}_{p=1}^{\mathbf{M}},$8, and Background Consistency $\{ \boldsymbol{G}_p \}_{p=1}^{\mathbf{M}},$9 (Pan et al., 1 Nov 2025).

The pose-control evaluation highlights a further distinction between explicit 4D fields and implicit video generation. Using Relative Pose Error, Diff4Splat reports translation RPE $\boldsymbol{\mu}_p \in \mathbb{R}^3$0 versus $\boldsymbol{\mu}_p \in \mathbb{R}^3$1 for the implicit baseline, and rotation RPE $\boldsymbol{\mu}_p \in \mathbb{R}^3$2 versus $\boldsymbol{\mu}_p \in \mathbb{R}^3$3. For full 360° loops, the appendix reports cycle-consistency of 34.5 dB PSNR between first and last frames. These numbers are presented as evidence that a rendered explicit 4D Gaussian scene follows camera trajectories more reliably than a purely implicit video model (Pan et al., 1 Nov 2025).

Ablation studies attribute performance to the explicit dynamic representation, the motion loss, and the progressive schedule. Removing the deformation Gaussian field causes ghosting, blur, and spikes under large motion. Removing the motion loss worsens FVD from $\boldsymbol{\mu}_p \in \mathbb{R}^3$4 to $\boldsymbol{\mu}_p \in \mathbb{R}^3$5, KVD from $\boldsymbol{\mu}_p \in \mathbb{R}^3$6 to $\boldsymbol{\mu}_p \in \mathbb{R}^3$7, and QA-Quality from $\boldsymbol{\mu}_p \in \mathbb{R}^3$8 to $\boldsymbol{\mu}_p \in \mathbb{R}^3$9, while also degrading geometric consistency scores. Omitting progressive training destabilizes learning and yields poorer visual quality under the same budget. This supports the paper’s claim that geometry, appearance, and motion must be predicted jointly rather than delegated to a post-hoc dynamic reconstruction stage (Pan et al., 1 Nov 2025).

6. Position in the literature and limitations

Diff4Splat occupies a distinct position among splat-based generative systems. DiffSplat generates static 3D Gaussian splats from text or a single image by repurposing pretrained image diffusion models for structured splat-grid latents, but it targets static 3D objects rather than dynamic 4D scenes (Lin et al., 28 Jan 2025). Splat4D is a diffusion-enhanced 4D Gaussian splatting pipeline in which diffusion is used to generate or refine multi-view video evidence before reconstructing 4D Gaussians, rather than directly predicting a deformable Gaussian field from a single image (Yin et al., 11 Aug 2025). FluSplat, by contrast, addresses sparse-view 3D scene editing from two images and a text instruction, using a feed-forward image-editing stage plus feed-forward 3DGS lifting, but it is a sparse-view editing method rather than a single-image 4D generation model (Huang et al., 21 Apr 2026). This suggests that Diff4Splat’s distinguishing feature is the direct fusion of video diffusion priors with explicit dynamic Gaussian reconstruction in a single feed-forward system (Pan et al., 1 Nov 2025).

The paper’s stated limitations are centered on ambiguity, extrapolation, and compute. Because the input is a single image, motion ambiguity is fundamental: multiple plausible dynamic evolutions may explain the same observation. The appendix explicitly notes that a bird might plausibly glide or flap. Rendering at novel timestamps from substantially different viewpoints can produce artifacts, since temporal deformation must be inferred and propagated under sparse observation. Generalization also degrades on out-of-distribution inputs such as unusual categories or artistic styles, where geometry and motion may become less reliable. In the conclusion, the authors further note that video generation remains the computational bottleneck: scene generation is feed-forward, but latent video denoising or sampling is still the expensive stage (Pan et al., 1 Nov 2025).

The paper does not claim to solve topology changes or very complex occlusion and disocclusion perfectly. It instead argues that future work should pursue faster denoisers or parallel inference, stronger temporal coherence, and material property prediction. A plausible implication is that Diff4Splat establishes a specific design point—explicit dynamic 3D Gaussian reconstruction from a single image under camera and text control—while leaving open the broader problem of physically grounded, topology-aware 4D scene generation (Pan et al., 1 Nov 2025).