Video Diffusion-Enhanced 4D Gaussian Splatting

• The paper introduces a method that combines video diffusion’s motion priors and texture enhancement with explicit dynamic 4D Gaussian representations to generate, reconstruct, and regularize evolving 3D scenes.
• It employs diverse strategies—from HexPlane deformation to per-attribute latent variation—to model temporal dynamics and render high-fidelity visuals via efficient differentiable rasterization.
• Empirical results highlight improvements in SSIM, PSNR, LPIPS, and runtime across applications such as AR/VR, human reconstruction, and volumetric capture.

Video diffusion-enhanced 4D Gaussian splatting denotes a class of methods that couple video diffusion models with explicit dynamic Gaussian representations in order to generate, reconstruct, regularize, or refine temporally evolving 3D content. In these systems, diffusion typically provides motion priors, dense multi-view supervision, masked inpainting, timestamp-interpolated pseudo-views, or post-render detail enhancement, while 4D Gaussian splatting provides an explicit representation of geometry and appearance together with efficient differentiable rasterization. Representative formulations include DreamGaussian4D’s image-to-4D Gaussian Splatting plus video-to-video texture refinement (Ren et al., 2023), Diffusion4D’s 4D-aware orbital video synthesis followed by coarse-to-fine Gaussian fitting (Liang et al., 2024), cascaded dense-view generation in 4DVD (Yang et al., 6 Aug 2025), uncertainty-aware refinement in Splat4D (Yin et al., 11 Aug 2025), timestamp-aligned pseudo-view distillation in VDEGaussian (Xiao et al., 4 Aug 2025), latent direct video-to-4D synthesis in Gaussian Variation Field Diffusion (Zhang et al., 31 Jul 2025), camera-conditioned dense multi-view expansion for human reconstruction in Flex4DHuman (Cheng et al., 11 Jun 2026), and diffusion-based closeup enhancement on top of dynamic Gaussian capture pipelines (Philip et al., 31 Oct 2025).

1. Conceptual scope and methodological lineage

The field emerged from a broader shift away from purely implicit dynamic radiance fields and toward explicit dynamic Gaussian representations. A central motivation was that prior 4D NeRF- or SDS-driven systems often required long optimization schedules and exhibited weak controllability or multi-view inconsistency. DreamGaussian4D formalized one influential response: retain a high-quality static 3D Gaussian Splatting representation, model dynamics as explicit spatial transformations over time, and use a pre-trained image-to-video diffusion model both as a motion source and as a video-to-video texture prior (Ren et al., 2023). Diffusion4D pushed this logic further by training a 4D-aware video diffusion model to synthesize orbital views of dynamic assets, then fitting an explicit 4D Gaussian representation in a coarse-to-fine manner (Liang et al., 2024).

Subsequent work diversified the role of video diffusion rather than standardizing it. PLA4D argued that competing video- and multiview-diffusion SDS priors create motion-geometry conflicts and instead used a text-generated video as a pixel-level anchor, reserving SDS for a narrower pose-conditional refinement step (Miao et al., 2024). 4DVD decoupled dense-view video generation into coarse layout synthesis and structure-aware appearance refinement, explicitly motivated by the cost and fragility of directly modeling the full view-time grid (Yang et al., 6 Aug 2025). Splat4D treated video diffusion as an uncertainty-guided inpainting prior within an iterative render-mask-refine-update loop (Yin et al., 11 Aug 2025), while VDEGaussian used a test-time adapted video diffusion model as a source of temporally consistent pseudo mid-frames for self-supervised dynamic urban scene modeling (Xiao et al., 4 Aug 2025).

A second line of development positioned video diffusion not merely as a regularizer but as a front-end that makes downstream 4DGS well-posed. Flex4DHuman synthesized synchronized dense multi-view videos from monocular or sparse-view human footage using only relative camera-pose conditioning and optional text prompts, then passed the generated views into FreeTimeGS (Cheng et al., 11 Jun 2026). Gaussian Variation Field Diffusion went further by learning a latent variable field of temporal Gaussian attribute deltas and diffusing directly in that compact latent space, thereby avoiding per-instance 4DGS fitting during training (Zhang et al., 31 Jul 2025). In production-oriented systems, diffusion appears again after reconstruction: detail-enhanced Gaussian splatting for large-scale volumetric capture reconstructed dynamic Gaussian content first and then fine-tuned an actor-specific diffusion model for 4K facial closeups (Philip et al., 31 Oct 2025).

2. Gaussian representations, temporal parameterizations, and rendering

Across this literature, the underlying representation remains an explicit set of Gaussian primitives with time-dependent attributes. In DreamGaussian4D, a scene is a set of $N$ Gaussians $S = \{g_i\}_{i=1..N}$, each parameterized by mean $\mu_i \in \mathbb{R}^3$, covariance $\Sigma_i \in \mathbb{R}^{3 \times 3}$, opacity $\alpha_i \in (0,1)$, and color $c_i \in \mathbb{R}^3$. Rendering follows ordered alpha blending of projected Gaussians,

$$C(u) = \sum_{i \in S(u)} T_i \alpha_i c_i, \qquad T_i = \prod_{j < i} (1 - \alpha_j),$$

with depth ordering over the set of Gaussians intersecting pixel $u$ (Ren et al., 2023). Flex4DHuman, Splat4D, VDEGaussian, and related systems use the same general rasterization principle, sometimes with view-dependent color represented by spherical harmonics and sometimes with view-independent RGB (Cheng et al., 11 Jun 2026).

What varies most is the temporal parameterization. DreamGaussian4D deforms a canonical Gaussian by explicit translation, rotation, and scale over time, with

$$\mu_i(\tau) = R_i(\tau)\mu_i + d_i(\tau), \qquad \Sigma_i(\tau) = A_i(\tau)\Sigma_iA_i(\tau)^\top,$$

where a HexPlane-conditioned MLP predicts $\{d_i(\tau), R_i(\tau), S_i(\tau)\}$ (Ren et al., 2023). Gaussian Variation Field Diffusion also starts from a canonical Gaussian set $S = \{g_i\}_{i=1..N}$0, but expresses dynamics as full per-attribute deltas,

$S = \{g_i\}_{i=1..N}$1

with variations in position, scale, rotation, color, and opacity decoded from a compact latent learned from animation data (Zhang et al., 31 Jul 2025). By contrast, Splat4D uses per-frame Gaussian sets rather than an explicit global deformation field, and enforces temporal coherence through uncertainty-guided diffusion enhancement and masked photometric optimization across the sequence (Yin et al., 11 Aug 2025).

Urban-scene and production systems adopt yet other motion models. VDEGaussian inherits the Periodic Vibration Gaussian formulation, with oscillatory mean and time-varying opacity governed by a velocity vector, a peak time, a cycle length, and a lifespan parameter (Xiao et al., 4 Aug 2025). Detail Enhanced Gaussian Splatting for large-scale volumetric capture uses Poly4DGS, where position, rotation, scale, and opacity evolve as low-order polynomials in time; empirically, the paper reports $S = \{g_i\}_{i=1..N}$2, $S = \{g_i\}_{i=1..N}$3, $S = \{g_i\}_{i=1..N}$4, and $S = \{g_i\}_{i=1..N}$5 (Philip et al., 31 Oct 2025). A plausible implication is that video diffusion does not prescribe any single 4DGS kinematic model; it interfaces with a spectrum ranging from canonical deformation fields to per-frame splats and polynomial motion models.

3. Modes of video-diffusion integration

The defining feature of the area is not the Gaussian representation itself but where video diffusion enters the pipeline. Recent methods distribute that role very differently.

Method family	Diffusion role	4DGS coupling
DreamGaussian4D	Driving-video generation and V2V UV refinement	HexPlane deformation of static GS
Diffusion4D / 4DVD / Flex4DHuman	Dense multi-view video synthesis	4DGS fit to synthesized views
Splat4D / VDEGaussian	Inpainting or pseudo-view prior during optimization	Iterative Gaussian update with masks or uncertainty
Gaussian Variation Field Diffusion	Direct latent generation of temporal Gaussian variations	Decoder outputs animated GS
Detail Enhanced GS	Post-render RGBA detail enhancement	Enhances rendered dynamic GS closeups

DreamGaussian4D is exemplary of explicit supervision. It first initializes static GS with DreamGaussianHD, then learns HexPlane-based Gaussian deformation under a photometric loss against a 14-frame driving video generated by a pre-trained image-to-video diffusion model, with SDS from a 3D-aware image diffusion model used only to propagate motion to unseen regions. After mesh extraction, it performs video-to-video UV refinement by denoising a rendered sequence with a pre-trained image-to-video diffusion model and backpropagating a reconstruction loss into UV textures (Ren et al., 2023). The diffusion model is therefore not the 4D representation; it is a motion supervisor and a temporal texture prior.

Diffusion4D, 4DVD, and Flex4DHuman instead treat video diffusion as a view synthesizer that produces the supervision needed by downstream explicit reconstruction. Diffusion4D trains a 4D-aware video diffusion model on orbital views, conditions it on a motion magnitude metric, samples with DDIM and 3D-aware classifier-free guidance, and then fits 4D Gaussian splats in a coarse-to-fine schedule (Liang et al., 2024). 4DVD similarly generates dense-view videos in two cascaded stages: low-resolution dense-view layout generation and structure-aware spatio-temporal refinement with MAP-based appearance propagation (Yang et al., 6 Aug 2025). Flex4DHuman converts monocular or sparse-view human video into synchronized dense multi-view videos using a Wan 2.1 1.3B text-to-video DiT with five-axis positional encoding, after which FreeTimeGS reconstructs temporally evolving Gaussian primitives from the synthesized views (Cheng et al., 11 Jun 2026).

A third pattern uses diffusion as an in-loop correction mechanism. Splat4D renders pseudo-multi-view sequences from current per-frame Gaussian sets, estimates inconsistency masks from DINOv2 features and a WildGaussians-style uncertainty predictor, uses a Dynamicrafter-style video diffusion model to inpaint masked regions while preserving unmasked content and temporal coherence, and then updates Gaussian parameters through masked photometric and LPIPS losses (Yin et al., 11 Aug 2025). VDEGaussian adapts DynamiCrafter to the target scene, generates pseudo mid-frames at desired timestamps, aligns them with rendered interpolations through joint timestamp optimization, and distills only high-error content through a learnable uncertainty map (Xiao et al., 4 Aug 2025).

A fourth pattern shifts diffusion into latent Gaussian dynamics. Gaussian Variation Field Diffusion learns a VAE that directly encodes canonical Gaussian splats and their temporal variation fields from mesh animations, then trains a temporal-aware Diffusion Transformer to sample the variation latent conditioned on an input video and canonical Gaussian features (Zhang et al., 31 Jul 2025). This is not view synthesis followed by fitting; it is direct generation of animated Gaussian parameters.

An important boundary case is uncertainty-aware regularization with image diffusion rather than video diffusion. “4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization” fine-tunes Stable Diffusion with DreamBooth and applies SDEdit-style DDIM refinements to uncertain rendered novel views, explicitly stating that there is no explicit video-diffusion temporal score coupling (Kim et al., 2024). This clarifies that “diffusion-enhanced 4DGS” and “video diffusion-enhanced 4DGS” are overlapping but not identical categories.

4. Objectives, controllability, and consistency mechanisms

The training objectives used in this area reflect the chosen coupling strategy. DreamGaussian4D’s dynamic stage uses a reference-view photometric loss,

$S = \{g_i\}_{i=1..N}$6

augmented by SDS on random views for unseen-region propagation. Its texture refinement stage uses

$S = \{g_i\}_{i=1..N}$7

with a fixed noise level $S = \{g_i\}_{i=1..N}$8, and the paper reports that too low a noise level oversmooths textures while too high a level adds noise (Ren et al., 2023). Notably, the same work reports that no explicit temporal loss is required because spatial-temporal smoothness implicitly arises from the HexPlane decomposition.

Flex4DHuman uses a different control regime centered on camera-conditioned multi-view generation. Its diffusion model is trained with flow matching,

$S = \{g_i\}_{i=1..N}$9

and its five-axis positional encoding combines time, spatial coordinates, discrete view index, and continuous relative $\mu_i \in \mathbb{R}^3$0 camera geometry within attention. Temporal rollout reuses clean historical target-view tokens with overlap, which the paper states reduces drift and preserves synchronization across views (Cheng et al., 11 Jun 2026). Downstream 4DGS fitting then minimizes photometric, LPIPS, SSIM, smoothness, and covariance regularization terms.

Diffusion4D introduces an explicit motion control variable. It defines a 3D-to-4D motion magnitude metric

$\mu_i \in \mathbb{R}^3$1

between a dynamic orbital video and a matched static orbital reference, and pairs standard latent diffusion training with a motion magnitude reconstruction loss. Sampling uses a 3D-aware classifier-free guidance term that mixes conditional, unconditional, and pretrained 3D-aware predictions (Liang et al., 2024). A plausible implication is that motion strength becomes an explicit conditioning axis rather than a byproduct of the denoiser.

Splat4D and VDEGaussian make uncertainty central. Splat4D derives binary masks from per-pixel uncertainty estimates,

$\mu_i \in \mathbb{R}^3$2

and optimizes masked photometric and LPIPS losses only on unreliable regions after video-diffusion inpainting (Yin et al., 11 Aug 2025). VDEGaussian introduces an uncertainty-distilled alignment loss

$\mu_i \in \mathbb{R}^3$3

together with total variation regularization on the learnable uncertainty map. The closed-form optimum,

$\mu_i \in \mathbb{R}^3$4

makes the weighting directly proportional to squared residuals, thereby emphasizing fast-moving or poorly reconstructed regions (Xiao et al., 4 Aug 2025).

Controllability also differs substantially across methods. DreamGaussian4D exposes motion through the choice of driving video, including style, speed, and amplitude (Ren et al., 2023). Flex4DHuman adds optional text control and scales quality monotonically with more reference views (Cheng et al., 11 Jun 2026). Splat4D supports text/image-conditioned 4D generation, 4D human generation, and text-guided content editing through upstream diffusion modules such as SVD, Champ, and InstructPix2Pix, with the Gaussian refinement loop maintaining coherence (Yin et al., 11 Aug 2025). PLA4D’s focal alignment and front-view anchor video provide a contrasting notion of control: camera intrinsics and motion are aligned in pixel space before dynamic Gaussian optimization (Miao et al., 2024).

5. Reported empirical behavior and application domains

Reported results indicate that video diffusion can improve 4DGS through several different pathways: faster optimization, denser supervision, stronger temporal consistency, or higher-frequency detail. The numerical gains, however, are highly benchmark-dependent and should be read within each paper’s task definition.

Method	Setting	Selected reported result
DreamGaussian4D	Image-to-4D	CLIP-I 0.9227 vs 0.8544 for Animate124; ~6.5 min
Diffusion4D	Image-to-4D	SSIM 0.83, PSNR 16.7, LPIPS 0.21, FVD 560.8; 8m
4DVD	4D Gaussian reconstruction	LPIPS 0.136, CLIP-S 0.919, FVD 438.41; 381 s
Splat4D	Consistent4D	LPIPS 0.090, CLIP-S 0.97, FVD-V 282.79
Gaussian Variation Field Diffusion	100-instance test set	PSNR 18.47, LPIPS 0.114, SSIM 0.901, FVD 476.83; 4.5 s
Flex4DHuman	DNA-Rendering	25.44 PSNR, 0.9516 SSIM, 0.0617 LPIPS
VDEGaussian	Waymo subset	30.49 PSNR, 0.894 SSIM, 0.209 LPIPS

DreamGaussian4D reported better faithfulness and detail than Animate124 on eight public examples, and on the Consistent4D benchmark its “Ours-Fast” and “Ours” variants outperformed Consistent4D at a fraction of the time, with the full variant reporting LPIPS 0.12, CLIP 0.92, and FVD 729.74 versus LPIPS 0.16, CLIP 0.87, and FVD 1133.44 for Consistent4D (Ren et al., 2023). Diffusion4D reported end-to-end generation in 8 minutes, compared with hours for SDS-heavy baselines, and achieved better SSIM, PSNR, LPIPS, and FVD than prior image-to-4D baselines in its setting (Liang et al., 2024). 4DVD reported 381 seconds total runtime on a single A6000 GPU and stronger multi-view video and 4D reconstruction metrics than STAG4D, 4Diffusion, SV4D, and L4GM (Yang et al., 6 Aug 2025).

Human-centric reconstruction shows the value of dense synchronized multi-view synthesis. Flex4DHuman reported 25.44 PSNR, 0.9516 SSIM, and 0.0617 LPIPS on DNA-Rendering with one reference and 47 target views, beating Diffuman4D-mono-skeleton by +9.32 dB PSNR and surpassing Diffuman4D-GT-skeleton by +1.21 dB PSNR despite using no target-view geometry. On ActorsHQ, evaluated through FreeTimeGS at ground-truth cameras, it reported 21.32 PSNR, 0.856 SSIM, and 0.277 LPIPS versus 17.97, 0.815, and 0.307 for Diffuman4D-mono-skeleton (Cheng et al., 11 Jun 2026).

Monocular content-creation systems emphasize spatial-temporal coherence. Splat4D reported on Consistent4D that LPIPS improved to 0.090, CLIP-S to 0.97, FVD-F to 390.85, and FVD-V to 282.79, outperforming Consistent4D, STAG4D, SV4D, and 4Diffusion in that benchmark. Its ablations also showed degradation without the uncertainty mask or without U-Net fine-tuning (Yin et al., 11 Aug 2025). VDEGaussian, on a Waymo Open Dataset subset, reported 30.49 PSNR, 0.894 SSIM, and 0.209 LPIPS, with an approximate PSNR gain of 2 dB over PVG/DeSiReGS, and attributed the largest ablation gain to uncertainty distillation rather than adaptation or timestamp optimization alone (Xiao et al., 4 Aug 2025).

Latent direct generation methods target both fidelity and speed. Gaussian Variation Field Diffusion reported PSNR 18.47, LPIPS 0.114, SSIM 0.901, CLIP 0.935, and FVD 476.83 on a 100-instance test set at 512×512, with runtime 4.5 seconds per sequence on a single A100. Its comparison table showed higher PSNR, lower LPIPS, and better FVD than Consistent4D, SC4D, STAG4D, DreamGaussian4D, and L4GM in that setting (Zhang et al., 31 Jul 2025).

Application domains now extend well beyond object-centric prompt-based generation. Flex4DHuman explicitly targets simulation, gaming, AR/VR, and video re-shooting (Cheng et al., 11 Jun 2026). Splat4D includes text/image-conditioned 4D generation, 4D human generation, and text-guided content editing (Yin et al., 11 Aug 2025). VDEGaussian addresses self-supervised dynamic urban scene modeling, especially fast-moving actors in undersampled captures (Xiao et al., 4 Aug 2025). Large-scale volumetric capture systems use diffusion not to generate geometry but to raise rendered dynamic Gaussian outputs to production-grade closeups: the full detail-enhancement method reported PSNR 31.29, SSIM 0.8871, LPIPS 0.1812, TPSNR 31.27, and FVD 213.52 on Face Rig test data (Philip et al., 31 Oct 2025).

6. Limitations, misconceptions, and open research directions

A recurrent misconception is that video diffusion-enhanced 4DGS refers to a single optimization recipe in which a video denoiser supplies SDS gradients directly to Gaussian parameters. The literature shows otherwise. DreamGaussian4D uses video diffusion as a motion source and as a V2V prior, explicitly noting that the video diffusion model is used as a post-process prior in optimization rather than as SDS during the dynamic stage (Ren et al., 2023). PLA4D explicitly argues against unconstrained multi-DM SDS coupling and shifts supervision into pixel space (Miao et al., 2024). Flex4DHuman, 4DVD, and Diffusion4D use video diffusion primarily to synthesize or densify supervision before explicit 4DGS fitting (Cheng et al., 11 Jun 2026).

Another misconception is that temporally coherent video priors automatically resolve 3D consistency. VDEGaussian states that direct use of video diffusion pseudo-views is inadequate because current video diffusion models lack robust pixel-level 3D consistency and fine-grained pose control, leading to alignment conflicts with 3DGS (Xiao et al., 4 Aug 2025). Splat4D likewise notes that if the enhancer introduces large cross-view discrepancies, masks may become extensive, increasing reliance on diffusion priors and risking hallucinations (Yin et al., 11 Aug 2025). Related uncertainty-aware work with image diffusion reports that over-regularization or hallucination can occur where diffusion guidance conflicts with scene geometry, even when uncertainty masking mitigates the risk (Kim et al., 2024). This suggests that explicit camera conditioning, masking, alignment, or downstream geometric optimization remains necessary.

Several limitations recur across papers. DreamGaussian4D states that final visual fidelity is still below the most photorealistic 4D generations and that poor temporal consistency in the driving video can degrade results (Ren et al., 2023). Flex4DHuman reports reduced robustness to highly out-of-distribution camera motions, very long horizons, fast motion blur, and complex cloth dynamics, with an additional bias toward studio rigs in the training data (Cheng et al., 11 Jun 2026). 4DVD assumes object-centric videos and fixed view rigs, and notes that highly complex geometry or motion remains limited by network capacity and training data (Yang et al., 6 Aug 2025). VDEGaussian requires test-time adaptation and still inherits midpoint-pose bias from the pretrained video model (Xiao et al., 4 Aug 2025). Production systems remain resource-intensive and identity-specific: detail-enhanced Gaussian capture depends on actor-specific Face Rig training data and still exhibits residual flicker in some silhouettes and specular regions (Philip et al., 31 Oct 2025).

Open directions are correspondingly varied. DreamGaussian4D lists stronger temporal priors or flow-aware refinement, higher-resolution HexPlane variants or multi-resolution planes, and more advanced video diffusion backbones as future work (Ren et al., 2023). 4DVD points toward stronger conditioning mechanisms, end-to-end training of diffusion and 4DGS, and larger dynamic datasets (Yang et al., 6 Aug 2025). Gaussian Variation Field Diffusion proposes tighter alignment or end-to-end 4D diffusion that jointly produces the canonical Gaussian set and its temporal variations (Zhang et al., 31 Jul 2025). In production contexts, future relighting modules or eye-specific models are suggested to improve environment compatibility and closeup realism (Philip et al., 31 Oct 2025). A plausible implication is that the next phase of the field will be defined less by whether diffusion is used, and more by how precisely diffusion priors are aligned with explicit geometry, camera control, and downstream rendering constraints.