DreamGaussian4D: Efficient 4D Scene Representation

• DreamGaussian4D is a framework for dynamic 4D scene representation using explicit 3D Gaussian splatting with time-evolving deformation fields.
• It integrates dual-domain, hexplane, and neural-voxel deformation models to efficiently capture spatial-temporal dynamics and achieve high-fidelity rendering.
• The system enables real-time rendering, rapid training, and seamless asset export, significantly advancing dynamic scene synthesis and animation workflows.

The DreamGaussian4D Framework defines an efficient class of explicit 4D scene representations for dynamic scene synthesis, animation, and photorealistic real-time rendering. It leverages spatially explicit 3D Gaussian splatting, extended with compact parametric or neural deformation fields, to efficiently model dynamic content over time. The core approach under DreamGaussian4D encompasses methods across both generative and reconstruction paradigms, including explicit deformation-based temporal modeling, neural-voxel-based factorization, and spatial-temporal consistency refinements, unified by the goal of compact, fast, and high-fidelity 4D content generation and rendering (Ren et al., 2023, Lin et al., 2023, Wu et al., 2023).

1. Core Representation: 4D Gaussian Splatting

DreamGaussian4D represents dynamic scenes as explicit sets of anisotropic 3D Gaussians whose parameters evolve over time through learned or optimized deformation fields. Each Gaussian $G_i$ is parameterized by its center $\mu_i\in\mathbb{R}^3$, covariance $\Sigma_i\in\mathbb{R}^{3\times3}$, color $c_i\in\mathbb{R}^3$, and density $w_i$. At any time $t$, deformation fields $\phi$ update each Gaussian:

$$\mu_i(t) = \mu_i + \Delta\mu_i(\mu_i, t), \quad \Sigma_i(t) = R_i(\mu_i, t)\Sigma_i R_i(\mu_i, t)^T$$

Rendering at time $t$ uses the deformed set $\{G_i(t)\}$ via front-to-back differentiable splatting (Kerbl et al. 2023), yielding temporally coherent images or videos (Ren et al., 2023, Wu et al., 2023, Lin et al., 2023). This framework is agnostic as to how the deformation $\Delta \mu_i(\cdot, t)$ is parameterized (fully explicit, neural, or hybrid).

2. Temporal Dynamics and Deformation Modeling

DreamGaussian4D supports several deformation-field parameterizations:

• Dual-Domain Deformation Model (DDDM): Each attribute (position, rotation, color) per Gaussian is modeled as a sum of an order-$N$ polynomial in time and an $L$-term truncated Fourier series, with the full time-dependent value:

$$S_d(t) = S_{0,d} + \sum_{n=0}^{N} a_n t^n + \sum_{l=1}^{L}\left(f^l_{\mathrm{sin}}\cos(lt) + f^l_{\mathrm{cos}}\sin(lt)\right)$$

This enables efficient modeling of both smooth and high-frequency temporal behavior, with explicit, jointly-optimized closed-form coefficients, eliminating the need for neural implicit fields (Lin et al., 2023).

• HexPlane Neural Factorization: Temporal/spatial dependencies are captured by factorizing the 4D $(x,y,z,t)$ domain into six 2D feature planes (e.g., $H_{xy}, H_{xz},...$). For each Gaussian, features are bilinearly sampled from these planes at $(\mu_i, t)$ and decoded by a small MLP to output deformations $(\Delta\mu_i, \Delta R_i, \Delta s_i)$. This yields a compact, expressive temporal model, well-suited for generative tasks (Ren et al., 2023, Wu et al., 2023).
• Neural-Voxel Deformation Networks: In reconstruction settings, canonical 3D Gaussians are deformed via a neural field indexed by both spatial and temporal location, typically using a voxel-encoded network and multi-head MLPs to predict per-Gaussian offsets, scale changes, and rotations at each time (Wu et al., 2023).

These methods can be complemented by mechanisms for time scaling per-particle (to adapt to heterogeneous motion velocities) and explicit per-attribute regularization (e.g., time smoothness, KNN-based local rigidity, and temporal correlations).

3. Training Pipelines and Optimization

Training proceeds in modular phases:

• Static Initialization: Gaussians are initialized via sparse multi-view structure-from-motion (SfM) on key frames, followed by static optimization (photometric/reconstruction loss) for base geometry (Lin et al., 2023, Ren et al., 2023, Wu et al., 2023).
• Dynamic Optimization: With motion turned on, deformation parameters (polynomial/Fourier, HexPlane grid, or neural voxel features) are jointly optimized. Mini-batches of camera/ray samples are drawn from video frames, and rendered images are compared to ground truth using photometric MSE, SSIM, score-distillation (SDS from 3D-aware diffusion), and temporal/rigidity regularization terms (Lin et al., 2023, Ren et al., 2023, Wu et al., 2023).

Loss function design follows the principle:

$$\mathcal L = \underbrace{\mathcal L_{\rm photo} + \lambda_t \mathcal L_t+\lambda_s \mathcal L_s}_{\rm deformation\,regularization} + \underbrace{\mathcal L_{\rm rec} + \mathcal L_{\rm SDS}}_{\rm supervision}$$

Where $\mathcal L_{\rm t}$ encourages temporal smoothness and $\mathcal L_{\rm s}$ enforces local rigidity among neighboring Gaussians.

• Fine Texture/Consistency Refinement: For generative synthesis, DreamGaussian4D supports a video-to-video diffusion UV-space refinement. Raw mesh+UVs extracted from the dynamic GS sequence are refined via denoising diffusion on the UV-texture atlas, enhancing temporal and spatial coherence (Ren et al., 2023).

4. Rendering, Acceleration, and Memory Efficiency

The fully explicit architecture ensures real-time rendering:

• Forward Rendering: At test time, no neural inference is required for explicit models—deformation is a per-particle polynomial/Fourier/MLP evaluation plus projection and rasterization via fast tile-based EWA splatting. GPU-parallelized kernels (e.g., via Taichi, CUDA) allow the entire process (for $M\sim200$k particles) to complete in $1$–$2$ ms per frame, with framerates exceeding 100+ FPS on modern hardware (Lin et al., 2023, Ren et al., 2023).
• Memory Footprint: Efficiency is achieved by factorizing deformation parameters (e.g., SoA on GPU, $\leq$100 floats/particle, HexPlane grid resolution $32^3/32$, etc.). For typical scene complexity, memory consumption is $18$–$90$ MB per scene (Ren et al., 2023, Wu et al., 2023).
• Adaptive Densification and Pruning: During training, points/anchors are adaptively split if photometric or deformation gradients are large, and pruned if underutilized, to maintain fidelity while minimizing storage (Lin et al., 2023).

5. Quantitative Performance and Benchmarks

Empirical assessment consistently shows strong performance:

Benchmark/Method	Train Time	FPS	Storage	PSNR (dB)	SSIM	LPIPS	CLIP	FVD	FID-VID
4D-GS (static, real)	40 min	55	52 MB	19.7	0.680	–	–	–	–
DreamGaussian4D [(Ren et al., 2023)/(Wu et al., 2023)]	40 min	30–82	18–90 MB	25.2–34.05	0.845–0.98	0.049–0.02	0.92	729	45.0
Gaussian-Flow (Lin et al., 2023)	12 min	125	80 MB	26.3	0.862	–	–	–	–
4DSTR (Liu et al., 10 Nov 2025)	–	80	+0.23 GiB	–	–	0.12	0.92	795	45.0

DreamGaussian4D achieves a $5\times$ speedup in training and up to $250\times$ faster rendering compared to per-frame 3DGS (Lin et al., 2023). Qualitatively, it avoids ghosting artifacts seen in implicit neural field methods, and produces crisper dynamic detail (Ren et al., 2023, Wu et al., 2023). On recognized benchmarks (e.g., HyperNeRF, Neu3D), DreamGaussian4D matches or surpasses baselines in objective metrics and visual fidelity.

6. Applications, Controllability, and Export

DreamGaussian4D finds direct application in:

• Dynamic scene reconstruction: From monocular or multi-view video, real-time novel-view synthesis and freeviewpoint video.
• 4D generative content creation: Image-to-4D synthesis, driven by video-diffusion models; allows artist- or externally-driven motion control (e.g., by swapping the driving video clip).
• Export and integration: Meshed and UV-textured outputs can be directly exported as animated .obj/.fbx/.glTF assets for integration in Blender, Unreal, Unity, or other 3D engines, compatible with standard production pipelines (Ren et al., 2023).

Motion controllability is natively supported: users can control style, speed, and qualitative flow of motion by selecting or editing driving video input. Constraint mechanisms (e.g., bounding-box or L2 penalties on deformations, HexPlane grid resolution adjustment) enable fine-grained temporal and spatial smoothness/flexibility control (Ren et al., 2023).

7. Relation to Broader 4D and Dynamic Scene Research

DreamGaussian4D’s framework has influenced, and been adapted by, related work:

• 4DSTR (Liu et al., 10 Nov 2025): Incorporates spatial-temporal rectification and Mamba-based temporal correlation modules to guarantee scale/rotation consistency and adapt to rapid motion variations, further improving spatial-temporal coherence and rendering quality.
• SD-GS (Yao et al., 10 Jul 2025): Uses structured deformable anchor grids and deformation-aware densification for compact, efficient dynamic motion representation, addressing the memory-fidelity trade-off.
• Single-image dynamic scene video (Jin et al., 4 Apr 2025): Extends DreamGaussian4D’s methods to single image animation scenarios, combining explicit 4D GS with 3D motion consistency estimation for photorealistic animated landscapes from monocular input.
• Comparison to NeRF/Deformable Implicit Fields: DreamGaussian4D differs fundamentally from NeRF-based methods by virtue of its explicit, splatting-centric, non-neural (or lightly neural) temporal deformation model, achieving order-of-magnitude improvements in training and rendering efficiency, with native asset export capabilities (Lin et al., 2023, Ren et al., 2023, Wu et al., 2023).

In summary, the DreamGaussian4D Framework unifies explicit 3D Gaussian splatting, temporally and spatially factorized dynamic deformation modeling, and neural refinement to provide a comprehensive, high-efficiency, high-fidelity pipeline for 4D scene representation, content generation, and interactive rendering (Ren et al., 2023, Lin et al., 2023, Wu et al., 2023).