---
title: 4D Gaussian Splatting in Dynamic Scene Rendering
url: https://www.emergentmind.com/topics/4d-gaussian-splatting-4d-gs
type: topic
---

# 4D Gaussian Splatting in Dynamic Scene Rendering

4D Gaussian Splatting (4D-GS) is an explicit, volumetric representation and rendering framework for dynamic (time-varying) 3D scenes. Introduced by Yang et al. [2310.10642], 4D-GS addresses the inherent limitations of previous neural implicit and deformable radiance field approaches by directly modeling the full 4D spatio-temporal volume—space (ℝ³) and time (ℝ)—with a set of highly expressive, anisotropic, rotated 4D Gaussian primitives. The method achieves photorealistic novel-view synthesis at real time, supporting diverse downstream applications in video-based scene capture, digital twins, interactive editing, and efficient dynamic view rendering.

## 1. Core 4D Gaussian Primitive Representation

A dynamic scene is encoded as a sum of $N$ explicit 4D Gaussian primitives $\{G_i\}$, each parameterized by a 4D mean $\mu_i = (\mu_x, \mu_y, \mu_z, \mu_t) \in \mathbb{R}^4$ and a full covariance $\Sigma_i \in \mathbb{R}^{4 \times 4}$:
\[
G_i(x, t) = \exp\!\left(-\frac{1}{2}\left( [x, t] - \mu_i \right)^\top \Sigma_i^{-1} \left( [x, t] - \mu_i \right) \right)
\]
where $x \in \mathbb{R}^3$, $t \in \mathbb{R}$.

To enable stable optimization and full 4D anisotropy (including spatio-temporal orientation), $\Sigma_i$ is factored as:
\[
\Sigma_i = R_i S_i^2 R_i^\top
\]
where $S_i = \mathrm{diag}(s_x, s_y, s_z, s_t)$ and $R_i \in \mathrm{SO}(4)$ (a 4D rotation), parameterized using two quaternions. Each primitive thus defines an oriented, ellipsoidal support in space–time, with $s_t$ controlling its temporal extent; $\mu_t$ encodes its temporal position; and the full $\Sigma_i$ enables modeling of non-axis-aligned motion (e.g., scene elements moving along oblique space–time paths).

Conditional and marginalization identities from multivariate Gaussians yield:
- The spatial “slice” at time $t$ is a 3D Gaussian with:
  \[
  \begin{aligned}
    \mu_{xyz\,|\,t} &= \mu_{1:3} + \Sigma_{1:3,4} \Sigma_{4,4}^{-1} (t - \mu_t) \\
    \Sigma_{xyz\,|\,t} &= \Sigma_{1:3,1:3} - \Sigma_{1:3,4} \Sigma_{4,4}^{-1} \Sigma_{4,1:3}
  \end{aligned}
  \]
- The temporal marginal weight:
  \[
  p_i(t) = \exp\left( -\frac{1}{2}(t - \mu_t)^2 \Sigma_{4,4}^{-1} \right)
  \]

## 2. Appearance Model: 4D Spherindrical Harmonics

View- and time-dependent color is modeled with a compact, explicit expansion:
\[
c_i(d, t) = \sum_{n=0}^{N} \sum_{l=0}^{L} \sum_{m=-l}^l a_{i,n,l,m} Z_{n,l}^m(t, \theta, \phi)
\]
where $d = (\theta, \phi)$ are spherical camera directions and $Z_{n,l}^m$ is the 4D spherindrical basis:
\[
Z_{n,l}^m(t, \theta, \phi) = \cos\left( \frac{2\pi n}{T} t \right) Y_l^m(\theta, \phi)
\]
for a scene duration $T$, $Y_l^m$ being spherical harmonics.

This separable basis efficiently captures both high-frequency view-dependent reflectance and time-evolving appearance, with the learned coefficients $a_{i,n,l,m}$ per Gaussian.

## 3. Rendering Pipeline and Differentiable Splatting

The rendered color $\mathcal{I}(u, v, t)$ at pixel $(u, v)$ and time $t$ is computed by:
- Projecting each conditional 3D Gaussian (at $t$) into image space using camera parameters, linearizing projection via the Jacobian $J$.
- Computing the 2D projected Gaussian parameters:
  \[
  \begin{aligned}
    \mu_i^{2d} &= \mathrm{Proj}(\mu_{xyz\,|\,t}; E, K)_{1:2} \\
    \Sigma_i^{2d} &= (J E \Sigma_{xyz\,|\,t} E^\top J^\top)_{1:2,1:2}
  \end{aligned}
  \]
- Compositing splats with per-pixel weights:
  \[
  \mathcal{I}(u, v, t) = \sum_{i=1}^N p_i(t) \, p_i(u, v\,|\,t)\, \alpha_i\, c_i(d, t)\, \prod_{j < i} \left[1 - p_j(t)\, p_j(u, v\,|\,t)\, \alpha_j\right]
  \]
where $\alpha_i$ is a learned opacity.

GPU tile-based splat rasterization and depth-sorted blending (alpha compositing) yield efficient $>100$ FPS rendering at high resolutions. Gaussians with negligible $p_i(t)$ are pruned per frame.

## 4. Optimization and Training Protocol

Supervision is applied via photometric $\ell_2$ loss on (pixel, time) samples:
\[
\mathcal{L}_{\mathrm{photo}} = \sum_k \left\|\,\mathcal{I}(u_k, v_k, t_k) - \mathcal{I}^{\mathrm{gt}}(u_k, v_k, t_k)\,\right\|_2^2
\]

Adaptive densification and pruning are performed using spatial/temporal gradient magnitudes:
- Gaussians with low spatial gradient are pruned (insufficient reconstruction).
- High-gradient Gaussians are split in full 4D space–time (to capture detail).
- The mean temporal gradient of $\mu_t$ is monitored to ensure even time coverage.

Training batches rays sampled uniformly in $(u, v, t)$, rather than sequential frames, enforcing temporal consistency and suppressing flicker.

Initialization uses colored point clouds (e.g., COLMAP) at $t=0$, with $\mu_t$ initialized randomly in $[0, T]$ and temporal scale $s_t = T/2$. End-to-end training runs for $\sim30$k iterations (batch size 4), with densification rate halved at halfway point.

## 5. Empirical Performance and Benchmarks

On the Plenoptic Video (multi-view, real) benchmark, 4D-GS achieves:
- PSNR = 32.01, DSSIM = 0.014, LPIPS = 0.055
- $\sim$114 FPS on a single NVIDIA GPU

This surpasses prior neural dynamic scene models (DyNeRF, HexPlane, K-Planes, StreamRF, etc.) on both fidelity (PSNR, LPIPS) and real-time speed (often >10$\times$ faster than NeRF-based methods).

On monocular, under-constrained synthetic (D-NeRF) scenes, 4D-GS attains PSNR = 34.09 at real-time frame rates.

## 6. Methodological Distinctions and Theoretical Properties

- **True 4D Native Representation:** By representing spacetime as an explicit collection of 4D Gaussians, 4D-GS avoids overparametrizing time via separate deformation fields or per-frame duplication. All space–time correlations (motion, temporal occlusion, appearance drift) are encoded natively via the $\Sigma_i$ and spherindrical expansion.
- **Compact View-Time Appearance:** Spherindrical harmonics provide a parsimonious but expressive basis for handling high-frequency view and time effects, enabling both photorealism and efficient memory use.
- **Scalability and Flexibility:** The rasterization and compositing algorithm is GPU-friendly and scales with the number of visible Gaussians per frame, not the number of input images or total scene length.
- **Optimization Simplicity:** No additional regularizers or motion priors are required. All geometry, appearance, and motion are learned end-to-end, with dynamic splitting and pruning providing automatic model adaptation.

## 7. Extensions and Applications

The 4DGS framework catalyzed further research exploring:
- **Geometry-consistent extensions** for sparse camera inputs by integrating multi-view stereo priors [2511.23044].
- **Aggressive model compression** through pruning, quantization, and entropy-aware encoding to facilitate edge deployment [2510.10030, 2410.13613, 2503.13948].
- **Hybrid 3D–4D schemes** to segregate static background (as pure 3D Gaussians) from dynamic elements [2505.13215].
- **Generative content creation** via diffusion-driven 4DGS pipelines [2312.17142].
- **Real-time scene editing** and semantic manipulation leveraging 4DGS’s explicit structure [2510.01991].
- **Self-calibration from monocular video** without external structure-from-motion [2406.01042].

4D Gaussian Splatting has become a foundational approach for real-time, explicit, photorealistic dynamic scene representation, providing both practical utility and a mathematically tractable paradigm for space–time visual modeling [2310.10642, 2412.20720].

Source: https://www.emergentmind.com/topics/4d-gaussian-splatting-4d-gs