Hybrid 3D-4D Gaussian Splatting

• Hybrid 3D-4D Gaussian Splatting is a radiance field representation that models static scenes with 3D Gaussians and dynamic scenes with 4D Gaussians.
• It employs a conversion mechanism based on temporal scale thresholds to distinguish and optimize static and dynamic regions, reducing redundant computations.
• The hybrid approach significantly improves reconstruction accuracy, training speed, and memory usage for applications like video synthesis, SLAM, and medical imaging.

Hybrid 3D-4D Gaussian Splatting refers to a class of explicit radiance field representations that combine both static (3D) and dynamic (4D, space-time) anisotropic Gaussian primitives for efficient, high-fidelity modeling of static and dynamic scenes. This hybridization exploits the compactness and computational efficiency of 3D Gaussian splatting for time-invariant (static) regions, while reserving the expressive power of fully time-varying 4D Gaussians for dynamic elements, yielding significant gains in memory, training speed, and reconstruction accuracy for video, medical imaging, SLAM, and other spatio-temporal applications.

1. Mathematical Foundation and Hybrid Representation

In 3D Gaussian Splatting, a static scene is modeled as a set of $N_{3D}$ anisotropic Gaussians, each defined by mean $\boldsymbol\mu_i \in \mathbb{R}^3$, spatial covariance $\Sigma_i \in \mathbb{R}^{3 \times 3}$, color $c_i$ (typically via spherical harmonics), and opacity $\alpha_i$. The per-point density is

$$G_i(\mathbf{x}) = \exp\left(-\frac{1}{2}(\mathbf{x} - \boldsymbol\mu_i)^\top \Sigma_i^{-1} (\mathbf{x} - \boldsymbol\mu_i)\right).$$

Hybridization introduces a set of $N_{4D}$ Gaussians with joint mean $$\boldsymbol{\kappa}_i = [\mu_{x,i}; \mu_{t,i}] \in \mathbb{R}^4$$ and full covariance $\Sigma_i \in \mathbb{R}^{4 \times 4}$. Rendering to a time $t$ slices the 4D Gaussian along the temporal axis, yielding a time-conditional 3D Gaussian:

$\boldsymbol\mu_i \in \mathbb{R}^3$0

Opacity and appearance attributes may also be made time-dependent, $\boldsymbol\mu_i \in \mathbb{R}^3$1.

Hybrid 3D-4D Gaussian Splatting schemes (Oh et al., 19 May 2025) begin with a fully 4D representation and iteratively identify temporally invariant Gaussians. These are reparametrized as 3D (dropping their temporal parameters), reducing the number of free parameters dedicated to static regions. Regions with significant motion retain full 4D parameterization, supporting accurate modeling of complex dynamic scene elements.

2. Dynamic-Static Disentanglement and Conversion Criteria

The fundamental task is separating scene regions into static and dynamic, adapting Gaussian allocation accordingly. Static Gaussians exhibit large temporal scales, measurable by the learned time-axis scale parameter $\boldsymbol\mu_i \in \mathbb{R}^3$2 (temporal scale $\boldsymbol\mu_i \in \mathbb{R}^3$3). A Gaussian is considered stationary and thus converted to 3D when $\boldsymbol\mu_i \in \mathbb{R}^3$4, where $\boldsymbol\mu_i \in \mathbb{R}^3$5 is a dataset-specific threshold set in the valley separating dynamic and static scale distributions. Following conversion, these primitives lose their time dimension:

$\boldsymbol\mu_i \in \mathbb{R}^3$6

Dynamic regions, indicated by lower temporal scale, remain as full 4D Gaussians, which undergo further densification, splitting, or pruning (Oh et al., 19 May 2025).

This process avoids redundant temporal parameters for stationary backgrounds, significantly compressing the representation and enabling the optimizer to focus dynamic capacity on nonrigid scene elements.

3. Rendering and Optimization Pipeline

Rendering proceeds by projecting all active Gaussians—3D (for static regions) and 4D time-sliced (for dynamics)—into world and camera space, then to the image plane as anisotropic 2D Gaussians. These are blended by depth order via the alpha compositing formula:

$\boldsymbol\mu_i \in \mathbb{R}^3$7

The explicit splat form efficiently accumulates color and opacity, leveraging modern GPU rasterization workflows.

Training optimizes photometric $\boldsymbol\mu_i \in \mathbb{R}^3$8 or $\boldsymbol\mu_i \in \mathbb{R}^3$9 losses between rendered and observed images,

$\Sigma_i \in \mathbb{R}^{3 \times 3}$0

with no explicit temporal-consistency or opacity-reset terms required for stable convergence in the hybrid setting (Oh et al., 19 May 2025). Pruning and densification operate separately on the 3D and 4D Gaussian sets. After conversion and pruning, static (3D) Gaussians are updated in every batch, while dynamic (4D) Gaussians can be split for detail or culled if inactive.

Optimizers (e.g., Adam) and learning rate schedules are inherited from the base 4DGS frameworks.

4. Memory and Computational Efficiency

Hybridization yields dramatic resource reductions. On typical N3V datasets, full 4DGS models require $\Sigma_i \in \mathbb{R}^{3 \times 3}$13.3 million 4D Gaussians ($\Sigma_i \in \mathbb{R}^{3 \times 3}$22.1 GB). Hybrid 3D-4DGS (Oh et al., 19 May 2025) reduces this to $\Sigma_i \in \mathbb{R}^{3 \times 3}$3843k 4D and 230k 3D Gaussians, totaling 273 MB—a compression factor of %%%%24$\Sigma_i \in \mathbb{R}^{4 \times 4}$25%%%%. Training time drops from 5.5 hours to $\Sigma_i \in \mathbb{R}^{3 \times 3}$612 minutes (25$\Sigma_i \in \mathbb{R}^{3 \times 3}$7 reduction), and runtime performance substantially increases (from 114 fps to 208 fps). These gains result directly from the reduced parameter set and less frequent dynamic updates in static regions.

Quality is unaffected or slightly improved. PSNR on 10 s N3V clips rises to 32.25 dB (vs. 32.01 dB), SSIM to 0.946 (vs. 0.945), and flicker in static backgrounds is reduced, confirming that full 4D representation is unnecessary for stationary content.

5. Practical Challenges and Hyperparameter Selection

While hybridization achieves parameter and performance efficiency, it introduces sensitivity to the temporal scale threshold $\Sigma_i \in \mathbb{R}^{3 \times 3}$8. Underestimating $\Sigma_i \in \mathbb{R}^{3 \times 3}$9 (too low) forces dynamic modeling of slow-moving or quasi-static components into the static set, sacrificing fidelity in those regions. Overestimating $c_i$0 (too high) delays conversion, wasting resources. The optimal value of $c_i$1 is dataset-specific; for example, $c_i$2 suffices for 10-second N3V sequences, while $c_i$3 is used for shorter 50-frame Technicolor captures (Oh et al., 19 May 2025).

Longer video sequences and higher resolution frames benefit disproportionately, since static Gaussians only require storage once, independent of temporal range, enabling efficient extension to, e.g., 40 s videos or large-format scenes.

A plausible implication is that future methods incorporating data-driven or learned temporal-scale classification could further automate and refine this static-dynamic separation.

6. Applications and Broader Context

Hybrid 3D-4D Gaussian Splatting is directly applicable to dynamic neural scene reconstruction, photorealistic novel-view video synthesis, real-time SLAM, medical imaging (e.g., vessel tracking from DSA (Liu et al., 2024)), and 4D video content generation. By judiciously deploying 4D Gaussians only over truly dynamic content, these frameworks balance memory usage, training/inference speed, and output quality. This trade-off distinguishes hybrid approaches from prior work that either treat the entire scene as static (losing temporal flexibility) or as fully dynamic (suffering redundant parameterization and computational overload).

In sum, the hybrid paradigm provides a scalable, explicit radiance field representation with state-of-the-art temporal fidelity and resource efficiency, and continues to inform emerging research on spatio-temporal scene modeling and representation learning (Oh et al., 19 May 2025).