---
title: Wasserstein-Constrained 4D Gaussian Splatting
url: https://www.emergentmind.com/topics/wasserstein-constrained-4dgs
type: topic
---

# Wasserstein-Constrained 4D Gaussian Splatting

Wasserstein-Constrained 4DGS (Four-Dimensional Gaussian Splatting) is a methodology for dynamic scene rendering that jointly models the smooth translation and deformation of 3D Gaussian primitives over time. By embedding state-space modeling within a neural 4DGS pipeline and regularizing trajectories via the 2-Wasserstein distance, this approach enforces temporal coherence, physical plausibility, and efficient optimization for photorealistic, temporally consistent multi-frame rendering of dynamic scenes [2412.00333].

## 1. Dynamic 4DGS Pipeline and Problem Formulation

The core pipeline ingests a sparse point cloud generated by Structure-from-Motion (SfM), from which it parameterizes a set of canonical 3D Gaussians. Each Gaussian primitive $i$ is described by a mean $\mu^{c(i)} \in \mathbb{R}^3$, rotation $R^{c(i)} \in \mathrm{SO}(3)$, and scale $S^{c(i)} \in \mathbb{R}^{3 \times 3}$. The covariance is $\Sigma^{c(i)} = R^{c(i)} S^{c(i)} S^{c(i)\,T} R^{c(i)\,T}$. For each time step $t$, a deformation network $f_\theta$—parameterized as an MLP—predicts a time-dependent “observation” Gaussian
$$
\mathcal{N}_t^{\mathrm{Ob}(i)} = f_\theta\left(\mathcal{N}^{c(i)},\,t\right) = \left(\mu_t^{\mathrm{Ob}(i)},\,\Sigma_t^{\mathrm{Ob}(i)}\right).
$$
A state-space model—including predictor and filter—merges network predictions with a dynamical prior to promote smooth and physically plausible temporal trajectories. The resulting filtered Gaussians $\hat{\mathcal{N}}_t^{(i)}$ are rendered by a differentiable splatting scheme to generate each RGB frame at time $t$.

Key methodological challenges identified include:
- Suppression of abrupt jumps or flicker in means $\mu$ and covariances $\Sigma$ across frames.
- Unified modeling of both translation and shape deformation with a geometrically faithful metric.
- Realtime or near-realtime temporal optimization within high-dimensional 4DGS systems [2412.00333].

## 2. State-Space Modeling for Temporal Coherence

Each dynamic Gaussian’s state at time $t$ is $x_t = [\mu_t;\;\mathrm{vec}(\Sigma_t)] \in \mathbb{R}^{3+6}$ (as only the symmetric degrees of $\Sigma_t$ are stored). The transition (predictor) step can follow either a simple Euclidean rule or optimal transport geometry:
- **Euclidean baseline**:
  $$
  \mu_{t+1}^P = \mu_t + (\mu_t-\mu_{t-1}),\quad \Sigma_{t+1}^P = \Sigma_t + (\Sigma_t-\Sigma_{t-1})
  $$
- **Wasserstein-geometry version**:
  $$
  v_t = -\log_{\mathcal{N}_t} \left(\mathcal{N}_{t-1}\right), \qquad
  \mathcal{N}_{t+1}^P = \exp_{\mathcal{N}_t}(v_t)
  $$
Observations at each time point are provided by the deformation network as noisy measurements of the ground-truth state.

This state-space abstraction enables the filtering and regularization of temporal Gaussian trajectories in a mathematically structured manner, supporting the integration of both neural observations and a physically inspired dynamical prior [2412.00333].

## 3. State Consistency Filtering via Kalman-like Fusion

Temporal consistency is enforced through a Kalman filter-inspired state update combining the prior (predictive) state $\mathcal{N}_t^P$ and network observation $\mathcal{N}_t^{\mathrm{Ob}}$:
$$
K_t = \Sigma_t^{\mathrm{Ob}} \left(\Sigma_t^{\mathrm{Ob}} + \Sigma_t^P\right)^{-1}
$$
$$
\begin{aligned}
\hat{\mu}_t    &= \mu_t^{\mathrm{Ob}} + K_t \left(\mu_t^P - \mu_t^{\mathrm{Ob}}\right) \\
\hat{\Sigma}_t &= \Sigma_t^{\mathrm{Ob}} + K_t \left(\Sigma_t^P - \Sigma_t^{\mathrm{Ob}}\right)
\end{aligned}
$$
This filter update suppresses erratic MLP-driven changes (“flicker”) by optimally weighting neural predictions against the dynamical prior. A plausible implication is improved stability in the resulting Gaussian trajectories and reduced image-space optical-flow artifacts.

## 4. Wasserstein Geometry and Regularization

The framework leverages Wasserstein geometry—specifically the 2-Wasserstein distance—to regularize both state estimation and trajectory evolution. For Gaussians $\mathcal{N}_1=(m_1,\Sigma_1)$, $\mathcal{N}_2=(m_2,\Sigma_2)$, the 2-Wasserstein metric is:
$$
W_2^2(\mathcal{N}_1,\mathcal{N}_2) =
\|m_1-m_2\|^2 +
\operatorname{Tr}\left(\Sigma_1 + \Sigma_2 - 2 \left(\Sigma_1^{1/2}\Sigma_2\Sigma_1^{1/2}\right)^{1/2}\right)
$$
In this model, $\Sigma_i$ is decomposed as $R_i S_i^2 R_i^T$ and the trace term is computed efficiently to avoid redundancy. Regularization losses include:
- **State-Observation Alignment**:
  $$
  \mathcal{L}_{\mathrm{SOA}} = \sum_{t,i} W_2^2\left(\mathcal{N}_t^{P(i)},\,\mathcal{N}_t^{\mathrm{Ob}(i)}\right)
  $$
- **Temporal Smoothness**:
  $$
  \mathcal{L}_{\mathrm{WR}} = \sum_{t=1}^{T}\sum_{i=1}^{N} W_2^2\left(\hat{\mathcal{N}}_t^{(i)},\,\hat{\mathcal{N}}_{t-1}^{(i)}\right)
  $$
The total loss is formulated as
$$
\mathcal{L}_{\mathrm{total}} =
\mathcal{L}_{\mathrm{render}} +
\lambda_{\mathrm{SOA}} \mathcal{L}_{\mathrm{SOA}} +
\lambda_{\mathrm{WR}}\mathcal{L}_{\mathrm{WR}}
$$
where $\mathcal{L}_{\mathrm{render}}$ is the standard photometric error.

Logarithmic and exponential maps in the space of symmetric positive definite (SPD) matrices ensure that covariance updates remain SPD and follow geodesic paths, thereby preserving physical plausibility during rapid or deformable motions [2412.00333].

## 5. End-to-end Algorithmic Workflow

The following pseudocode exemplifies the update routine per Gaussian and time step:
```python
Input: N^c(i), DeformNet f_θ, hyper-params λ
Initialize: μ₀, Σ₀ from SfM.

for t = 1…T do
    1. If t>1:
        v_t = − log_{N_t}(N_{t−1})
        N_{t}^P = exp_{N_{t−1}}(v_{t−1})
    2. N_{t}^{Ob} = f_θ(N^c(i), t)
    3. K_t = Σ_t^{Ob} (Σ_t^{Ob} + Σ_t^P)^{-1}
       \hat{N}_t = N_t^{Ob} + K_t (N_t^P − N_t^{Ob})
    4. Render \hat{N}_t and accumulate \mathcal{L}_{render}
    5. \mathcal{L}_{SOA} += W_2^2( N_t^P, N_t^{Ob})
       if t>1: \mathcal{L}_{WR} += W_2^2(\hat{N}_t,\hat{N}_{t−1})
end for

Backpropagate total loss, update θ and Gaussian parameters.
```
This joint optimization ensures end-to-end differentiability across geometric, temporal, and rendering domains, permitting seamless integration of neural modeling and physical regularization [2412.00333].

## 6. Empirical Evaluation and Performance

Experiments utilize both synthetic (D-NeRF: moving digits, animated characters, $800 \times 800$ resolution) and real-world (Plenoptic Video: people, objects, $1352 \times 1014$) dynamic scene datasets. The method was implemented on an NVIDIA A800 GPU with the Adam optimizer, $150$k training iterations (initial $3$k for static initialization), filter enabled from $6$k, and Wasserstein regularization activated from $20$k iterations onwards.

The evaluation metrics include PSNR (peak signal-to-noise ratio), SSIM, LPIPS, runtime FPS, and total training time.

| Dataset        | Method           | PSNR (dB) | SSIM   | LPIPS  | FPS    |
|----------------|------------------|-----------|--------|--------|--------|
| D-NeRF         | 4D-GS baseline   | 31.8      | 0.958  | 0.032  | 87     |
|                | Ours             | 34.45     | 0.970  | 0.026  | 45.5   |
| Plenoptic      | 4D-GS baseline   | 29.91     | 0.928  | 0.168  | 76     |
|                | Ours             | 31.62     | 0.940  | 0.140  | 37     |

Ablation results detail the contribution of individual innovations:
- Adding the state consistency filter yields a +0.80 dB PSNR increase and optical-flow AEPE reduced from 1.45 to 1.02.
- Wasserstein regularization, compared to linear regularization, provides +1.0 dB PSNR and a 57% reduction in training time.
- Full Wasserstein geometry (log/exp maps on SPD) further increases PSNR by +0.50 dB.
Combined, these components provide +2.0 dB PSNR over observation-only baselines and outperform previous dynamic scene rendering methods on all reported datasets [2412.00333].

## 7. Analysis: Motion Smoothness, Plausibility, and Implications

The fusion filter (Kalman-style) reduces network-induced noise, yielding improved optical-flow consistency. Wasserstein regularization jointly constrains both mean and covariance, damping flicker while accommodating plausible object deformation. Manifold-structured log/exp updates guarantee covariances remain SPD and follow geodesics, beneficial for rapid or complex shape variation.

This methodology embeds 3D Gaussian trajectory estimation into a state-space model guided by optimal-transport geometry. Empirically, this enables smoother transitions, reduced artifacts, and enhanced temporal coherence compared to naive neural deformation or simple regularization. A plausible implication is that Wasserstein-constrained 4DGS yields physically plausible, high-fidelity dynamic scene renderings, supporting scalable real-world deployment in dynamic view synthesis, neural graphics, and temporally adaptive reconstruction pipelines [2412.00333].

Source: https://www.emergentmind.com/topics/wasserstein-constrained-4dgs