Neighbor-Constrained 4D Gaussian Splatting

• Neighbor-Constrained 4D Gaussian Splatting extends 3D Gaussian Splatting by integrating spatial, temporal, and semantic constraints to ensure local coherence and suppress artifacts like flicker and drift.
• Techniques such as state-space filtering and Wasserstein regularization, along with KDTree kNN grouping, enforce temporal smoothness and local geometric consistency.
• Empirical results report significant PSNR/SSIM improvements and up to 30% reduction in temporal instability, demonstrating enhanced dynamic view synthesis in challenging scenarios.

Neighbor-constrained 4D Gaussian Splatting is a family of techniques for dynamic scene reconstruction and rendering that imposes local coherence and geometric regularization on 4D Gaussian primitives. These approaches extend the 3D Gaussian Splatting (3DGS) representation to capture temporal evolution while introducing constraints—rooted in notions of neighborhood (spatial, temporal, or semantic)—to regularize the highly under-constrained problem of spatio-temporal (4D) geometry learning. Neighbor constraints suppress artifacts such as temporal flicker, geometry drift, and incoherent deformation, enabling temporally consistent, high-fidelity dynamic view synthesis. Multiple methodologies instantiate these constraints using state-space filtering, Wasserstein geometry, view-based or metric neighborhoods, and semantic cues.

1. Mathematical Foundation of 4D Gaussian Splatting and Neighbor Relations

At its core, a 4D Gaussian Splatting model parameterizes a dynamic scene with a collection of Gaussians whose parameters evolve over time. A typical formulation uses, for each primitive at time $t$:

• Mean position $\mu_t^{(i)} \in \mathbb{R}^3$
• Symmetric positive-definite covariance $\Sigma_t^{(i)} \in \mathrm{SPD}(3)$ or $\Sigma_i \in \mathrm{SPD}(4)$ in spatio-temporal models
• Weight (intensity) $w_t^{(i)}$ (opacity/density)

The full 4D representation over $T$ frames and $N$ primitives is $\{ (w_t^{(i)}, \mu_t^{(i)}, \Sigma_t^{(i)}) \}$. Covariance matrices are typically factorized as $R S S^\top R^\top$ with $R \in \mathrm{SO}(3)$ or $\mu_t^{(i)} \in \mathbb{R}^3$0 and $\mu_t^{(i)} \in \mathbb{R}^3$1 diagonal.

Neighbor relations are induced either temporally (successive frames), spatially (KDTree neighbors, Euclidean distance), view-based (ray grouping), or semantically (class labels and KNN on surfaces).

2. Temporal Neighbor Regularization via State-Space Filtering and Wasserstein Geometry

The approach in "Gaussians on their Way: Wasserstein-Constrained 4D Gaussian Splatting with State-Space Modeling" (Deng et al., 2024) formulates the prediction of Gaussian parameters as a dynamical system and enforces neighbor constraints temporally using the following mechanisms:

• State Consistency Filter: Each Gaussian's parameters are updated by merging a prior (from the previous frame via a linear dynamical model) and a data observation (from a neural deformation field), employing a Kalman-like filter structure. Formally, the update is

$\mu_t^{(i)} \in \mathbb{R}^3$2

$\mu_t^{(i)} \in \mathbb{R}^3$3

$\mu_t^{(i)} \in \mathbb{R}^3$4

• Wasserstein Regularization: Neighbor constraints are realized as penalization of the squared 2-Wasserstein distance $\mu_t^{(i)} \in \mathbb{R}^3$5 between predicted and observed states as well as between consecutive temporal states, with objective terms:
  • State-observation alignment: $\mu_t^{(i)} \in \mathbb{R}^3$6
  • Temporal smoothness: $\mu_t^{(i)} \in \mathbb{R}^3$7
• Optimization: The total loss $\mu_t^{(i)} \in \mathbb{R}^3$8 blends rendering error with these regularizers.

This enforces that each primitive moves smoothly through the Wasserstein space, coupling means and covariances temporally. Experimentally, this reduces flicker, increases temporal coherence, and materially improves PSNR/SSIM over baselines (Deng et al., 2024).

3. Neighborhood Consistency via View-Space and Metric Grouping

Alternative strategies define neighborhoods based on spatial or view-based proximity.

• Ray-based View Groups ("Relaxed Rigidity with Ray-based Grouping" (Leea et al., 26 Mar 2026)):
  • For each image pixel, all Gaussians intersected by its ray, and with $\mu_t^{(i)} \in \mathbb{R}^3$9-blending weight above a threshold $\Sigma_t^{(i)} \in \mathrm{SPD}(3)$0, are grouped as neighbors $\Sigma_t^{(i)} \in \mathrm{SPD}(3)$1.
  • Motion Coherence Regularization (MCR): For each group, penalize directional inconsistency of displacement vectors over small $\Sigma_t^{(i)} \in \mathrm{SPD}(3)$2, using a cosine loss:

  $\Sigma_t^{(i)} \in \mathrm{SPD}(3)$3 - Spectral Regularization: Enforce consistency in eigen-spectrum of within-group position covariances across time. - These regularizations require no external motion priors and improve PSNR/SSIM as well as perceptual stability (Leea et al., 26 Mar 2026).

• KDTree-based Metric Neighborhoods ("Laplacian Analysis Meets Dynamics Modelling" (Zhou et al., 7 Aug 2025)):

  • For each Gaussian $\Sigma_t^{(i)} \in \mathrm{SPD}(3)$4 with mean $\Sigma_t^{(i)} \in \mathrm{SPD}(3)$5, $\Sigma_t^{(i)} \in \mathrm{SPD}(3)$6 nearest neighbors $\Sigma_t^{(i)} \in \mathrm{SPD}(3)$7 are defined via Euclidean distance.
  • The neighbor penalty loss:

  $\Sigma_t^{(i)} \in \mathrm{SPD}(3)$8 - Adaptive split strategies refine local geometry only when neighbor statistics indicate under-resolution, dynamically allocating computational capacity to regions of high detail or motion. - This enhances both fidelity (PSNR/SSIM) and computational efficiency (Zhou et al., 7 Aug 2025).

4. Semantic and Geometry-Guided Constraints

Neighbor constraints can also be informed by semantic segmentation and explicit geometric priors.

• Semantic KNN Regularization ("UrbanGS" (Li et al., 2024)): For Gaussians classified as "ground," a KNN (typically 16) is computed over canonical 3D centroids. The following scale-consistency regularizer is applied:

$\Sigma_t^{(i)} \in \mathrm{SPD}(3)$9

This ensures local geometric consistency on low-texture surfaces. Global temporal consistency is enforced by freezing all parameters for static classes across time.

• Spatio-Temporal Covariance Coupling ("Geometry-Consistent 4D Gaussian Splatting" (Li et al., 28 Nov 2025)):
  • 4D Gaussians with mean $\Sigma_i \in \mathrm{SPD}(4)$0 and spatio-temporal covariance $\Sigma_i \in \mathrm{SPD}(4)$1 implicitly encode local 4D neighborhoods through off-diagonal (space-time) terms, which steer joint evolution of clusters of Gaussians.
  • Regularization terms integrate MVS- and MDE-derived geometric priors via pixel-pair ranking and patch smoothness penalties, using neighbor and patch sampling strategies.

5. Loss Functions, Training, and Hyperparameters

Most frameworks formulate a composite objective that augments the per-pixel photometric and perceptual losses with neighbor-driven regularizers:

Loss Term	Role	Reference Papers
Wasserstein Temporal Reg.	Enforces smooth evolution of Gaussians across frames	(Deng et al., 2024)
State-Observation Align.	Penalizes model–observation divergence in Wasserstein	(Deng et al., 2024)
Neighborhood Penalty (Lneigh)	Local geometric consistency among kNNs	(Zhou et al., 7 Aug 2025)
Motion Coherence/Spectral Reg.	Preserves locally rigid/group motion and shape spectrum	(Leea et al., 26 Mar 2026)
Semantic KNN Consistency	Coherence for low-texture static regions	(Li et al., 2024)
Global Consistency	Frozen parameters for static Gaussians	(Li et al., 2024)

Hyperparameters (e.g., $\Sigma_i \in \mathrm{SPD}(4)$2, $\Sigma_i \in \mathrm{SPD}(4)$3, $\Sigma_i \in \mathrm{SPD}(4)$4, weighting ratios between losses) are generally data- and domain-dependent, with ablations indicating significant sensitivity to their setting.

6. Empirical Impact and Benchmark Results

Neighbor-constrained 4D Gaussian Splatting frameworks report consistent improvements over baseline or unconstrained 4DGS methods across standard metrics and datasets:

• PSNR gains ranging from +0.8 dB to +6.8 dB, SSIM improvements up to +0.02, and LPIPS reductions of ~0.01–0.02 have been demonstrated across D-NeRF, HyperNeRF, Neu3D, and real-world outdoor datasets (Deng et al., 2024, Zhou et al., 7 Aug 2025, Leea et al., 26 Mar 2026, Li et al., 2024, Li et al., 28 Nov 2025).
• Temporal flicker and noise, as measured by optical flow AEPE, decrease significantly when state-space filtering and temporal smoothness are applied (e.g., ~30% reduction in (Deng et al., 2024)).
• In sparse-view or challenging scenarios, geometry-consistent and semantic schemes yield improvements in dynamic and static region fidelity, and allow efficient deployment on edge devices (Li et al., 28 Nov 2025).
• Adaptive splitting coupled to neighborhood consistency results in both increased reconstruction fidelity and a 25–30% reduction in primitive count (Zhou et al., 7 Aug 2025).

7. Significance and Prospective Developments

Neighbor-constrained 4D Gaussian Splatting unifies geometric, temporal, semantic, and data-driven priors into coherent frameworks that maintain both high local detail and global consistency in dynamic scene representations. The main significance is the ability to synthesize temporally stable, spatially accurate, and photorealistic dynamic views using moderate computational resources, without the need for external optical flow or dense ground-truth. Variants differ by their definition of "neighbor": time-adjacent Gaussians, spatial kNN, ray-wise co-blenders, or semantic peers.

Future research directions plausibly include:

• Integrating learned, adaptive neighbor definitions that exploit scene structure.
• Joint optimization of neighborhood size and weighting.
• Extending constraint formulations to handle topological events, occlusions, and multi-object interactions in long-term dynamic sequences.
• Tighter integration with learning-based multiview geometry and semantic segmentation modules.

Empirical evidence supports the necessity of neighbor constraints for robust dynamic reconstruction, especially in low-texture, sparse-input, or highly deformable environments. Their modularity allows seamless integration into existing 4DGS pipelines.