FreeTimeGS Parametrization in Dynamic Scene Rendering

• FreeTimeGS Parametrization is a dynamic framework that extends spatiotemporal Gaussian splatting by using 4D Gaussian primitives to capture evolving scene geometry, appearance, and motion.
• It leverages neural deformation fields and hybrid representations to enable high-fidelity view synthesis, sensor calibration, and efficient video compression.
• The approach employs self-supervised photometric, depth, and physics-informed losses to ensure temporal coherence and realistic, robust dynamic reconstructions.

Spatiotemporal Gaussian Splatting is an explicit, differentiable framework for representing, reconstructing, and rendering dynamic (time-varying) scenes or physical fields as an ensemble of parameterized Gaussian primitives whose properties evolve in space and time. In contrast to static 3D Gaussian Splatting, spatiotemporal variants extend the parametric domain of each Gaussian to capture shape, appearance, and motion across both spatial and temporal dimensions, enabling high-fidelity novel view synthesis and downstream tasks such as calibration, super-resolution, and physically informed simulation.

1. Mathematical Foundations and Core Representation

The foundational unit in spatiotemporal Gaussian splatting is the anisotropic Gaussian function parametrized over both spatial and temporal axes. For static 3D scenes, each primitive is characterized by a center $\mu \in \mathbb{R}^3$, covariance $\Sigma \in \mathbb{R}^{3 \times 3}$ (usually decomposed into rotation and diagonal scales), spherical-harmonic (SH) color coefficients, and an opacity scalar. In dynamic (spatiotemporal) settings, these parameters are generalized as follows:

• 4D Gaussian primitive: For space-time $(x, t) \in \mathbb{R}^3 \times \mathbb{R}$,

$$G(x, t) = w\, \exp\left(-\frac{1}{2} \begin{bmatrix} x - \mu \ t - \tau \end{bmatrix}^\top \Sigma^{-1} \begin{bmatrix} x - \mu \ t - \tau \end{bmatrix} \right)$$

with $w$ a weight, $\mu \in \mathbb{R}^3$ spatial mean, $\tau \in \mathbb{R}$ temporal mean, $\Sigma \in \mathbb{R}^{4 \times 4}$ a full spatiotemporal covariance (Li et al., 28 Nov 2025), and associated vector-valued color and opacity. 3D-only dynamic variants use time-dependent deformations for means, scales, and rotations per primitive (Zhu et al., 21 Jan 2024, Lee et al., 21 Oct 2024).

• Deformation fields: Rather than fixed trajectories, dynamic scenes employ neural or structured deformation fields $D(x, t)$ to parameterize temporal evolution, implemented via plane-factorized grids (“HexPlanes” or K-planes) and small MLPs (Zhu et al., 21 Jan 2024, Yu et al., 27 Mar 2025). Temporal “deltas” are then injected into each Gaussian’s attributes, allowing differentiable, data-driven nonrigid motion.
• Hybrid representations: Many frameworks combine 3D (static) and full 4D (space-time) Gaussians, converting temporally stable primitives to 3D-only to improve efficiency without sacrificing fidelity (Oh et al., 19 May 2025).
• Other parameter reduction: In high dimensions (e.g., 4D flow MRI), axes-aligned covariances are preferred for tractability and convergence guarantees, reducing per-primitive parameter count (Jo et al., 14 Nov 2025).

2. Rendering, Compositing, and Differentiability

Rendering spatiotemporal Gaussians extends standard 3D splatting along several axes:

• 4D-to-3D slicing: At a query time $t_0$, each 4D Gaussian is sliced (“conditioned”) to generate a corresponding 3D Gaussian ellipsoid at that instant. The temporal decay factor models how influential each primitive is at time $t_0$ (Li et al., 28 Nov 2025).
• Alpha compositing: The resulting 3D primitives are projected into the camera plane, yielding 2D elliptical splats whose projected opacity and color (possibly view- or time-dependent) are blended along each ray in a front-to-back sorted order. The final pixel color is

$$C = \sum_{i} c_i(t)\,\alpha_i(t) \prod_{j<i} (1-\alpha_j(t))$$

where $c_i(t)$ is the SH-composed color, and $\alpha_i(t)$ is the projected opacity of the $i$th splat at time $t$ (Li et al., 28 Nov 2025, Zhu et al., 21 Jan 2024).

• Differentiable splatting: The entire rendering process is fully differentiable, enabling backpropagation through not only color and shape parameters, but also through deformation fields and hierarchical feature grids (Zhu et al., 21 Jan 2024, Yu et al., 27 Mar 2025).

3. Optimization and Supervision Strategies

Spatiotemporal Gaussian Splatting frameworks employ a diverse mix of self-supervised and auxiliary losses for robust scene discovery, geometric accuracy, and temporal coherence:

• Photometric and Structure Losses: Core optimization is driven by per-pixel $L_1$ or $L_2$ distance between rendered and observed images, sometimes combined with SSIM (Zhu et al., 21 Jan 2024, Lee et al., 6 Mar 2025, Li et al., 28 Nov 2025).
• Depth and Geometry Regularization: When sparse or noisy depths are available (e.g., from stereo or monocular estimators), geometry-consistent losses are employed, including structure losses (smoothed $L_1$ between rendered and observed depth), global depth ranking, and local patch normalization for spatiotemporal consistency (Li et al., 28 Nov 2025).
• Temporal and Surface Constraints: Temporal total variation (TV) penalties smooth transitions across frames; surface-aligned SDF and normal-consistency terms tighten Gaussian support onto observed tissue or object surfaces (Zhu et al., 21 Jan 2024).
• Deformation and Physics Priors: For scenes with rigid or nearly-rigid motion, acceleration-consistency constraints grounded in Newtonian mechanics enforce plausible and smooth object trajectories (Xu et al., 4 Aug 2025, Xu et al., 21 Nov 2025). Kalman filtering is often used to fuse pose estimates from photometric, optical flow, and event-camera data, correcting for drift and noise.
• Compression and Pruning: Deformation-aware pruning discards Gaussians with negligible motion or low photometric importance (Liu et al., 23 Jun 2024, Javed et al., 7 Dec 2024). Gradient-aware mixed-precision quantization and trajectory simplification (Ramer–Douglas–Peucker-based) further compress time-varying attributes for lightweight deployment (Javed et al., 7 Dec 2024).

4. Applications and Domain-Specific Innovations

Spatiotemporal Gaussian Splatting underpins a rapidly growing set of applications and is being continually extended by domain-driven innovations:

• Dynamic View Synthesis and Video Compression: GC-4DGS demonstrates high-fidelity rendering from a handful of input views through geometry-consistent supervision (Li et al., 28 Nov 2025). Temporally compressed splatting enables efficient real-time video encoding and decoding with up to $67\times$ compression (Javed et al., 7 Dec 2024).
• Deformable Medical Reconstruction: EndoGS leverages HexPlanes and deformation-aware supervision for real-time surgical tissue modeling from single-view video, achieving superior rendering under occlusion and complex dynamics (Zhu et al., 21 Jan 2024). X$^2$-Gaussian extends continuous-time Gaussian splatting to dynamic 4D CT, introducing self-supervised periodic losses to learn physiological breathing cycles (Yu et al., 27 Mar 2025).
• Sensor Calibration and Fusion: 3DGS-Calib performs joint spatial and temporal calibration of LiDAR–camera rigs, exploiting the speed and differentiability of Gaussian Splatting and achieving sub-degree, sub-10 cm, and sub-10 ms alignment in minutes (Herau et al., 18 Mar 2024).
• Physics-Informed Flow and Motion Recovery: PINGS-X applies normalized axes-aligned spatiotemporal splatting to super-resolve 4D flow MRI, achieving convergence guarantees, parameter efficiency, and rapid training, outperforming PINN and neural-operator baselines (Jo et al., 14 Nov 2025). PEGS incorporates Newtonian acceleration constraints, event streams, and adaptive annealing for robust rigid-body tracking over large spatiotemporal spans (Xu et al., 21 Nov 2025).
• Spatiotemporal Disentanglement: STD-GS introduces explicit decomposition of static and dynamic regions using frame-event-driven clustering and event-based priors, enhancing motion reconstruction in high-dynamic scenes (Zhou et al., 29 Jun 2025).

5. Efficiency-Driven Representational Schemes

The parameter and computational complexity of spatiotemporal splatting motivates a series of efficiency-focused innovations:

• Hybrid and Adaptive Models: Hybrid 3D–4D splatting adaptively converts temporally stable Gaussians to 3D-only representations, saving memory and accelerating training by $3$–$5\times$ with no loss in quality (Oh et al., 19 May 2025).
• Attribute and Feature Pruning: LGS realizes over $9\times$ compression in surgical reconstruction by aggressive pruning of (a) motion-insignificant Gaussians, (b) minimal-use spherical-harmonic color attributes, and (c) pooled and condensed 4D deformation fields (Liu et al., 23 Jun 2024).
• Explicit Dynamic Splatting: Fully Explicit Dynamic Gaussian Splatting (Ex4DGS) quantizes dynamic Gaussians at sparse keyframes, interpolating attributes in-between, with progressive dynamic/static separation and point-wise backtracking to cull spurious points—enabling memory-efficient, high frame-rate rendering (Lee et al., 21 Oct 2024).
• Compressed 2D/3D Video Models: For purely 2D or lower-dimensional time-varying data (e.g., GaussianVideo), deformable base Gaussians with efficient spatiotemporal encoders yield competitive PSNR at $5$–$12\times$ the speed and a fraction of the memory of NeRV baselines (Lee et al., 6 Mar 2025).

6. Limitations, Challenges, and Directions for Future Research

Several challenges persist:

• Sparse or Noisy Supervision: Geometry learning degrades with limited or inconsistent view data. Robust fusions of geometric and monocular/temporal priors, as in GC-4DGS, remain an ongoing area of paper (Li et al., 28 Nov 2025).
• Handling Nonrigid or Aperiodic Dynamics: Extensions to arbitrary nonrigid motion or aperiodic dynamics (e.g., pathological breathing, joint motion) may require richer deformation priors or online adaptation (Yu et al., 27 Mar 2025).
• Parameter Scaling: High-dimensional splatting (e.g., $q=4$ or more) is mitigated by axes alignment, but further parameter reduction and adaptive density control (splitting/merging strategies) are crucial for managing memory and ensuring convergence (Jo et al., 14 Nov 2025).
• Event-driven and Multi-sensor Fusion: Integrating data from asynchronous event streams, frame imagery, and additional sensors (e.g., depth, LiDAR) for robust, temporally coherent reconstructions, remains an open field (Xu et al., 21 Nov 2025, Zhou et al., 29 Jun 2025).
• Edge Deployability and Real-time Constraints: Hardware-aware acceleration, including efficient GPU splatting, quantized models, and feature condensation, are active areas for embedded and AIoT deployment (Javed et al., 7 Dec 2024, Liu et al., 23 Jun 2024, Li et al., 28 Nov 2025).

Ongoing research continues to generalize spatiotemporal Gaussian splatting to new domains, physically informed modeling, and hierarchical or graph-based primitive organizations, widening its utility for dynamic scene understanding, sensor fusion, and time-resolved scientific computing.