RetimeGS: Continuous-Time 4D Gaussian Splatting

• RetimeGS is a continuous-time 4D Gaussian Splatting representation that uses temporally parameterized Gaussian primitives for smooth dynamic scene rendering.
• It leverages optical flow-guided trajectory initialization and supervision to accurately model fast motion, non-rigid deformations, and severe occlusions.
• Triple-rendering supervision and targeted temporal regularization yield artifact-free, high-fidelity reconstructions with superior PSNR, SSIM, and LPIPS metrics compared to prior methods.

RetimeGS is a continuous-time 4D Gaussian Splatting (4DGS) representation and reconstruction framework for dynamic scene rendering at arbitrary temporal resolutions. It was introduced to address the limitations of discrete-timestamp 4DGS approaches, specifically temporal aliasing and ghosting artifacts that arise during temporal interpolation. By explicitly parameterizing the temporal support and smooth trajectories of individual 3D Gaussian primitives, and employing flow-based supervision, RetimeGS enables temporally coherent, artifact-free high-quality frame synthesis across scenes exhibiting fast motion, non-rigid deformation, and severe occlusions (Wang et al., 14 Mar 2026).

1. Continuous-Time Parameterization and Rendering

RetimeGS extends conventional 4DGS by treating time as a continuous variable in the definition of each scene primitive. Each Gaussian primitive $p$ is assigned:

• Temporal parameters: Center $\mu_{\tau,p}$, left/right half-widths $\tau_{l,p}$ and $\tau_{r,p}$ define the temporal window of activation.
• Spatial trajectory: A pseudo-mean $\boldsymbol\mu_p$ (at $\mu_{\tau,p}$) and three velocity vectors $$\boldsymbol v_{1,p}, \boldsymbol v_{2,p}, \boldsymbol v_{3,p}$$ support cubic trajectory modeling.
• Attributes: Anisotropic scale $\boldsymbol s_p$, base opacity $\sigma_p$, and SH color coefficients $\boldsymbol h_p$.
• Time-varying rotation: $\boldsymbol q_p(t)$, typically a low-order polynomial in time.

Temporal support is governed by a short-tailed double sigmoid kernel:

$$\sigma_{\tau,p}(t) = \tilde\psi_l\!\left(\frac{t-(\mu_{\tau}-\tau_l)}{\gamma}\right)\;\tilde\psi_r\!\left(\frac{(\mu_{\tau}+\tau_r)-t}{\gamma}\right)$$

where $\gamma$ controls fade-in/out smoothness. Each primitive's spatial mean at time $t$ is given by a Catmull–Rom spline through four control points $p_0,...,p_3$, resulting in $C^1$-continuous, physically plausible object motion.

At any $t$, rendering follows the standard 3DGS alpha compositing: each primitive's spatial position, scale, and orientation are evaluated at $t$ and rendered with temporally weighted opacity, followed by depth sorting and blending.

2. Optical Flow-Guided Trajectory Initialization and Supervision

RetimeGS employs multi-view, bidirectional optical flow to initialize and supervise the trajectories of primitives:

• Initialization: A per-frame point cloud (computed by VGGT) is projected into each camera, with 2D forward/backward flows (WAFT) sampled and then back-projected to estimate initial velocities and means for all primitives.
• Supervision: For each interval $[t_{i-1}, t_i]$, 3D-2D flow is rendered (using Catmull–Rom control points) and compared with ground-truth per-pixel flows $\mathbf F^{\mathrm{fwd}}$, $\mathbf F^{\mathrm{bwd}}$ via an $L_1$ loss. Flow supervision accelerates learning of spatial trajectories and enforces physically realistic motion, especially for fast and non-rigid dynamics.
• Flow supervision loss is annealed during training to favor final photometric accuracy once trajectories are stable.

3. Triple-Rendering Supervision and Occlusion Handling

To address under-coverage and occlusion failures prevalent in single-composite supervision, RetimeGS introduces triple-rendering:

• For each interior frame $t_i$, supervise three separate renders:
  • $I_{\mathrm{all}}(t_i)$: All primitives.
  • $I_{\mathrm{prev}}(t_i)$: Only primitives active in $[t_{i-1}, t_i]$ (with opacity normalization).
  • $I_{\mathrm{next}}(t_i)$: Only primitives in $[t_i, t_{i+1}]$.

Each is supervised by a standard RGB loss,

$$L_{\mathrm{rgb}} = \sum_i \sum_{\mathrm{subset} \in \{\mathrm{all}, \mathrm{prev}, \mathrm{next}\}} \|I_{\mathrm{subset}}(t_i) - I^{\mathrm{gt}}(t_i)\|_2^2.$$

At scene boundaries, only applicable subsets are supervised. This prevents holes and ensures that spatial regions occluded or partially covered in one interval remain well-explained through adjacent intervals.

4. Targeted Temporal Regularization Strategies

RetimeGS incorporates additional mechanisms for robust, continuous-time modeling:

• Dynamic stretching: After initial convergence, primitives with zero or near-zero velocity and similar base color are stretched temporally to cover multiple intervals, increasing static-region efficiency and reducing redundancy.
• MCMC-based relocation: Short-duration, high-opacity primitives are stochastically relocated to under-modeled dynamic areas, judged by a score $s = \sigma/(\tau_l + \tau_r)$, prioritizing capacity allocation for challenging regions.
• Opacity and scale regularization: $L_{\sigma}$ and $L_s$ discourage overly opaque or spatially large primitives, promoting sparsity and improved disentanglement.
• Trajectory smoothness: Catmull–Rom spline parameterization achieves $C^1$ continuity, eliminating artifacts caused by linear-velocity transitions.
• Occlusion handling: Triple renderings ensure information is not lost under occlusion—adjacent-interval primitives fill in any missing content.

5. Experimental Evaluation and Results

RetimeGS is evaluated on standard dynamic scene datasets, including DNA-Rendering (10 clips, 60 cameras at 15 FPS) and the Stage-Capture dataset (9 scenes, 32 cameras at 22 FPS). Comparative baselines include Deformation-GS, STGS, GaussianFlow, 2D-lifting (FILM per-view then 4DGS), and Ex4DGS.

• Quantitative metrics: On the Stage-Capture benchmark, RetimeGS attains PSNR 30.08 dB (1.3 dB higher than the next-best method), SSIM 0.904, and LPIPS 0.0225.
• Qualitative results: Under fast, non-rigid motion (e.g., waving, dancing), RetimeGS produces coherent slow-motion frames with minimal ghosting, in contrast to competing methods which exhibit semi-transparent artifacts or blurred occlusions.
• Ablations: Disabling triple-rendering, flow supervision, or dynamic stretching leads to degraded texture quality, partial reconstructions, or increased primitive counts. Switching from spline-based to linear trajectories introduces velocity discontinuities.

A summary comparison:

Method	PSNR (dB)	SSIM	LPIPS
RetimeGS	30.08	0.904	0.0225
Deformation-GS	28.7	0.890	0.041
GaussianFlow	27.8	0.860	0.038
STGS	28.1	0.872	0.035

6. Significance, Limitations, and Future Directions

RetimeGS demonstrates that explicit continuous-time parameterization and flow-guided supervision are essential for temporally coherent 4DGS reconstruction in dynamic, real-world scenes. This approach achieves artifact-free retiming and slow-motion playback, outperforming frame-indexed or deformation-based approaches.

• Major significance: Mitigation of temporal aliasing and ghosting, high rendering quality under severe inter-frame displacements and deformations, and robustness in occlusion scenarios.
• Limitations: The method relies on high-quality optical flow and multi-view data; performance under poor flow estimates or with sparse camera coverage may degrade. The static assignment of polynomial degrees or interval widths, while robust in practice, could be further optimized.
• Future directions: Refinement of temporal modeling (e.g., learning temporal support adaptively), improved flow estimation, extension to monocular input or unsupervised setups, and scaling to longer sequences or larger scenes.

7. Relation to Broader 4DGS and Video Retiming Research

RetimeGS is positioned among a spectrum of recent advances in dynamic scene representation:

• Unlike RTGS (Li et al., 8 Oct 2025), which focuses on efficiency and real-time SLAM for static 3DGS on edge devices, RetimeGS is designed for temporal flexibility and high-quality rendering in dynamic/dense capture setups.
• It directly addresses temporal aliasing, a documented failure mode of previous interval-indexed or deformation-based 4DGS pipelines [wu20244d, li2024spacetime, gao2024gaussianflow].
• RetimeGS differs from purely frame-based video retiming methods (e.g., Video-ReTime (Jenni et al., 2022)) in that it operates on temporally continuous 3D scene reconstructions, not 2D frame subsampling.

The adoption of continuous-time primitive parameterization—combined with optical flow guidance, triple-rendering, and dynamic resource allocation—establishes RetimeGS as a reference point for temporally-aware 4DGS modeling and rendering, with implications for VFX, slow-motion cinematography, and dynamic scene analysis.