RetimeGS: Continuous-Time Reconstruction of 4D Gaussian Splatting

Abstract: Temporal retiming, the ability to reconstruct and render dynamic scenes at arbitrary timestamps, is crucial for applications such as slow-motion playback, temporal editing, and post-production. However, most existing 4D Gaussian Splatting (4DGS) methods overfit at discrete frame indices but struggle to represent continuous-time frames, leading to ghosting artifacts when interpolating between timestamps. We identify this limitation as a form of temporal aliasing and propose RetimeGS, a simple yet effective 4DGS representation that explicitly defines the temporal behavior of the 3D Gaussian and mitigates temporal aliasing. To achieve smooth and consistent interpolation, we incorporate optical flow-guided initialization and supervision, triple-rendering supervision, and other targeted strategies. Together, these components enable ghost-free, temporally coherent rendering even under large motions. Experiments on datasets featuring fast motion, non-rigid deformation, and severe occlusions demonstrate that RetimeGS achieves superior quality and coherence over state-of-the-art methods.

• The paper introduces a 4D scene representation that uses regularized temporal opacity windows and Catmull–Rom spline trajectories to reconstruct dynamic scenes.
• The methodology integrates bidirectional flow supervision, triple rendering, and dynamic stretching to mitigate ghosting and enhance temporal coherence.
• Experimental results demonstrate that RetimeGS outperforms baselines in PSNR, SSIM, and LPIPS, effectively handling rapid motion and occlusion challenges.

RetimeGS: Continuous-Time Reconstruction of 4D Gaussian Splatting

Motivation and Problem Formulation

Temporal retiming—generating temporally coherent novel frames at arbitrary timestamps from dynamic multi-view video—is essential for applications including slow-motion cinematography, VR rendering, and timeline manipulation in post-production. While recent advances in 4D Gaussian Splatting (4DGS) have enabled high-quality dynamic scene synthesis, existing approaches are restricted to reconstructing at discrete capture times. When queried for intermediate frames, these models frequently produce temporal aliasing and ghosting artifacts, especially under sparse temporal sampling or large non-rigid motion. This overfitting to frame indices fundamentally impedes the quality and coherence of continuous-time view synthesis.

Figure 1: Illustration of temporal overfitting to input frames $t_1$ and $t_2$, causing ghosting at $t_{1.5}$ in 4D primitive-based methods.

RetimeGS introduces a 4D scene representation and a suite of tailored supervisory strategies that address both temporal aliasing and trajectory estimation, aiming to achieve ghost-free, temporally consistent scene interpolation for arbitrary time queries. The architecture regularizes temporal opacity and incorporates smooth, spline-based primitive motion modeling, overcoming the limitations of both canonical-space deformation fields and frame-indexed 4D primitives.

Methodology

Core Representation

RetimeGS parametrizes each Gaussian primitive in space and time as:

• Temporal Support: A short-tailed, regularized opacity window defined by sigmoid functions bounded by learned temporal offsets relative to the primitive's temporal center.
• Spline Trajectory: The spatial position is governed by a Catmull–Rom spline, whose control structure is parameterized around adjacent frame velocities and supervised using bidirectional optical flow estimates.

  Figure 2: The RetimeGS pipeline leverages regularized temporal opacity and spline-based spatial positioning combined with explicit RGB and flow-based supervision for robust, continuous-time reconstruction.

At initialization, temporally-localized primitive groups are allocated around each inter-frame interval, each intended to jointly reconstruct two consecutive frames and associated intermediate frames within their support. To avoid temporal clustering and overfitting, a regime of dynamic stretching allows static primitives to extend their temporal support across multiple intervals if they are detected to be redundant, which optimizes primitive allocation efficiency.

Training Strategies

RetimeGS incorporates four synergistic components:

• Bidirectional Flow Trajectory Supervision: Supervises spline control point trajectories for each primitive using dense multi-view forward and backward optical flow fields, ensuring that interpolated positions remain consistent with coarse motion correspondences and mitigating the pairwise correspondence problem found in pure deformation-field methods.
• Triple Rendering Supervision: During training, for each input time $t_i$, RGB reconstruction losses are applied not only to the full set of primitives but also independently to each adjacent primitive group (those supported on intervals $[t_{i-1}, t_i]$ and $[t_i, t_{i+1}]$), enforcing that both sets are sufficient to explain the observed data at $t_i$. This mechanism dramatically reduces incomplete coverage and artifacts at intermediate times.
• Dynamic Stretching and Periodic Relocation: Static primitives may dynamically extend their temporal lifespan if they match static neighbors and satisfy low motion constraints. The Markov Chain Monte Carlo (MCMC)-based scheduled relocation encourages the overall primitive distribution to adaptively cover dynamic, hard-to-model regions with higher density.
• Flow-Aware Initialization: 3D primitive parameters are initialized using average multi-view 3D optical flow computed from projected dense flow fields, providing a strong geometric prior and preventing local optima associated with poor primitive placement.

Experimental Results

RetimeGS was validated across the DNA-Rendering and self-captured Stage-Capture datasets, which feature significant inter-frame motion, non-rigid deformations, partial occlusions, and challenging textural or visibility variations.

Figure 3: Qualitative comparison on the DNA-Rendering dataset and the in-house Stage-Capture set, highlighting superior artifact reduction and motion fidelity.

Quantitatively, RetimeGS achieves leading performance across PSNR, SSIM, and LPIPS, outperforming all compared baselines, including canonical deformation-based 4DGS (“Deform-GS”), unconstrained primitive-based methods (STGS, GaussianFlow), and lifted 2D interpolation (FILM+STGS).

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Deform-GS	28.45	0.867	0.0272
STGS	25.34	0.825	0.0357
GaussianFlow	25.91	0.825	0.0339
2D Lifting (FILM+STGS)	28.79	0.886	0.0267
RetimeGS	30.08	0.904	0.0225

RetimeGS uniquely suppresses both ghosting (opacity overfit) and trajectory drift, especially in regions with rapid appearance change or occlusion/disocclusion events where previous methods systematically fail or blur the reconstructed sequence.

Ablation and Analysis

Ablation studies validate the necessity of each architectural and training strategy:

• Flow supervision and initialization are critical for maintaining texture and motion fidelity in fast-moving regions, with their removal resulting in pronounced distortion.
• Triple rendering is required to avoid incomplete reconstructions at interval centers, as adjacent primitive sets localize information spatially without this constraint.

Figure 4: Qualitative ablation showing degradation without flow-based initialization and supervision.

Figure 5: Dynamic stretching visualized—magenta indicates temporally stretched (static) primitives, teal dynamic—revealing efficient primitive allocation.

• Spline trajectory modeling avoids sharp, unrealistic velocity changes, and heatmap visualizations confirm reduced errors in moving object boundaries over linear trajectory baselines.

  Figure 6: Error heatmaps demonstrate that spline trajectories yield more precise motion for occluding boundaries.

• Failure case analysis demonstrates degradation at extremely low FPS, where even robust bidirectional flow cannot provide sufficient primitive correspondence. As sampling sparsity increases, ghosting and misassociation become severe.

  Figure 7: With extremely sparse capture, optical flow supervision breaks down and RetimeGS produces erroneous intermediates.

Discussion and Future Directions

RetimeGS advances the state-of-the-art in continuous-time dynamic scene reconstruction, providing a tractable, robust path toward temporally coherent retiming under challenging settings. By unifying parametric temporal regularization, physically plausible primitive trajectories, and flow-guided supervisory signals, this model effectively bridges the weaknesses of prior deformation-field and uncontrolled primitive-based dynamic splatting.

On the practical side, the method yields smooth and artifact-free interpolation suitable for high-demand applications in VR, broadcast, and visual effects. However, the intrinsic dependency on accurate optical flow—and its failure under high-motion, low FPS scenarios—hints at a limit shared with many contemporary methods. Future extensions might consider self-supervised or motion-prior-guided approaches for coarse trajectory initialization, multi-segment global optimization for single-primitive support, or integration of explicit scene semantics for even more robust tracking under occlusion or disocclusion.

Figure 8: Generalizes to scenes with challenging opacity changes and non-stage/capture conditions, indicating model flexibility.

Conclusion

RetimeGS establishes a new paradigm for reconstructing and rendering high-fidelity dynamic scenes at arbitrary timestamps from multi-view video. The design’s explicit regularization of temporal support, sophisticated trajectory modeling, and suite of targeted supervisory signals yield quantifiable improvements in both fidelity and temporal coherence relative to existing baselines. While limitations remain in regimes of extreme temporal sparsity, the methodology provides a solid groundwork for future research on generalizable, temporally continuous dynamic scene representations (2603.13783).