DSD-GS: Dynamic-Static Decomposition of Gaussian Splatting for Efficient and High-Fidelity Dynamic Scene Reconstruction

Abstract: Dynamic scene reconstruction and novel view synthesis are fundamental to next-generation visual intelligence applications such as virtual reality, robotics, and digital twins. However, high-fidelity reconstruction of complex, time-varying scenes from arbitrary viewpoints remains a significant challenge. Existing dynamic 3DGS methods suffer from computational inefficiency, since they model all Gaussians as dynamic components. While recent decomposition-based approaches address this issue, they still struggle with degraded reconstruction quality and prolonged training time. To mitigate these limitations, we propose a novel dynamic reconstruction framework built upon an efficient static-dynamic decomposition strategy using a Feed-Forward Gaussian Splatting encoder and an optical flow model. By eliminating redundant computations on static regions, our method achieves state-of-the-art performance, outperforming existing baselines across rendering quality, training and rendering speed, and storage efficiency. Notably, on the Neural 3D dataset, our framework requires only 10 minutes for training and achieves a rendering speed of over 700 FPS on a single NVIDIA RTX 5090 GPU at resolution of 1352x1014. Furthermore, our decomposition strategy eliminates the need for COLMAP preprocessing and enables deterministic initialization, thereby enhancing both efficiency and reproducibility.

• The paper introduces a deterministic feed-forward Gaussian encoder that replaces stochastic COLMAP for efficient initialization.
• It employs optical flow-based per-pixel decomposition to accurately separate static and dynamic scene components.
• Experimental results show improved PSNR, rendering speeds over 700 FPS, and training time reductions up to 4× compared to baselines.

DSD-GS: Dynamic-Static Decomposition of Gaussian Splatting for Efficient and High-Fidelity Dynamic Scene Reconstruction

Introduction

Dynamic scene reconstruction and novel view synthesis (NVS) present significant challenges, particularly in achieving high fidelity and efficiency for 3D environments with time-varying content. Traditional approaches based on Neural Radiance Fields (NeRF) offer photorealistic quality but are computationally prohibitive for real-time applications due to the overhead of implicit MLP-based rendering. The introduction of 3D Gaussian Splatting (3DGS) methodologies enabled explicit, point-based, and differentiable scene representation, previously limited to static settings.

Recent dynamic 3DGS variants extended applicability to spatio-temporal data but suffered from critical inefficiencies by modeling all Gaussians as dynamic. This approach yielded elevated computational and storage footprints and degraded background stability. Initial dynamic-static decomposition attempts in the literature improved efficiency but compromised on reconstruction quality and further extended training time. The work addressed here proposes Dynamic-Static Decomposing Gaussian Splatting (DSD-GS), leveraging a deterministic, optimization-free, feed-forward Gaussian Splatting encoder together with optical flow-based per-pixel decomposition, and introduces architectural and algorithmic innovations that collectively yield state-of-the-art quality and efficiency.

Figure 1: The DSD-GS approach achieves high-fidelity rendering, recovers fine details (left), and quantitatively improves PSNR, training time, and render FPS (right) compared to baselines.

Methodology

Feed-Forward Gaussian Initialization and Decomposition

DSD-GS eliminates dependence on stochastic and costly COLMAP-based point cloud initialization by employing a pre-trained, pixel-aligned, feed-forward Gaussian Splatting (FFGS) encoder. Given multi-view images, the method establishes a deterministic one-to-one mapping between image pixels and 3D Gaussian primitives, resulting in a dense, pixel-aligned initial point cloud within a single forward pass. View selection utilizes geometric similarity or proximity in pose space, contingent on camera configuration, to optimize the input provided to FFGS, thereby improving downstream depth estimation and initialization stability.

The static-dynamic decomposition is accomplished immediately post-initialization via an optical flow model (e.g., FlowFormer), classifying each Gaussian as static or dynamic based on per-pixel displacement magnitudes extracted from reference and temporally sampled frames, with robust handling of both fixed and free camera trajectories through RANSAC-based motion compensation. This architectural decision introduces negligible overhead ($\mathcal{O}(1)$ seconds total), obviates learning overhead, and removes initialization variance.

Figure 2: Overview of the DSD-GS pipeline: initialization, dynamic-static decomposition, and separate static/dynamic optimization flows.

Figure 3: Visualization of dynamic-static decomposition results, demonstrating effective separation, especially when using $\Delta t$ interval optical flow sampling.

Gaussian Representation and Optimization

Static regions are modeled with reduced 0th-order SH color basis for memory and compute efficiency, while dynamic regions adopt time-dependent parametric functions for geometry and opacity. All primitives in dynamic regions utilize RBF-based temporal presence modeling and polynomial parameterizations for spatial/rotational trajectories, removing the need for auxiliary neural motion predictors and enabling fast frame-wise updates.

A two-phase optimization schedule is implemented. Initially, both static and dynamic primitives are jointly optimized in a time-independent fashion to refine scene geometry and attributes. Static Gaussian parameters are subsequently frozen. Edge-Detection-based Density Control Limit (ED-DCL) leverages Sobel filtering for data-driven densification constraints, preventing overfitting, limiting unnecessary point proliferation, and controlling final model footprint.

The second phase trains only the dynamic subset with fixed static background, utilizing a minimal loss function (weighted sum of $\mathcal{L}_1$ and D-SSIM terms), with Static-Caching Rasterization (SCR) replacing redundant per-frame sorting and compositing of unchanged static primitives with view-wise caching of static background layers, composited via dynamic foreground transmittance.

Experimental Results

Quantitative and Qualitative Evaluation

On the Neural 3D [dynerf] and HyperNeRF datasets, DSD-GS establishes strong improvement across PSNR, SSIM, LPIPS, training time, rendering FPS, and storage. The approach consistently surpasses the 30 dB PSNR threshold in real, complex dynamic scenes, with training time reductions of up to $4\times$ compared to the best prior explicit methods and rendering speeds exceeding 700 FPS at over 1MP resolution on RTX 5090-class hardware.

Figure 4: Qualitative comparison between DSD-GS and recent baselines on Neural 3D and HyperNeRF datasets, emphasizing static background fidelity and temporal coherence.

Figure 5: Performance stability and stochasticity comparison with 4DGS, illustrating the effect of initialization randomness and the PSNR variance across trials.

Figure 6: DSD-GS exhibits minimal inter-run PSNR variance and superior stability compared to dynamic 3DGS baselines.

Ablation studies demonstrate that disabling dynamic-static decomposition results in severe degradation of background accuracy, significant training slowdowns, and doubled storage requirements, establishing the necessity of the architecture. Edge density-based control and view selection further suppress spurious densification, enhancing compactness and trainability. Static-caching provides an additional $2\times$ FPS gain by eliminating redundant work on invariant background primitives.

Analysis of Decomposition and Memory Overhead

Figure 7: Per-frame dynamic-static decomposition on diverse scenes, confirming correct isolation of nonrigid objects against varying backgrounds.

Figure 8: Densification trends: nearest-pair view selection (NPS, orange) produces more stable Gaussian counts and avoids the explosive densification observed with random camera pair inputs (blue).

Despite high overall Gaussian counts due to pixel-aligned initialization, the final model remains more compact than all direct dynamic baselines except recent theoretical minima exploiting advanced pruning [gaussianspa, mini-splatting]. The static caching approach induces a marginal additional memory requirement ($\sim$0.3 GB at 1MP and 20 views) but scales linearly with the number of views and resolution, highlighting a potential bottleneck in future large-scale or multi-view deployments.

Figure 9: Temporal consistency of static background PSNR across all baseline methods, with DSD-GS obtaining the highest average and lowest variance (over 70 dB).

Limitations

Current limitations include sub-optimal separation in free-camera datasets due to residual errors in motion compensation and a tendency for excessive Gaussian proliferation in high-resolution scenes—a direct result of the pixel-aligned feed-forward initialization policy. Static-caching, while crucial for accelerating rasterization, places additional memory demands that grow with view count and resolution. Integration with advanced, learning-based camera motion disambiguation, hybrid initialization strategies, or adaptive multi-scale Gaussian selection mechanisms would be promising future directions.

Theoretical and Practical Implications

The explicit, deterministic split of static and dynamic regions in DSD-GS introduces a principled, per-pixel motion segmentation paradigm into dynamic 3DGS pipelines, essentially decoupling the learning and compositing of time-invariant and time-variant elements. This structural checkpointing leads to algorithmic reproducibility, reliable convergence, and substantial reductions in spurious temporal artifacts (e.g., background flicker, geometric instability). Practically, the method advances the deployment of high-speed, high-fidelity dynamic NVS for real-world AR/VR and robotics, and further lowers the barrier for reproducible research and application benchmarking by removing stochastic dependencies in pipeline initialization.

Conclusion

DSD-GS provides an efficient, high-fidelity solution for dynamic 3D scene reconstruction by effectively decoupling static and dynamic regions via deterministic pixel-aligned initialization and fast optical flow-based decomposition. By adopting explicit time-dependent parameterizations and static caching, it achieves both state-of-the-art image quality and unprecedented computational efficiency. The methodology's design introduces favorable theoretical properties and practical scalability, with future work warranted in free-camera domain adaptation and initialization compactness.

---

References:

• "DSD-GS: Dynamic-Static Decomposition of Gaussian Splatting for Efficient and High-Fidelity Dynamic Scene Reconstruction" (2605.30863)
• Supporting works: [3dgs], [dynerf], [4dgs], [stg], [swift4d], [degauss], [gaussianspa], [mini-splatting]