SplineGS: Robust Motion-Adaptive Spline for Real-Time Dynamic 3D Gaussians from Monocular Video (2412.09982v2)

Abstract: Synthesizing novel views from in-the-wild monocular videos is challenging due to scene dynamics and the lack of multi-view cues. To address this, we propose SplineGS, a COLMAP-free dynamic 3D Gaussian Splatting (3DGS) framework for high-quality reconstruction and fast rendering from monocular videos. At its core is a novel Motion-Adaptive Spline (MAS) method, which represents continuous dynamic 3D Gaussian trajectories using cubic Hermite splines with a small number of control points. For MAS, we introduce a Motion-Adaptive Control points Pruning (MACP) method to model the deformation of each dynamic 3D Gaussian across varying motions, progressively pruning control points while maintaining dynamic modeling integrity. Additionally, we present a joint optimization strategy for camera parameter estimation and 3D Gaussian attributes, leveraging photometric and geometric consistency. This eliminates the need for Structure-from-Motion preprocessing and enhances SplineGS's robustness in real-world conditions. Experiments show that SplineGS significantly outperforms state-of-the-art methods in novel view synthesis quality for dynamic scenes from monocular videos, achieving thousands times faster rendering speed.

• The paper introduces the Motion-Adaptive Spline (MAS), which uses cubic Hermite splines for efficient, continuous modeling of dynamic 3D Gaussian trajectories.
• SplineGS employs a robust joint optimization strategy for camera parameters and 3D Gaussians, eliminating the need for unreliable Structure-from-Motion preprocessing.
• Experimental results show SplineGS achieves superior rendering quality with significantly faster real-time speeds compared to state-of-the-art methods.

Overview of SplineGS: Robust Motion-Adaptive Spline for Real-Time Dynamic 3D Gaussians from Monocular Video

The paper "SplineGS: Robust Motion-Adaptive Spline for Real-Time Dynamic 3D Gaussians from Monocular Video" introduces a dynamic 3D Gaussian Splatting framework named SplineGS, designed for novel spatio-temporal view synthesis from monocular video inputs. This research addresses the compelling challenges in synthesizing novel views from dynamic scenes with substantial efficiency and accuracy, eliminating the dependence on pre-computed camera parameters such as those derived from COLMAP, which are often unreliable for dynamic, real-world scenarios.

Key Contributions

The primary contributions of SplineGS are encapsulated as follows:

• Motion-Adaptive Spline (MAS) Methodology: Central to SplineGS is the innovative MAS, which employs cubic Hermite splines to effectively model continuous dynamic trajectories of 3D Gaussian objects with minimal computational overhead. This marks a significant progression from traditional static or fixed-degree polynomial trajectories.
• Dynamic Gaussian Deformation Modeling: Utilizing MAS, the framework introduces Motion-Adaptive Control points Pruning (MACP) which strategically prunes control points during dynamic deformation modeling. This maintains the qualitative integrity of dynamic modeling while optimizing efficiency for varied motion complexities.
• Joint Optimization Strategy: SplineGS incorporates a dual-stage optimization process for estimating both camera parameters and 3D Gaussian attributes. By leveraging photometric and geometric consistency, the method ensures robust intrinsic parameter estimation without the need for faulty Structure-from-Motion preprocessing.

Experimental Validation

The empirical results delineate the effectiveness of SplineGS compared to state-of-the-art methods across multiple datasets, including NVIDIA and DAVIS:

• Quality of Novel View Synthesis: SplineGS demonstrates superior performance in novel view rendering quality, achieving an impressive average PSNR improvement over existing methods. Specifically, it reported a PSNR that is 1.1 dB higher with renderings up to 8,000 times faster than competing methodologies.
• Rendering Efficiency: The results highlight the framework's capability for real-time rendering at significantly high speeds, achieving up to 400 FPS on lower resolution datasets.
• Temporal Consistency: SplineGS achieves marked improvements in temporal consistency, as evidenced by visual motion tracking results where dynamic 3D Gaussian trajectories are modeled with greater accuracy and continuity over time.

Practical and Theoretical Implications

The implementation of SplineGS has several practical implications, such as enabling higher fidelity and faster rendering capabilities for applications in virtual reality (VR) and augmented reality (AR), where real-time performance is crucial. Theoretically, SplineGS enriches the domain of novel view synthesis with a robust alternative to the implicit volumetric rendering models that significantly increase computational inefficiency, presenting a viable pathway for real-time applications.

Future Prospects

Looking forward, additional exploration of joint optimization frameworks incorporating deblurring methods could further enhance the utility of SplineGS by directly addressing the quality degradation issues posed by motion blur in highly dynamic scenes. Furthermore, extending this framework to multi-view settings could enhance versatility and open up new avenues for real-time 3D reconstruction in more complex environments.

This paper exemplifies a meaningful stride in dynamically rendering monocular video inputs, reinforcing the potential and versatility of advanced spline-based modeling techniques in real-time computer vision applications.