Space-time 2D Gaussian Splatting for Accurate Surface Reconstruction under Complex Dynamic Scenes (2409.18852v1)

Abstract: Previous surface reconstruction methods either suffer from low geometric accuracy or lengthy training times when dealing with real-world complex dynamic scenes involving multi-person activities, and human-object interactions. To tackle the dynamic contents and the occlusions in complex scenes, we present a space-time 2D Gaussian Splatting approach. Specifically, to improve geometric quality in dynamic scenes, we learn canonical 2D Gaussian splats and deform these 2D Gaussian splats while enforcing the disks of the Gaussian located on the surface of the objects by introducing depth and normal regularizers. Further, to tackle the occlusion issues in complex scenes, we introduce a compositional opacity deformation strategy, which further reduces the surface recovery of those occluded areas. Experiments on real-world sparse-view video datasets and monocular dynamic datasets demonstrate that our reconstructions outperform state-of-the-art methods, especially for the surface of the details. The project page and more visualizations can be found at: https://tb2-sy.github.io/st-2dgs/.

• The paper proposes a novel space-time 2D Gaussian splatting method that outperforms traditional volumetric approaches in reconstructing complex dynamic scenes.
• It introduces a compositional opacity deformation strategy and depth/normal regularizers to accurately recover occluded surfaces.
• Evaluation on CMU Panoptic and D-NeRF datasets confirms state-of-the-art accuracy, superior geometric fidelity, and efficient joint optimization over time.

Space-time 2D Gaussian Splatting for Accurate Surface Reconstruction Under Complex Dynamic Scenes

The paper "Space-time 2D Gaussian Splatting for Accurate Surface Reconstruction Under Complex Dynamic Scenes" by Shuo Wang et al. introduces a novel approach to tackle the challenges posed by accurate geometry reconstruction in dynamically complex environments. The paper addresses the limitations of traditional surface reconstruction methods, particularly when applied to scenes with multiple interacting entities and significant occlusions. By leveraging a space-time 2D Gaussian Splatting technique, the authors propose a method that significantly improves both geometric accuracy and rendering efficiency.

Introduction

Spatial and temporal variations in dynamic scenes introduce significant challenges in surface reconstruction. Traditional volumetric and mesh-based approaches either suffer from high computational costs or fail in maintaining geometric fidelity in the presence of occlusions and dynamic interactions. Neural rendering methods have made strides in generating detailed textures and 4D surfaces; however, they often involve extensive training times and substantial storage demands.

Key Contributions

The contributions of this paper can be summarized as follows:

1. Space-time 2D Gaussian Splatting: The authors present a spacetime 2D Gaussian Splatting approach, the first particle-based surface model explicitly designed for complex dynamic scene reconstruction. This model is faster than traditional volumetric density fields and offers higher geometric accuracy.
2. Opacity Deformation Strategy: They introduce a compositional opacity deformation strategy that handles occlusions effectively, ensuring the accurate recovery of occluded surfaces.
3. Joint Optimization Framework: The paper proposes a framework that optimizes both the canonical model and its deformations over multiple timestamps, achieving precise surface reconstruction for dynamic scenes.
4. Depth and Normal Regularizers: The approach employs depth and normal regularizers to enforce the placement of 2D Gaussian disks on object surfaces.

Methodology

Geometry and Opacity Deformation

The proposed methodology involves learning a canonical set of 2D Gaussian splats and employing a deformation network to adapt these Gaussians over time. The canonical Gaussians are parameterized and activated continuously to handle dynamic surface changes. The proposed time-varying opacity model specifically addresses occlusion challenges by retaining geometric precision and minimizing noise.

Regularization

To ensure that the deformed Gaussian disks closely align with object surfaces, the method incorporates depth and normal regularizers. Additionally, a foreground mask loss is implemented to prevent elongated Gaussians at object silhouettes, aiming for enhanced foreground-background separation.

Evaluation and Results

The authors evaluate their approach on both real-world (CMU Panoptic dataset) and synthetic (D-NeRF) datasets.

CMU Panoptic Dataset

The performance of the proposed method is compared against state-of-the-art methods including SDFFlow, Tensor4D, and 4DGS on the CMU Panoptic dataset, which features intricate dynamic scenes. The results demonstrate that the new approach outperforms previous methods, achieving:

• Accuracy: 10.1 mm (best)
• Completeness: 19.2 mm (second-best)
• Overall Chamfer Distance: 14.6 mm (best)

D-NeRF Dataset

On the synthetic D-NeRF dataset, which evaluates novel view synthesis, the method maintains quality on par with the best existing methods while offering higher reconstruction accuracy. The technique achieves comparable PSNR, SSIM, and LPIPS metrics while improving geometric fidelity.

Implications and Future Directions

The proposed Space-time 2D Gaussian Splatting offers significant implications for real-time dynamic scene reconstruction applications, potentially benefiting fields such as virtual reality, augmented reality, and dynamic scene rendering in gaming and simulation environments. The model's efficiency and scalability make it suitable for real-world applications where computational resources might be limited.

Future Work

The research opens several avenues for future exploration, including:

• Enhancing the deformation network for more complex interactions and movements.
• Extending the method to handle dynamic lighting conditions and more diverse scene compositions.
• Integrating the approach with real-time capture systems to enable live dynamic scene reconstruction.

Conclusion

The Space-time 2D Gaussian Splatting method proposed by Shuo Wang et al. demonstrates a substantial advancement in the field of dynamic surface reconstruction. By addressing the challenges of occlusions and dynamic content adaptation, the method achieves impressive geometric accuracy and efficiency. This work provides a robust foundation for further innovations in dynamic scene reconstruction and real-time rendering technologies.