EndoGS: Deformable Endoscopic Tissues Reconstruction with Gaussian Splatting (2401.11535v3)

Abstract: Surgical 3D reconstruction is a critical area of research in robotic surgery, with recent works adopting variants of dynamic radiance fields to achieve success in 3D reconstruction of deformable tissues from single-viewpoint videos. However, these methods often suffer from time-consuming optimization or inferior quality, limiting their adoption in downstream tasks. Inspired by 3D Gaussian Splatting, a recent trending 3D representation, we present EndoGS, applying Gaussian Splatting for deformable endoscopic tissue reconstruction. Specifically, our approach incorporates deformation fields to handle dynamic scenes, depth-guided supervision with spatial-temporal weight masks to optimize 3D targets with tool occlusion from a single viewpoint, and surface-aligned regularization terms to capture the much better geometry. As a result, EndoGS reconstructs and renders high-quality deformable endoscopic tissues from a single-viewpoint video, estimated depth maps, and labeled tool masks. Experiments on DaVinci robotic surgery videos demonstrate that EndoGS achieves superior rendering quality. Code is available at https://github.com/HKU-MedAI/EndoGS.

• The paper introduces EndoGS, a novel method that applies Gaussian Splatting to reconstruct deformable tissues from endoscopic video footage.
• It leverages depth-guided supervision and deformation fields within a multi-resolution framework to accurately model dynamic surgical scenes and handle occlusions.
• Experimental results on DaVinci robotic surgery videos demonstrate superior performance in rendering quality and speed compared to existing approaches.

Enhancing Deformable Tissue Reconstruction in Robotic Surgery with Gaussian Splatting

Method Overview

The paper introduces EndoGS, a novel technique employing Gaussian Splatting (GS) for the reconstruction of deformable tissues in endoscopic video footage, targeted at applications in robotic surgery. By incorporating deformation fields and depth-guided supervision, EndoGS effectively deals with the dynamics of surgical scenes and tool occlusion issues, resulting in high-fidelity 3D reconstructions from a single viewpoint. This method builds upon the foundation of 3D Gaussian Splatting, further extending its utility in the medical domain by specifically tackling the challenges associated with endoscopic video analysis.

Technical Contributions

Key contributions of this paper are as follows:

• Introduction of Gaussian Splatting for Medical 3D Reconstruction: It pioneers the application of Gaussian Splatting in the medical imaging field, showcasing its potential for endoscopic surgical procedures.
• Dynamic Scene Modeling: The method models surgical scenes as combinations of static and deformable parameters over time, leveraging a multi-resolution voxel plane approach for encoding spatial-temporal information.
• Depth-Guided Supervision and Occlusion Handling: It innovates by incorporating depth estimation and spatiotemporal weighting masks for occlusion handling, alongside total variation regularization to maintain quality across dimensions.

Implementation & Optimization

The EndoGS pipeline demonstrates a structured approach to deformable tissue reconstruction:

1. The initial 3D Gaussian models capture the static scene structure.
2. Deformation fields, modeled via MLPs, account for dynamic scene changes over time.
3. Spatial-temporal weight masks, combined with depth maps, enhance the model's learning from visible tissue areas while addressing tool occlusions.
4. The model employs depth-guided and total variation losses to regulate the 3D reconstruction process, ensuring both local coherence and global accuracy.

Experimental Evaluation

Evaluations on DaVinci robotic surgery videos revealed that EndoGS outperforms existing methods in rendering quality significantly. With respect to image quality metrics (PSNR, SSIM, LPIPS) and rendering speed (FPS), EndoGS presented superior performance, affirming its efficacy and efficiency for real-time applications. Ablation studies further validated the importance of depth regularization and spatial total variation loss, underscoring their roles in enhancing the reconstruction quality and consistency.

Future Directions and Limitations

Despite its advantages, the technique encounters challenges related to the single-viewpoint nature of the input videos, which may limit the method's applicability for more complex surgical tasks. Potential improvements could involve exploring multi-viewpoint reconstruction approaches and introducing surface-oriented Gaussian optimization strategies to refine the model's accuracy and reliability.

Conclusion

EndoGS represents a substantive advance in the reconstruction of deformable tissues from endoscopic video footage. By adeptly addressing the intricacies of dynamic scene modeling and occlusion, the method sets a new standard for 3D visualization in robotic surgeries. Its implications extend beyond immediate surgical applications, offering avenues for enhanced surgical training, planning, and simulation technologies. Future research will undoubtedly expand on this foundation, further unlocking the potential of Gaussian Splatting in medical imaging and beyond.