Gaussian Pancakes: Geometrically-Regularized 3D Gaussian Splatting for Realistic Endoscopic Reconstruction (2404.06128v2)

Abstract: Within colorectal cancer diagnostics, conventional colonoscopy techniques face critical limitations, including a limited field of view and a lack of depth information, which can impede the detection of precancerous lesions. Current methods struggle to provide comprehensive and accurate 3D reconstructions of the colonic surface which can help minimize the missing regions and reinspection for pre-cancerous polyps. Addressing this, we introduce 'Gaussian Pancakes', a method that leverages 3D Gaussian Splatting (3D GS) combined with a Recurrent Neural Network-based Simultaneous Localization and Mapping (RNNSLAM) system. By introducing geometric and depth regularization into the 3D GS framework, our approach ensures more accurate alignment of Gaussians with the colon surface, resulting in smoother 3D reconstructions with novel viewing of detailed textures and structures. Evaluations across three diverse datasets show that Gaussian Pancakes enhances novel view synthesis quality, surpassing current leading methods with a 18% boost in PSNR and a 16% improvement in SSIM. It also delivers over 100X faster rendering and more than 10X shorter training times, making it a practical tool for real-time applications. Hence, this holds promise for achieving clinical translation for better detection and diagnosis of colorectal cancer.

• The paper introduces a novel pipeline that integrates 3D Gaussian Splatting with RNNSLAM, achieving an 18% boost in PSNR and 16% in SSIM for endoscopic 3D reconstructions.
• It applies geometric and depth regularization to reduce artifacts, significantly enhancing photorealistic rendering quality in medical imaging.
• The method enables real-time rendering (over 100 FPS) and drastically reduces training times, advancing colorectal cancer diagnostics in clinical settings.

Gaussian Pancakes: Enhancing 3D Endoscopic Reconstructions with Geometric Regularization

Introduction

The detection and diagnosis of colorectal cancer remain a pivotal challenge in medical imaging, particularly regarding the limitations of conventional colonoscopy techniques. These limitations, such as a restricted field of view and insufficient depth information, significantly hinder the identification of pre-cancerous lesions. The novel approach presented, "Gaussian Pancakes," integrates 3D Gaussian Splatting (3D GS) with a Recurrent Neural Network-based Simultaneous Localization and Mapping (RNNSLAM) system. This integration not only introduces geometric and depth regularization into the 3D GS framework but also promises smoother and more accurate 3D reconstructions, exhibiting clear enhancements in novel view synthesis quality. Rigorous evaluations underscore its performance advantages over current leading methods, notably an 18% increase in PSNR (Peak Signal-to-Noise Ratio) and a 16% improvement in SSIM (Structural Similarity Index Measure).

Methodology

The presented methodology hinges on leveraging 3D GS for lifelike texture rendering and RNNSLAM for robust 3D reconstruction, starting with RNNSLAM-generated camera poses, depth maps, and meshes. The process involves:

• Integration with SLAM: A pioneering pipeline merges 3D GS with the SLAM system for photorealistic 3D reconstructions. This pipeline demonstrates robustness over conventional Structure-from-Motion (SfM) methods.
• Improvement of Base GS Method: Incorporation of geometric and depth regularizations aligns Gaussians with colon surfaces effectively, mitigating geometric error and artifacts in novel view synthesis.
• Advancements in Endoscopic Radiance Fields: Optimization of the training and rendering process for radiance fields in surgical scenes dramatically reduces training times and improves image rendering speeds, enhancing clinical applicability.

Experimental Evaluation

The evaluation encompassed three datasets—Simulation, In-Vivo, and Phantom—exhibiting our method's superior performance across a suite of metrics, notably in photometric errors (PSNR, SSIM, and LPIPS) and computational efficiency (rendering speed and training time). Notably, Gaussian Pancakes achieved rendering speeds greater than 100 FPS and reduced training times to less than 2% compared to other models across all datasets.

Results and Discussion

Gaussian Pancakes demonstrated marked improvements in photometric error measures and computational efficiency. Despite facing a slightly higher Depth MSE compared to REIM-NeRF, our method effectively managed data noise and prioritized high-quality image generation and smooth reconstructions. An ablation study further reflected the incremental benefits of each methodological enhancement.

Conclusion and Future Directions

Gaussian Pancakes represents a significant advancement in 3D endoscopic reconstruction, especially in the context of colorectal cancer diagnostics. By combining RNNSLAM's robust 3D reconstruction capabilities with geometrically and depth-regularized 3D GS, this method not only enhances 3D reconstruction accuracy and texture detail but also substantially reduces computational demands. Moving forward, efforts will focus on integrating SuperPoint for improved point cloud creation and refining depth map accuracy, aiming to further bolster the method's precision and clinical utility.

This contribution not only advances the state of 3D endoscopic reconstruction but also opens new avenues for research in medical imaging and AI-driven diagnostics, underlining the potential for more effective and efficient colorectal cancer detection strategies.