GSFusion: Online RGB-D Mapping Where Gaussian Splatting Meets TSDF Fusion (2408.12677v3)

Abstract: Traditional volumetric fusion algorithms preserve the spatial structure of 3D scenes, which is beneficial for many tasks in computer vision and robotics. However, they often lack realism in terms of visualization. Emerging 3D Gaussian splatting bridges this gap, but existing Gaussian-based reconstruction methods often suffer from artifacts and inconsistencies with the underlying 3D structure, and struggle with real-time optimization, unable to provide users with immediate feedback in high quality. One of the bottlenecks arises from the massive amount of Gaussian parameters that need to be updated during optimization. Instead of using 3D Gaussian as a standalone map representation, we incorporate it into a volumetric mapping system to take advantage of geometric information and propose to use a quadtree data structure on images to drastically reduce the number of splats initialized. In this way, we simultaneously generate a compact 3D Gaussian map with fewer artifacts and a volumetric map on the fly. Our method, GSFusion, significantly enhances computational efficiency without sacrificing rendering quality, as demonstrated on both synthetic and real datasets. Code will be available at https://github.com/goldoak/GSFusion.

• The paper introduces a hybrid mapping system that integrates TSDF fusion with 3D Gaussian splatting to achieve high-quality rendering and precise spatial understanding in real time.
• The methodology uses quadtree-based initialization to efficiently allocate Gaussians, significantly reducing computational load while preserving visual detail.
• Experiments on ScanNet++ and Replica show that GSFusion outperforms current methods in speed, rendering quality, and memory efficiency.

GSFusion: Online RGB-D Mapping Where Gaussian Splatting Meets TSDF Fusion

The paper "GSFusion: Online RGB-D Mapping Where Gaussian Splatting Meets TSDF Fusion" by Jiaxin Wei and Stefan Leutenegger presents an innovative approach to RGB-D mapping, leveraging both volumetric fusion techniques and advancements in 3D Gaussian splatting (3DGS). This paper particularly addresses the challenges related to real-time high-quality rendering and computational efficiency, which are critical for applications in AR/VR, robotics, and computer vision.

Methodology Overview

The authors propose GSFusion, a hybrid mapping system that integrates Truncated Signed Distance Field (TSDF) fusion with 3D Gaussian splatting. By integrating these two methodologies, the system constructs a dual-map representation: a 3D Gaussian map for high-quality rendering and a TSDF map for accurate spatial understanding. This dual-map approach ensures that the system can meet the geometric precision required for tasks like navigation and spatial reasoning while providing visually appealing renderings.

Key Innovations

1. Hybrid Mapping System:
   • TSDF Fusion: The authors utilize octree-based TSDF grid structures to capture detailed geometric information from RGB-D data frames. TSDF values are updated incrementally, ensuring real-time feasibility across complex scenes.
   • 3D Gaussian Splatting: GSFusion employs a novel scheme for Gaussian initialization that leverages image quadtree structures based on contrast detection. This significantly reduces the number of Gaussian primitives required, addressing the computational bottleneck that plagues traditional Gaussian splatting techniques.
2. Quadtree-Based Initialization:
   • By analyzing RGB images with quadtree segmentation, GSFusion efficiently allocates Gaussians at locations with significant visual contrasts. This approach ensures detailed and artifact-free renderings while maintaining a compact yet expressive map.
3. Efficient Online Optimization:
   • The optimization process for Gaussian parameters is enhanced by maintaining a keyframe list, which is periodically revisited to refine the map. This mitigates potential issues like map forgetting and overfitting, providing a balanced optimization throughout the scanning sequence.

Experimental Results

The authors conducted extensive evaluations on both synthetic (Replica) and real (ScanNet++) datasets, demonstrating the efficacy of GSFusion in terms of computational efficiency and rendering quality.

Notable Metrics and Comparisons

• Rendering Quality:
  • On the ScanNet++ dataset, GSFusion achieves an average PSNR of 28.84 and SSIM of 0.897 for training views after 10 iterations of global optimization, surpassing both SplaTAM and RTG-SLAM in visual fidelity.
  • On the Replica dataset, GSFusion matches closely with RTG-SLAM, achieving average PSNR, SSIM, and LPIPS metrics of 34.65, 0.949, and 0.056, respectively, after global optimization.
• Efficiency:
  • The system substantially outperforms existing methods in mapping speed, demonstrating an average frame rate of 6.14 fps on ScanNet++, which is at least five times faster than RTG-SLAM and 30 times faster than SplaTAM.
  • Memory Usage: GSFusion presents a significant reduction in model size (averaging 29.3 MB on ScanNet++), offering a compact representation without compromising quality.

Implications and Future Directions

The proposed GSFusion system sets a new benchmark for real-time RGB-D mapping, offering both high-quality rendering and computational efficiency. This makes it highly applicable for use in dynamic and resource-constrained environments such as mobile robotics, autonomous vehicles, and real-time AR/VR applications.

Future Directions

• Scale and Resolution: The integration of multi-resolution volumetric grids could be explored to extend the system's applicability to larger and more complex environments.
• Learning-Based Methods: Incorporating learning-based techniques could further optimize the mapping and rendering processes, potentially yielding improved adaptability to different scenes and sensor noise profiles.
• Hardware Accelerations: Leveraging advancements in hardware, such as specialized AI accelerators, could push the boundaries of real-time performance and visual quality even further.

In conclusion, GSFusion presents a robust framework that successfully combines the strengths of TSDF fusion and 3D Gaussian splatting, setting a foundation for further innovations in high-fidelity, real-time 3D mapping.