SGS-SLAM: Semantic Gaussian Splatting For Neural Dense SLAM (2402.03246v6)

Abstract: We present SGS-SLAM, the first semantic visual SLAM system based on Gaussian Splatting. It incorporates appearance, geometry, and semantic features through multi-channel optimization, addressing the oversmoothing limitations of neural implicit SLAM systems in high-quality rendering, scene understanding, and object-level geometry. We introduce a unique semantic feature loss that effectively compensates for the shortcomings of traditional depth and color losses in object optimization. Through a semantic-guided keyframe selection strategy, we prevent erroneous reconstructions caused by cumulative errors. Extensive experiments demonstrate that SGS-SLAM delivers state-of-the-art performance in camera pose estimation, map reconstruction, precise semantic segmentation, and object-level geometric accuracy, while ensuring real-time rendering capabilities.

• The paper introduces SGS-SLAM, which uses a 3D Gaussian Radiance Field to achieve rapid camera tracking and precise dense scene mapping.
• It leverages multi-channel optimization that integrates semantic, appearance, and geometric cues to deliver segmentation accuracy close to ground truth.
• The system enables disentangled object representation for precise scene editing without compromising mapping stability, benefiting robotics and mixed-reality applications.

Overview of SGS-SLAM

Semantic understanding is a pivotal component in the advancement of Dense Simultaneous Localization and Mapping (SLAM). As elaborated in the paper, the authors introduce SGS-SLAM, a novel system that marries semantic information with 3D Gaussian Splatting. The dominant approach in the field previously relied upon multi-layer perceptrons (MLPs) within NeRF-based methods, which struggled with detail at object edges due to over-smoothing, and suffered from efficiency issues particularly in large-scale environments.

The authors propose a shift towards a method leveraging a 3D Gaussian Radiance Field, which allows for rapid rendering and direct gradient flow, thus favoring efficiency and accuracy. They utilize multi-channel optimization, integrating semantic information with appearance and geometric constraints—an innovative approach aimed at enhancing both reconstruction quality and real-time rendering capabilities.

Advantages of SGS-SLAM

SGS-SLAM's key contributions are noteworthy:

1. A system based on 3D Gaussians provides swift camera tracking and scene mapping, differentiating it from MLP-based methods which produce over-smoothed effects at object boundaries. The new system achieves segmentation precision almost equivalent to ground truth data.
2. The integration of semantic maps which supervise parameter optimization and select key frames improves the quality of map reconstructions while optimizing camera tracking.
3. The method’s ability to disentangle object representation in a 3D scene lays a foundation for editing and manipulating specific scene elements without affecting the overall stability of scene rendering.

Performance Evaluation

The authors carry out comprehensive experiments to validate SGS-SLAM against existing methods, evaluating mapping, tracking, and semantic segmentation performance on both synthetic and real-world benchmarks. The results showcase clear advantages over NeRF-based approaches and neural implicit semantic SLAM systems. The method achieves superior rendering speeds and scene precision, and facilitated precise scene editing thanks to its disentangled 3D semantic representation.

Conclusion

SGS-SLAM stands as a significant contribution to SLAM literature, providing high-accuracy 3D semantic segmentation and high-fidelity dense map reconstruction, all while preserving a robust capability for real-time camera pose estimation. Its explicit volumetric representation utilizes 3D Gaussians and real-time switching between channels including color, depth, and semantic color. This method offers promising insights for robotics and mixed-reality applications due to its precise segmentation and efficient real-time performance. The capabilities for scene manipulation without retraining demonstrate the system's flexible utility in various practical scenarios.

Overall, SGS-SLAM presents not just a step forward for dense visual SLAM systems but also sets the stage for the development of highly accurate, efficient, and practical real-world SLAM applications.