LIV-GaussMap: LiDAR-Inertial-Visual Fusion for Real-time 3D Radiance Field Map Rendering (2401.14857v2)

Abstract: We introduce an integrated precise LiDAR, Inertial, and Visual (LIV) multimodal sensor fused mapping system that builds on the differentiable \pre{surface splatting }\now{Gaussians} to improve the mapping fidelity, quality, and structural accuracy. Notably, this is also a novel form of tightly coupled map for LiDAR-visual-inertial sensor fusion. This system leverages the complementary characteristics of LiDAR and visual data to capture the geometric structures of large-scale 3D scenes and restore their visual surface information with high fidelity. The initialization for the scene's surface Gaussians and the sensor's poses of each frame are obtained using a LiDAR-inertial system with the feature of size-adaptive voxels. Then, we optimized and refined the Gaussians using visual-derived photometric gradients to optimize their quality and density. Our method is compatible with various types of LiDAR, including solid-state and mechanical LiDAR, supporting both repetitive and non-repetitive scanning modes. Bolstering structure construction through LiDAR and facilitating real-time generation of photorealistic renderings across diverse LIV datasets. It showcases notable resilience and versatility in generating real-time photorealistic scenes potentially for digital twins and virtual reality, while also holding potential applicability in real-time SLAM and robotics domains. We release our software and hardware and self-collected datasets to benefit the community.

• The paper introduces LIV-GaussMap, a novel multimodal sensor fusion approach that integrates LiDAR, inertial, and visual data to construct precise Gaussian-based maps.
• It employs differentiable surface splatting and photometric gradient optimization to achieve high-fidelity, real-time 3D radiance field map rendering.
• Extensive evaluations on public datasets demonstrate superior robustness and performance over traditional SLAM systems, with open-source resources supporting further research.

Introduction

Recent advances in SLAM (Simultaneous Localization and Mapping) ushered in a new era of autonomous navigation solutions. However, traditional SLAM systems, constrained by single sensor capabilities, struggle with issues like sensitivity to light conditions or depth information fidelity. To circumvent these limitations, multimodal sensor fusion combines data from various sources such as cameras, LiDAR, and IMUs to improve the accuracy and robustness of maps.

System Overview

LiDAR-Inertial-Visual (LIV) systems represent a cutting-edge approach within this domain. A pivotal development in this space is the LIV-GaussMap - a LIV multi-modal sensor fused mapping system that achieves enhanced mapping fidelity through advanced methods such as differentiable surface splatting. A notable feature of this system is its tight coupling of map elements derived from LiDAR, visual, and inertial sensors. Initial Gaussian scene poses leverage a LiDAR-inertial system with size-adaptive voxels, which are then optimized using visual-derived photometric gradients. This process optimizes the quality and density of the environment reconstruction, working across both repetitive and non-repetitive LiDAR scanning modes.

Methodology and Results

The innovative LIV-GaussMap constructs dense, precise map structures using surface Gaussians generated through LiDAR-inertial measurements. It exploits the properties of ellipsoidal surface Gaussians, which account for unreasonable point cloud distributions that may occur due to challenging scan angles. By refining the Gaussian map structure with visual photometric gradients and optimizing the spherical harmonic coefficients, the system achieves real-time rendering of photorealistic scenes. This ability is vital for creating digital twins and virtual reality applications, as well as in the fields of SLAM and robotics. Rigorous testing on various public datasets confirms its robustness and performance, outperforming existing LiDAR-inertial visual systems, particularly on non-Lambertian surfaces.

Conclusion and Contributions

This paper presents a comprehensive summary of the contributions made by LIV-GaussMap. It showcases the construction of precise maps with Gaussian distributions, based on measurements from a LiDAR-inertial system, subsequently optimizing these constructs through advanced visual measurements that capture photometry across multiple viewpoints. The method shows applicability to a plethora of LiDAR types, facilitating structure construction and generating photorealistic renderings in real-time. Moreover, the authors have made their software, hardware, and datasets openly available on GitHub, thus aiding the broader research community. Finally, the system's efficacy in producing accurate, detailed representations marks a significant stride in multimodal sensor fusion in SLAM for both indoor and outdoor environments.