LiDAR-Inertial-Camera SLAM: Robust 3D Mapping

• LiDAR-Inertial-Camera SLAM is a system that fuses data from LiDAR sensors, IMUs, and cameras to achieve real-time, high-fidelity localization and mapping.
• The framework employs continuous-time odometry with non-uniform B-splines and differentiable optimization to maintain accuracy even under sensor degradation.
• Advanced techniques like 3D Gaussian Splatting and CUDA acceleration ensure efficient, photorealistic map generation for autonomous robotics and aerial applications.

A LiDAR-Inertial-Camera SLAM system is a simultaneous localization and mapping (SLAM) framework that fuses information from LiDAR sensors, inertial measurement units (IMUs), and visual cameras to achieve robust, accurate, and real-time estimation of a platform's trajectory and environment map. These systems are central to mobile robotics, aerial vehicles, and autonomous navigation in environments where robustness to sensor degeneracy, dynamic scenes, or sparse environments is critical. Recent research highlights the growing trend toward tightly-coupled, continuous-time formulations and advanced scene representations, such as 3D Gaussian Splatting, to deliver both geometric precision and photo-realistic visual quality (2507.04004).

1. System Architecture and Sensor Fusion

Modern LiDAR-Inertial-Camera SLAM systems, exemplified by Gaussian-LIC2, are organized into parallel modules with distinct responsibilities for real-time odometry and incremental high-fidelity mapping. The principal components are:

• Front-End Odometry: Utilizes a continuous-time factor graph, often parameterized by non-uniform B-splines, to tightly couple IMU, LiDAR, and camera observations. This enables asynchronous sensor fusion and trajectory querying at arbitrary timestamps. Key factors fused include:
  • IMU factors (high-rate inertial propagation and bias modeling).
  • LiDAR factors (point-to-plane residuals for scan registration).
  • Camera factors:
  • Reprojection constraints: Projecting feature points from the LiDAR-based map into camera images.
  • Photometric constraints: Comparing rendered images from a 3D Gaussian map against captured RGB frames, enabling direct use of appearance information from the evolving scene reconstruction.
• Back-End Mapping: Maintains an incremental and differentiably optimized 3D Gaussian map, supporting photo-realistic rendering and geometric regularization through both LiDAR and visual cues. This map is updated in real time and supplies additional constraints for further loop closure and trajectory refinement.

The synchronization and tight integration of these modalities enable robust performance even under LiDAR degradation or missing visual features, and support photo-realistic map creation for downstream applications (2507.04004).

2. 3D Gaussian Splatting and Scene Representation

The core geometric and radiance representation in advanced LiDAR-Inertial-Camera SLAM systems is 3D Gaussian Splatting. Each scene element is modeled as an anisotropic 3D Gaussian, parameterized by:

• A 3D center μ.
• A scale S and rotation R, forming the covariance as Σ = R S Sᵀ Rᵀ, encoding ellipsoidal extent and orientation.
• An opacity o.
• Spherical harmonics coefficients SH for view-dependent appearance.

For a given camera pose, each Gaussian is projected into the image plane as a 2D Gaussian, with its color and opacity “splatted” across pixels according to a spatially-varying weight:

$$\alpha = o \cdot \exp\left( -\frac{1}{2} (\mu' - \rho)^\top (\Sigma')^{-1} (\mu' - \rho) \right),$$

where μ′ and Σ′ are the projected mean and covariance, and ρ indexes image pixels. The final rendered image C(ρ) is obtained by compositing the weighted colors of all Gaussians in front-to-back order:

$$C(\rho) = \sum_{i=1}^n c_i \alpha_i \prod_{j=1}^{i-1} (1 - \alpha_j)$$

This explicit, differentiable representation enables rapid photorealistic rendering, real-time map expansion, and photometric supervision of pose estimation (2507.04004).

3. Depth Completion and Robust Gaussian Initialization

A key challenge in practical deployment, especially with sparse or solid-state LiDAR sensors, is under-reconstruction in LiDAR-blind regions. Gaussian-LIC2 overcomes this via a zero-shot depth completion strategy:

• Zero-Shot Depth Model: Fuses sparse LiDAR depth readings and RGB images using a lightweight neural network to generate dense depth maps for input frames.
• Patch-Guided Visual Point Extraction: The dense depth is filtered, and the image is divided into patches. In patches lacking direct LiDAR coverage but with reliable depth prediction, the minimum depth is back-projected to produce visual points.
• Joint Gaussian Initialization: Visual points supplement LiDAR points for initializing 3D Gaussians, ensuring coverage in both sensor-rich and -sparse regions.
• Hybrid Supervision: The subsequent optimization is guided by both geometric constraints (from accurate LiDAR depths) and photometric constraints (from projected RGB images).

This approach broadens system applicability and improves the quality of 3D reconstructions in diverse and challenging environments (2507.04004).

4. Continuous-Time Trajectory Estimation and Photometric SLAM

A distinguishing feature is the use of continuous-time trajectory representations, parameterized with non-uniform B-splines:

• The pose at time $t$ is

$$R(t) = F_1(R_{i-3}, ..., R_i, t_i, t_{i+1}, t),\quad p(t) = F_2(p_{i-3}, ..., p_i, t_i, t_{i+1}, t)$$

where $F_1$ and $F_2$ are spline functions for rotation and translation.

• The optimization problem fuses:
  • LiDAR point-to-plane constraints,
  • IMU integration and bias terms,
  • Camera constraints:
  • Option 1: Reprojection error,
  • Option 2: A photometric constraint, minimized as

  $$\mathcal{L} = \frac{1}{2} ||\widehat{C} - C||_1 + \frac{1}{2} \mathcal{L}_{D\text{-}SSIM}(\widehat{C}, C),$$

  where $\widehat{C}$ is the rendered RGB image from the Gaussian map and $C$ is the captured image.

• Photometric constraints provide dense, scene-wide supervision, making trajectory estimation robust to local LiDAR failure or visual feature sparsity. The refined pose from photometric optimization then closes the loop in the factor graph as an additional constraint.

This tightly-coupled, continuous-time, photometric SLAM formulation yields improved accuracy and resilience under diverse sensor conditions (2507.04004).

5. CUDA-Accelerated Optimization and Real-Time Operation

Real-time performance for both mapping and SLAM is achieved via dedicated CUDA acceleration techniques:

• Fast Tile-Based Culling: Only image tiles significantly affected by projected Gaussians are processed, reducing redundant computation especially for anisotropic (elliptical) Gaussians.
• Per-Gaussian Backpropagation: Optimizes computational load balancing during the backward pass by parallelizing over buckets of Gaussians and spatial tiles, mitigating atomic collisions in memory updates.
• Incremental Sparse Updates: Sparse Adam is used to update only the active Gaussians in each iteration; high- and low-order SH coefficients are processed separately, reducing overhead.
• Efficient Loss Computation: Differentiable SSIM loss is computed efficiently; memory transfers between host and device are handled with pinned and non-blocking techniques.

These strategies ensure the mapping backend can keep pace with sensor inputs and support online, incremental map growth and photometric supervision (2507.04004).

6. Downstream Applications

The integrated 3D Gaussian-based SLAM architecture enables advanced downstream applications:

• Video Frame Interpolation: By leveraging the continuous-time trajectory and scene model, the system can generate novel intermediate views between captured frames, effectively increasing temporal resolution.
• Rapid 3D Mesh Extraction: The optimized Gaussian map facilitates rendering of RGB-D images from arbitrary viewpoints. Fused depth maps, via truncated signed distance functions (TSDF) and Marching Cubes, yield high-fidelity 3D meshes colored by appearance or normals.
• Novel View Synthesis: The framework is evaluated for both in-sequence and out-of-sequence view synthesis, enabling evaluation for both standard localization and graphics-oriented tasks.

These applications are demonstrated using both public and a purpose-built self-collected dataset with ground-truth pose and depth for rigorous evaluation (2507.04004).

7. Experimental Results and System Evaluation

Quantitative evaluation on datasets such as R3LIVE, FAST-LIVO, MCD, and M2DGR—covering indoor, outdoor, and urban scenes—shows that Gaussian-LIC2 outperforms conventional LiDAR-Inertial-Camera SLAM methods (LVI-SAM, R3LIVE, FAST-LIVO2) as well as earlier radiance-field and 3DGS-based approaches (MonoGS, Co-SLAM, MM3DGS-SLAM, Gaussian-LIC).

Key performance indicators include:

• Localization accuracy: Lower start-to-end drift and reduced absolute pose error (APE), maintaining robustness even when only partial sensor coverage is present.
• Reconstruction fidelity: Sharper RGB and depth image renderings, improved mesh quality, and robust novel view synthesis, including for out-of-sequence targets.
• Computational efficiency: Real-time mapping is completed within the data capture interval, supporting live deployment.

The accompanying dataset includes ground-truth trajectories, depth maps, and challenging sequences to support further research (2507.04004).

---

In summary, Gaussian-LIC2 typifies the state of the art in LiDAR-Inertial-Camera SLAM, coupling accurate, continuous-time odometry with real-time, high-fidelity 3D Gaussian Splatting maps. Its hybrid initialization, deep zero-shot depth completion, and CUDA-optimized photometric mapping enable robust operation in sensor-degraded and large-scale settings, support downstream vision and graphics applications, and set benchmarks for future multi-modal SLAM system research (2507.04004).