LiHi-GS: LiDAR-Supervised Gaussian Splatting for Highway Driving Scene Reconstruction (2412.15447v2)

Abstract: Photorealistic 3D scene reconstruction plays an important role in autonomous driving, enabling the generation of novel data from existing datasets to simulate safety-critical scenarios and expand training data without additional acquisition costs. Gaussian Splatting (GS) facilitates real-time, photorealistic rendering with an explicit 3D Gaussian representation of the scene, providing faster processing and more intuitive scene editing than the implicit Neural Radiance Fields (NeRFs). While extensive GS research has yielded promising advancements in autonomous driving applications, they overlook two critical aspects: First, existing methods mainly focus on low-speed and feature-rich urban scenes and ignore the fact that highway scenarios play a significant role in autonomous driving. Second, while LiDARs are commonplace in autonomous driving platforms, existing methods learn primarily from images and use LiDAR only for initial estimates or without precise sensor modeling, thus missing out on leveraging the rich depth information LiDAR offers and limiting the ability to synthesize LiDAR data. In this paper, we propose a novel GS method for dynamic scene synthesis and editing with improved scene reconstruction through LiDAR supervision and support for LiDAR rendering. Unlike prior works that are tested mostly on urban datasets, to the best of our knowledge, we are the first to focus on the more challenging and highly relevant highway scenes for autonomous driving, with sparse sensor views and monotone backgrounds. Visit our project page at: https://umautobots.github.io/lihi_gs

• The paper introduces a novel LiDAR-supervised Gaussian Splatting model that integrates LiDAR measurements to enhance 3D highway scene reconstruction.
• It employs differentiable LiDAR rendering, range image projection, and decoupled pose optimization to outperform SOTA methods in challenging highway scenarios.
• This advancement lays the groundwork for improved autonomous driving systems by providing robust, photorealistic reconstructions in sparse and dynamic highway environments.

Summary of "LiHi-GS: LiDAR-Supervised Gaussian Splatting for Highway Driving Scene Reconstruction"

The authors introduce LiHi-GS, a novel Gaussian Splatting (GS) method, to improve 3D photorealistic scene reconstruction for highway driving scenarios. GS methods offer advantages in real-time rendering and scene editing over their implicit counterparts like Neural Radiance Fields (NeRFs). Prior work in autonomous driving has predominantly focused on feature-rich urban environments, leaving a gap for methods optimized for highway scenes, which are characterized by sparse sensor views and minimalistic backgrounds.

Key Contributions

1. LiDAR Integration: The introduction of a differentiable LiDAR rendering model allows GS to utilize LiDAR measurements fully, enhancing the scene geometry through LiDAR supervision during training. This approach resolves the limitations of previous GS methods that inadequately leverage LiDAR data.
2. Extended Benchmarking: Unlike previous methods mainly focused on urban driving scenes, LiHi-GS targets highway scenarios. Evaluations extend to objects at distances exceeding 200 meters, showcasing the potency of LiDAR in open and sparse environments.
3. Performance: LiHi-GS reportedly surpasses state-of-the-art (SOTA) methods in both image and LiDAR synthesis quality. This result holds for tasks such as view interpolation, ego-view changes, and scene editing.

The paper argues that the integration of dense and precise LiDAR measurements, as opposed to a primary reliance on image-based methods, facilitates a more accurate and complete understanding of 3D highway scenes. The authors propose LiHi-GS as the first GS method incorporating explicit LiDAR sensor modeling, thereby enabling realistic LiDAR data synthesis, which is critical for autonomous vehicle systems that rely on perception-rich environments.

Numerical Results

The numerical evaluations demonstrate LiHi-GS's superiority over existing methods. On metrics such as PSNR, LPIPS, and SSIM, which quantify image quality, alongside LiDAR mean and median errors, LiHi-GS shows improved performance across various test cases. The results are consistent across different highway locations, reflecting robustness in handling diverse and challenging environments.

Methodology

The proposed method involves several innovations:

• LiDAR Visibility Rate: To reconcile the differences between LiDAR and camera perceptions of the environment, a mechanism is designed to model LiDAR visibility separately.
• Range Image Rendering: A comprehensive analysis of projecting 3D Gaussians into LiDAR range image frames is presented, addressing limitations in earlier methods using pseudo-depth images from LiDAR point clouds.
• Pose Optimization: A decoupled camera-LiDAR pose optimization addresses temporal offsets, improving the accuracy of actor geometry even in fast-moving scenarios.

Implications for Future Work

The authors highlight the importance of accurately reconstructing highway scenes for the future development of autonomous driving technology. By demonstrating LiHi-GS's ability to render high-quality images and LiDAR point clouds in highway settings, the method sets a foundation for further research in optimizing large model-based scene synthesis tasks. The findings suggest potential enhancements to existing autonomous driving systems by integrating more effective LiDAR data and leveraging GS-based methodologies.

In conclusion, LiHi-GS presents a substantial advancement for scene reconstruction in autonomous highway driving. By bridging gaps between traditional image-focused techniques and dynamics learned from LiDAR data, the research contributes a significant step forward in understanding and synthesizing realistic driving environments critical for advanced autonomous systems.