HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting (2403.12722v1)

Abstract: Holistic understanding of urban scenes based on RGB images is a challenging yet important problem. It encompasses understanding both the geometry and appearance to enable novel view synthesis, parsing semantic labels, and tracking moving objects. Despite considerable progress, existing approaches often focus on specific aspects of this task and require additional inputs such as LiDAR scans or manually annotated 3D bounding boxes. In this paper, we introduce a novel pipeline that utilizes 3D Gaussian Splatting for holistic urban scene understanding. Our main idea involves the joint optimization of geometry, appearance, semantics, and motion using a combination of static and dynamic 3D Gaussians, where moving object poses are regularized via physical constraints. Our approach offers the ability to render new viewpoints in real-time, yielding 2D and 3D semantic information with high accuracy, and reconstruct dynamic scenes, even in scenarios where 3D bounding box detection are highly noisy. Experimental results on KITTI, KITTI-360, and Virtual KITTI 2 demonstrate the effectiveness of our approach.

• The paper introduces a unified pipeline that uses 3D Gaussian Splatting to infer geometry, semantics, and motion from RGB images in urban environments.
• It decomposes scenes into static and dynamic components, applying the unicycle model to regularize motion for more accurate dynamic object tracking.
• Extensive experiments on benchmarks like KITTI and Virtual KITTI 2 demonstrate state-of-the-art novel view and 3D semantic synthesis performance.

Holistic Urban 3D Scene Understanding via Gaussian Splatting

Introduction to the Approach

Urban scene understanding plays a crucial role in numerous applications such as autonomous driving and city planning. Traditionally, achieving a comprehensive understanding of urban scenes using only RGB images has been challenging due to the complexity and dynamic nature of urban environments. This paper introduces a novel pipeline utilizing 3D Gaussian Splatting for holistic urban scene understanding. The approach is distinctive in leveraging 3D Gaussians to infer geometry, appearance, semantics, and motion in a unified framework.

Methodology Overview

Scene Representation and Decomposition

The core of our method lies in decomposing the urban scene into static regions and multiple dynamically moving objects. Each component of the scene is represented using 3D Gaussians, which encapsulate both appearance and semantics. Specifically, dynamic objects are modeled in their canonical space and transformed to the global coordinate system, constrained by physically plausible motion models.

Unicycle Model for Regularizing Movement

A pivotal innovation in our approach is the application of the unicycle model to regularize the motion of dynamic objects. This model considerably mitigates the impact of noisy tracking data, enhancing the reconstruction of dynamic scenes. By introducing regularization terms that ensure consistency with the unicycle model, our method achieves smoother and more plausible motion trajectories for moving objects.

Multi-Modal Scene Understanding

A significant strength of our approach is its capacity to render various aspects of the scene, including novel viewpoints, semantic maps, and optical flow. This is accomplished through volume rendering techniques applied to the 3D Gaussian representation. Furthermore, by integrating semantic information within the 3D Gaussians, our method enables the extraction of accurate 3D semantic point clouds, advancing beyond merely generating accurate 2D semantic labels.

Learning with Noisy Labels

Our pipeline adeptly handles noisy input data, such as imprecise semantic labels, optical flow, and 3D tracking results. Through joint optimization and the introduction of physical motion constraints, our method robustly improves upon noisy initial estimates, facilitating the reconstruction of dynamic scenes from mere RGB image inputs.

Experimental Validation

Our approach is rigorously validated on multiple benchmarks, including KITTI, KITTI-360, and Virtual KITTI 2. The experimental results underscore the effectiveness of our method in various aspects of scene understanding. Notably, our technique achieves state-of-the-art performance in tasks such as novel view synthesis, novel view semantic synthesis, and 3D semantic reconstruction. These accomplishments demonstrate our method's capability to advance the frontier of urban scene understanding using only RGB images.

Implications and Future Directions

The proposed method bears significant implications for the development of advanced algorithms in the field of autonomous driving, virtual city modeling, and beyond. The ability to accurately model and understand urban scenes from economical RGB imagery opens new avenues for research and application. In future work, exploring the extension of our approach to include more extensive and complex urban environments, as well as incorporating additional modalities such as stereo or infrared imagery, could further enhance urban scene understanding capabilities.

In conclusion, our work on holistic urban scene understanding via Gaussian Splatting marks a significant step forward in the field of computer vision, presenting a robust method for dynamic scene reconstruction and understanding from RGB images alone.