3D StreetUnveiler with Semantic-Aware 2DGS (2405.18416v2)

Abstract: Unveiling an empty street from crowded observations captured by in-car cameras is crucial for autonomous driving. However, removing all temporarily static objects, such as stopped vehicles and standing pedestrians, presents a significant challenge. Unlike object-centric 3D inpainting, which relies on thorough observation in a small scene, street scene cases involve long trajectories that differ from previous 3D inpainting tasks. The camera-centric moving environment of captured videos further complicates the task due to the limited degree and time duration of object observation. To address these obstacles, we introduce StreetUnveiler to reconstruct an empty street. StreetUnveiler learns a 3D representation of the empty street from crowded observations. Our representation is based on the hard-label semantic 2D Gaussian Splatting (2DGS) for its scalability and ability to identify Gaussians to be removed. We inpaint rendered image after removing unwanted Gaussians to provide pseudo-labels and subsequently re-optimize the 2DGS. Given its temporal continuous movement, we divide the empty street scene into observed, partial-observed, and unobserved regions, which we propose to locate through a rendered alpha map. This decomposition helps us to minimize the regions that need to be inpainted. To enhance the temporal consistency of the inpainting, we introduce a novel time-reversal framework to inpaint frames in reverse order and use later frames as references for earlier frames to fully utilize the long-trajectory observations. Our experiments conducted on the street scene dataset successfully reconstructed a 3D representation of the empty street. The mesh representation of the empty street can be extracted for further applications. The project page and more visualizations can be found at: https://streetunveiler.github.io

• The paper introduces a novel framework that uses semantic-aware 2D Gaussian Splatting and time-reversal inpainting to reconstruct empty street scenes for autonomous driving.
• The methodology achieves superior performance by lowering LPIPS and delivering competitive FID scores on the Waymo Open dataset.
• The approach ensures temporal consistency across long video trajectories, mitigating occlusions from transient objects and enhancing 3D scene clarity.

Reconstructing Empty Street Scenes for Autonomous Driving: Insights from "StreetUnveiler"

Introduction

The paper "StreetUnveiler" presents a methodological framework to reconstruct empty street scenes from in-car camera videos, addressing the need for autonomous vehicle systems to operate in a clear and unobstructed digital environment. Autonomous driving relies heavily on accurate 3D reconstructions of street scenes, but the presence of temporary static objects, such as parked cars and pedestrians, complicates this task. The proposed method involves novel approaches in 3D representation and inpainting to create a clean street scene, free from transient occlusions.

Methodology

2D Gaussian Splatting (2DGS)

The 2D Gaussian Splatting (2DGS) technique forms the cornerstone of the presented framework. Unlike conventional object-centric 3D inpainting methods, which work well within small and thoroughly observed environments, street scenes encompass long trajectories and limited object observation periods. The paper leverages 2DGS because of its scalability and editability, which are crucial for managing the extensive and dynamic nature of street data.

2D Gaussian splatting represents geometry using points with Gaussian distributions projected onto 2D planes, overlaying these projections to create coherent 3D representations. This approach allows for precise and efficient rendering, as well as region-specific modification, by manipulating parameters like point positions, tangential vectors, and scaling factors.

Semantic Decomposition and Inpainting Mask Generation

Critical to removing occlusions is accurately distinguishing between observable, partially observable, and completely unobservable regions. This is achieved through semantic guidance and rendered alpha maps. The process begins by associating each 2D Gaussian point with a non-trainable, hard-label semantic category, which aids in gathering points with the same semantic label and simplifies object removal.

The rendered alpha map identifies completely unobservable regions as those with low opacity values after object removal. This enables the generation of an inpainting mask focused on only these unobservable regions, subsequently reducing the inpainting complexity and enhancing the quality of the filled regions.

Time-Reversal Inpainting Framework

Maintaining temporal consistency across frames is particularly challenging in long trajectory videos. The paper introduces a time-reversal inpainting framework wherein video frames are inpainted in reverse order. This strategy leverages the more detailed and comprehensive views of objects obtained in earlier frames to guide the inpainting of later frames, ensuring conformity and minimizing discrepancies across the video sequence.

The selection of a reference-based inpainting model, specifically the diffusion-based LeftRefill method, further stabilizes the process by employing a high-resolution to low-resolution guidance approach. This method capitalizes on the extensive pixel-matching capabilities inherent to diffusion models, which ensures that the inpainted regions remain consistent with the surrounding scene when viewed from different angles.

Experimental Results

The efficacy of StreetUnveiler was validated using the Waymo Open Perception Dataset, focusing on real-world street scenes. Several performance metrics, including LPIPS and FID scores, were calculated to assess the quality of object removal and reconstruction. StreetUnveiler demonstrated superior performance compared to state-of-the-art 2D and 3D inpainting methods, achieving lower LPIPS and competitive FID values.

Qualitative analysis highlighted that the proposed method resulted in clearer and more consistent inpainting across frames, as opposed to significant blurring and inconsistency when using alternative methods. The introduction of time-reversal inpainting and 2DGS representation were pivotal elements contributing to this improved performance.

Implications and Future Work

The successful implementation of StreetUnveiler holds substantial practical and theoretical implications. From a practical standpoint, the ability to reconstruct empty street scenes can streamline the development and deployment of autonomous driving systems, enhancing their reliability by removing transient occlusions that could interfere with navigation and sensor systems.

Theoretically, this work broadens the scope of 3D scene representation and inpainting frameworks, particularly in how they handle large-scale, dynamic environments with limited observation data. Future avenues may explore the integration of this methodology with real-time processing capabilities, further optimizing the 3D modeling pipeline for use in fast-paced and variable settings such as urban traffic.

Additionally, extending the framework to incorporate more sophisticated learning mechanisms for semantic labeling, perhaps through unsupervised or semi-supervised learning paradigms, could enhance its adaptability and accuracy. Subsequent research could also investigate more robust handling of dynamic, moving objects in addition to static occlusions for a more holistic enhancement of autonomous driving environments.

Conclusion

StreetUnveiler presents a significant advancement in the reconstruction of empty street scenes by introducing innovative uses of 2D Gaussian Splatting and a time-reversal inpainting framework. The method surmounts the challenges posed by long trajectories and limited observation periods inherent to in-car camera videos, providing a robust solution for environments crucial to autonomous driving. Future research and development in this domain can build upon these findings to further refine and expand the capabilities of autonomous driving systems.