Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud (1903.09847v4)

Published 23 Mar 2019 in cs.CV

Abstract: Monocular 3D scene understanding tasks, such as object size estimation, heading angle estimation and 3D localization, is challenging. Successful modern day methods for 3D scene understanding require the use of a 3D sensor. On the other hand, single image based methods have significantly worse performance. In this work, we aim at bridging the performance gap between 3D sensing and 2D sensing for 3D object detection by enhancing LiDAR-based algorithms to work with single image input. Specifically, we perform monocular depth estimation and lift the input image to a point cloud representation, which we call pseudo-LiDAR point cloud. Then we can train a LiDAR-based 3D detection network with our pseudo-LiDAR end-to-end. Following the pipeline of two-stage 3D detection algorithms, we detect 2D object proposals in the input image and extract a point cloud frustum from the pseudo-LiDAR for each proposal. Then an oriented 3D bounding box is detected for each frustum. To handle the large amount of noise in the pseudo-LiDAR, we propose two innovations: (1) use a 2D-3D bounding box consistency constraint, adjusting the predicted 3D bounding box to have a high overlap with its corresponding 2D proposal after projecting onto the image; (2) use the instance mask instead of the bounding box as the representation of 2D proposals, in order to reduce the number of points not belonging to the object in the point cloud frustum. Through our evaluation on the KITTI benchmark, we achieve the top-ranked performance on both bird's eye view and 3D object detection among all monocular methods, effectively quadrupling the performance over previous state-of-the-art. Our code is available at https://github.com/xinshuoweng/Mono3D_PLiDAR.

Authors (2)

Xinshuo Weng (42 papers)
Kris Kitani (96 papers)

Citations (249)

View on Semantic Scholar

Summary

Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud: A Summary

This paper addresses a crucial challenge in 3D scene understanding from a single image, particularly focusing on monocular 3D object detection. The paper proposes a novel pipeline that bridges the performance gap between traditional 3D sensing methods, which typically rely on expensive sensors like LiDAR, and 2D imaging that captures scenes through a single camera. The method introduces the pseudo-LiDAR point cloud concept, aiming to adapt LiDAR-based 3D detection networks to operate effectively with just monocular input.

The authors tackle the inherent challenges of monocular 3D scene understanding by circumventing the need for depth sensors. The strategy involves estimating depth from monocular images to generate a 3D point cloud representation of the scene, termed pseudo-LiDAR. This approach allows the application of existing LiDAR-based two-stage 3D detection networks, specifically Frustum PointNets, without requiring modifications for sensor-dependent inputs.

A significant contribution of this work is addressing the noise prevalent in pseudo-LiDAR compared to true LiDAR point clouds. This noise manifests as local misalignment and the presence of long tails in point cloud representation, challenges the authors manage through two primary innovations. Firstly, they propose a 2D-3D bounding box consistency constraint, ensuring that the 3D bounding box predictions have a high overlap with the 2D proposals when projected onto the image. This is operationalized through a bounding box consistency loss during training and an optimization process during testing. Secondly, the paper advocates for the use of instance masks instead of bounding boxes for better filtering of irrelevant points in the point cloud frustum, thus reducing noise and improving 3D box predictions.

The robustness of this approach is evidenced by its performance on the KITTI dataset, where it achieves top-ranked results for both bird’s eye view and 3D object detection among monocular methods. The significant improvement, effectively quadrupling the previous state-of-the-art performance, underscores the effectiveness of the proposed innovations in addressing the typical limitations of monocular 3D object detection.

The implications of this research are substantial both in practical terms and for future AI developments. Practically, the ability to harness existing LiDAR-based detection architectures using solely monocular input supports the deployment of low-cost, efficient vision systems in domains like autonomous driving and robotics. Theoretically, the paper opens avenues for further research into improving depth estimation techniques from images, refining pseudo-LiDAR representations, and exploring more advanced bounding box consistency methods.

In summary, this paper successfully extends the capabilities of well-established LiDAR-based methods to function with monocular data, an advancement with strong potential to influence future research and application directions in 3D computer vision. Future work could explore how to enhance the quality of pseudo-LiDAR and further minimize the gap with genuine LiDAR data, as well as investigate the scalability of the proposed improvements across various object categories and environmental conditions.

Related Papers

Find Related Papers

GitHub

GitHub - xinshuoweng/Mono3DPLiDAR: Official PyTorch Implementation for "Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud", ICCVW 2019 (132 stars)