RTS3D: Real-time Stereo 3D Detection from 4D Feature-Consistency Embedding Space for Autonomous Driving (2012.15072v1)

Abstract: Although the recent image-based 3D object detection methods using Pseudo-LiDAR representation have shown great capabilities, a notable gap in efficiency and accuracy still exist compared with LiDAR-based methods. Besides, over-reliance on the stand-alone depth estimator, requiring a large number of pixel-wise annotations in the training stage and more computation in the inferencing stage, limits the scaling application in the real world. In this paper, we propose an efficient and accurate 3D object detection method from stereo images, named RTS3D. Different from the 3D occupancy space in the Pseudo-LiDAR similar methods, we design a novel 4D feature-consistent embedding (FCE) space as the intermediate representation of the 3D scene without depth supervision. The FCE space encodes the object's structural and semantic information by exploring the multi-scale feature consistency warped from stereo pair. Furthermore, a semantic-guided RBF (Radial Basis Function) and a structure-aware attention module are devised to reduce the influence of FCE space noise without instance mask supervision. Experiments on the KITTI benchmark show that RTS3D is the first true real-time system (FPS$>$24) for stereo image 3D detection meanwhile achieves $10\%$ improvement in average precision comparing with the previous state-of-the-art method. The code will be available at https://github.com/Banconxuan/RTS3D

• The paper introduces a novel 4D feature-consistent embedding space that reduces dependency on depth supervision and improves computational efficiency.
• The paper employs semantic-guided RBF and structure-aware attention modules to mitigate noise and enhance detection accuracy without instance mask supervision.
• The paper demonstrates real-time performance on KITTI with over 24 FPS and a 10% precision gain over existing stereo vision methods.

Overview of RTS3D: Real-time Stereo 3D Detection from 4D Feature-Consistency Embedding Space for Autonomous Driving

This paper introduces RTS3D, a novel approach for real-time 3D object detection from stereo images, emphasizing its application in autonomous driving. The research addresses the limitations faced by traditional image-based 3D detection methods, particularly those using Pseudo-LiDAR representations which have historically fallen short in terms of efficiency and accuracy compared to LiDAR-based methods. The RTS3D framework innovates with the introduction of a 4D feature-consistent embedding (FCE) space, divergent from the conventional 3D occupancy space employed by previous methods.

Core Contributions

1. 4D Feature-Consistent Embedding Space: The authors propose an intermediate representation of the 3D scene that does not depend on depth supervision. The FCE space is designed to encapsulate both structural and semantic information of objects in the environment by examining multi-scale feature consistency derived from stereo image pairs. This method inherently reduces dependency on pixel-wise annotation for depth estimation, thereby optimizing computational efficiency.
2. Semantic-guided Radial Basis Function (RBF) and Structure-aware Attention Module: These modules are designed to mitigate noise within the FCE space. By leveraging semantic cues and developing a targeted attention mechanism, these modules enhance the accuracy and reliability of object detection without requiring instance mask supervision.
3. Real-time Performance with Improved Precision: The evaluation on the KITTI benchmark demonstrates that RTS3D not only achieves true real-time processing capabilities (FPS > 24) but also surpasses previous state-of-the-art stereo vision methods by a significant margin (10% gain in average precision).

Experimental Results

The RTS3D system is quantitatively validated with experiments conducted on the KITTI dataset, showcasing substantial advancements in detection performance while maintaining real-time operational capability. The system excelled with an FPS of over 24 while improving the average precision by 10% compared to existing methods, marking a significant leap in practical applications for autonomous vehicular systems.

Implications and Future Directions

RTS3D represents a pivotal step forward in image-based 3D object detection, bridging a critical gap between the image-based and LiDAR-based detection methodologies. Practically, this facilitates more accessible and affordable implementations in autonomous driving systems, which typically avoid costly LiDAR units and extensive label supervision. Theoretically, the introduction of the 4D FCE space opens new avenues for exploring stereo vision’s potential, especially in terms of computational efficiency and semantic richness.

In terms of future developments, the authors suggest enhancing stereo vision methods further by refining the feature-consistency space, potentially leading to even more robust and adaptive detection frameworks. Moreover, exploring cross-domain applications for similar embedding spaces in robotics or augmented reality could yield insightful results in different contexts.

RTS3D provides a compelling precedent for real-time stereo vision systems, indicating promising trajectories for continued innovation in autonomous navigation technologies.