Joint 3D Proposal Generation and Object Detection from View Aggregation (1712.02294v4)

Abstract: We present AVOD, an Aggregate View Object Detection network for autonomous driving scenarios. The proposed neural network architecture uses LIDAR point clouds and RGB images to generate features that are shared by two subnetworks: a region proposal network (RPN) and a second stage detector network. The proposed RPN uses a novel architecture capable of performing multimodal feature fusion on high resolution feature maps to generate reliable 3D object proposals for multiple object classes in road scenes. Using these proposals, the second stage detection network performs accurate oriented 3D bounding box regression and category classification to predict the extents, orientation, and classification of objects in 3D space. Our proposed architecture is shown to produce state of the art results on the KITTI 3D object detection benchmark while running in real time with a low memory footprint, making it a suitable candidate for deployment on autonomous vehicles. Code is at: https://github.com/kujason/avod

• The paper presents AVOD, a novel 3D object detection network that aggregates LIDAR and RGB views to generate reliable region proposals.
• It employs a Feature Pyramid-inspired extractor and a fusion-based RPN, achieving high recall (86% for car class) and precise 3D bounding box regression.
• The network demonstrates efficient computation, enhanced orientation estimation, and robust performance in real-world autonomous driving scenarios.

Joint 3D Proposal Generation and Object Detection from View Aggregation

The paper presents a novel 3D object detection network, AVOD (Aggregate View Object Detection), designed for autonomous driving scenarios. This network amalgamates features from LIDAR point clouds and RGB images to create shared features for two subnetworks: a Region Proposal Network (RPN) and a second-stage detection network. The RPN employs a multimodal feature fusion methodology on high-resolution feature maps to generate reliable 3D object proposals. These proposals are subsequently processed by the detection network to perform accurate 3D bounding box regression and object classification.

Key Contributions

1. Feature Extraction and Fusion:
   • The introduction of a feature extractor inspired by Feature Pyramid Networks (FPNs) that produces high-resolution feature maps from LIDAR and RGB images, facilitating the localization of small objects within the scene.
   • A fusion-based RPN that integrates features from multiple modalities to produce high-recall region proposals for various object classes, particularly small ones.
2. Novel 3D Bounding Box Encoding and Orientation Estimation:
   • The paper proposes a new 3D bounding box encoding that adheres to geometric constraints, resulting in better localization accuracy.
   • An explicit orientation vector regression method is introduced to resolve the ambiguity in orientation estimates inherent in bounding box representations.
3. Efficient Computation:
   • The network uses $1\times 1$ convolutions at the RPN stage and leverages a fixed look-up table of 3D anchor projections, allowing it to achieve high computational speed and a low memory footprint without sacrificing detection performance.

Experimental Results

• 3D Proposal Recall:
  • The proposed RPN variants show superior performance to traditional 3D proposal generation algorithms like 3DOP and Mono3D. For instance, the Feature Pyramid-based fusion RPN achieves 86% recall for the car class with just 10 proposals, significantly outperforming competing methods.
• 3D Object Detection:
  • On the KITTI dataset, AVOD's car detection performance is on par with MV3D, but AVOD exhibits markedly improved orientation estimation. The network achieves an average precision (AP) of 74.44% and an Average Heading Similarity (AHS) of 74.11% for the moderate difficulty car class, demonstrating a notable enhancement over MV3D.
  • For pedestrians and cyclists, the Feature Pyramid version of AVOD achieves 50.80% AP and 42.81% AP for the moderate difficulty class, respectively, indicating a substantial improvement over single-shot detectors like VeloFCN and 3D-FCN.

Practical Implications

The strong performance and efficient computational characteristics of AVOD suggest it is well-suited for real-world deployment in autonomous driving. The network's ability to generalize to new scenes and perform robustly under varying weather and lighting conditions further underlines its practical utility.

Theoretical Implications

The methodological innovations in AVOD, particularly the integration of multimodal fusion at the RPN stage, high-resolution feature extraction, and improved orientation regression, contribute significantly to the ongoing research in 3D object detection. These advancements illustrate the impact of leveraging rich, fused features from different sensor modalities in improving 3D detection tasks.

Future Directions

There are several potential avenues for future exploration based on AVOD:

1. Extension to Other Modalities:
   • Including additional sensor modalities such as radar or thermal imaging could further enhance detection robustness in adverse conditions.
2. Improvement in Smaller Object Classes:
   • Tailoring specific architectures or strategies for better handling of small and densely packed objects such as pedestrians and cyclists.
3. Optimization for Real-Time Deployment:
   • Though AVOD exhibits a competitively low runtime, ongoing research to further optimize its performance for low-power, real-time applications in autonomous vehicles would be beneficial.

In conclusion, the AVOD network demonstrates a comprehensive approach to 3D object detection through multimodal feature fusion, efficient bounding box encoding, and robust orientation estimation, establishing a strong foundation for both practical deployment and future research advancements in autonomous driving technologies.