Focal Sparse Convolutional Networks for 3D Object Detection (2204.12463v1)

Abstract: Non-uniformed 3D sparse data, e.g., point clouds or voxels in different spatial positions, make contribution to the task of 3D object detection in different ways. Existing basic components in sparse convolutional networks (Sparse CNNs) process all sparse data, regardless of regular or submanifold sparse convolution. In this paper, we introduce two new modules to enhance the capability of Sparse CNNs, both are based on making feature sparsity learnable with position-wise importance prediction. They are focal sparse convolution (Focals Conv) and its multi-modal variant of focal sparse convolution with fusion, or Focals Conv-F for short. The new modules can readily substitute their plain counterparts in existing Sparse CNNs and be jointly trained in an end-to-end fashion. For the first time, we show that spatially learnable sparsity in sparse convolution is essential for sophisticated 3D object detection. Extensive experiments on the KITTI, nuScenes and Waymo benchmarks validate the effectiveness of our approach. Without bells and whistles, our results outperform all existing single-model entries on the nuScenes test benchmark at the paper submission time. Code and models are at https://github.com/dvlab-research/FocalsConv.

• The paper introduces Focals Conv, a module that dynamically predicts spatial feature importance using focal loss to enhance 3D object detection.
• It extends the approach with Focals Conv-F, integrating RGB data to boost accuracy on benchmarks like KITTI and nuScenes.
• These modules efficiently focus computing on high-value regions, paving the way for real-time applications in autonomous driving.

Focal Sparse Convolutional Networks for 3D Object Detection

The paper introduces two novel modules, Focal Sparse Convolution (Focals Conv) and its multi-modal extension Focal Sparse Convolution with Fusion (Focals Conv-F), aimed at enhancing the ability of Sparse Convolutional Neural Networks (Sparse CNNs) to effectively process sparse 3D data, such as point clouds, for object detection tasks. Traditionally, Sparse CNNs treat all input features uniformly regardless of their spatial importance, potentially leading to suboptimal performance in tasks requiring discernment between foreground and background data.

The Focals Conv addresses this by incorporating a dynamic learning mechanism to predict the spatial importance of features, enabling adaptively dilated output shapes based on the predicted importance. This is contrasted with both regular sparse convolution, which may incur unnecessary computational cost, and submanifold sparse convolution, which may restrict necessary information flow between spatially disconnected features.

Core Contributions:

1. Focals Conv:
   • Incorporates predicted cubic importance maps to dynamically adjust the spatial sparsity of the features.
   • Learns spatial importance through an additional layer trained using focal loss, aiming to concentrate computational effort on more valuable foreground data.
2. Focals Conv-F:
   • Extends Focals Conv by fusing RGB features from images to enhance the importance prediction process. This module leverages the appearance information inherent in image data to better predict feature importance.

Both modules are designed to be lightweight and integrate seamlessly into existing Sparse CNN frameworks, demonstrating non-trivial improvements on 3D object detection benchmarks such as KITTI and nuScenes. Notably, the implementation of Focals Conv-F showed the most substantial gains, outperforming several state-of-the-art 3D detection methods. On the nuScenes benchmark, it achieved a mean Average Precision (mAP) of 68.9% with improvements registered across multiple object categories.

Implications and Future Directions:

1. Enhanced Feature Representation:
   • These modules showcase the importance of incorporating dynamic and learned sparsity in feature representation for 3D data, which could be extended to other tasks like 3D segmentation.
2. Efficient Computation:
   • The ability to focus computational efforts on regions of high importance could lead to more efficient use of resources, potentially facilitating real-time applications in autonomous driving scenarios.
3. Multi-Modal Integration:
   • The integration of multi-modal data (e.g., image and LIDAR) to inform feature importance opens avenues for exploring more complex fusion strategies that could improve model robustness and accuracy.
4. Theoretical Extension:
   • Future research could explore theoretical foundations and extensions of these adaptive methods, addressing data sparsity and irregularity more comprehensively in 3D and multi-modal contexts.

Overall, the contributions presented in this work underlie the potential of adaptive sparsity in enhancing the performance of 3D object detection networks, paving the way for advancements in both practical applications and theoretical understanding.