GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection (2103.17202v1)

Abstract: Modern 3D object detectors have immensely benefited from the end-to-end learning idea. However, most of them use a post-processing algorithm called Non-Maximal Suppression (NMS) only during inference. While there were attempts to include NMS in the training pipeline for tasks such as 2D object detection, they have been less widely adopted due to a non-mathematical expression of the NMS. In this paper, we present and integrate GrooMeD-NMS -- a novel Grouped Mathematically Differentiable NMS for monocular 3D object detection, such that the network is trained end-to-end with a loss on the boxes after NMS. We first formulate NMS as a matrix operation and then group and mask the boxes in an unsupervised manner to obtain a simple closed-form expression of the NMS. GrooMeD-NMS addresses the mismatch between training and inference pipelines and, therefore, forces the network to select the best 3D box in a differentiable manner. As a result, GrooMeD-NMS achieves state-of-the-art monocular 3D object detection results on the KITTI benchmark dataset performing comparably to monocular video-based methods. Code and models at https://github.com/abhi1kumar/groomed_nms

• The paper introduces GrooMeD-NMS, a differentiable NMS integrated into training to improve monocular 3D object localization.
• It reformulates classical NMS using unsupervised grouping and matrix operations to bridge the gap between training and inference.
• Evaluation on the KITTI dataset demonstrates enhanced precision and computational efficiency under challenging detection scenarios.

Analysis of GrooMeD-NMS for Monocular 3D Object Detection

The paper introduces GrooMeD-NMS, an innovative approach to Non-Maximal Suppression (NMS) specifically designed for monocular 3D object detection. The research addresses a critical limitation in conventional 3D object detection pipelines: NMS is traditionally employed only during inference, creating a disparity between training and inference phases that potentially limits the overall performance of the detection system.

Core Contributions and Methodology

The primary contribution of the paper is the formulation of a Grouped Mathematically Differentiable NMS (GrooMeD-NMS) that is integrated into the training pipeline of monocular 3D object detectors, enhancing the correlation between classification and 3D localization. The GrooMeD-NMS is characterized by several key features:

1. Mathematically Differentiable Matrix Formulation: The authors reformulate classical NMS as a series of matrix operations, including sorting, grouping, and masking, to obtain a mathematically differentiable closed-form approximation. This approach uses elementary matrix operations, allowing for the inclusion of NMS directly in the training loop and providing gradients to guide learning.
2. Unsupervised Grouping and Masking: By grouping boxes based on unsupervised clustering of Intersection over Union (IoU) overlaps, the NMS computation is optimized, avoiding the necessity for complex matrix inversion. Masking further simplifies computations by considering only relevant box interactions within each group.
3. Imagewise Average Precision Loss: To tackle class imbalance post NMS, the authors propose an Imagewise Average Precision (AP) loss that ranks boxes on an image-specific basis, enhancing the training signal and improving the selection of well-localized 3D boxes.

Experimental Results and Findings

The empirical evaluation is conducted on the KITTI dataset, a standard benchmark for 3D object detection. Key findings from the experiments include:

• Performance Improvement: The GrooMeD-NMS significantly outperforms traditional NMS approaches, achieving state-of-the-art results on the KITTI benchmark for monocular 3D object detection. The method demonstrates superior localization capabilities, particularly under challenging conditions (Moderate and Hard settings).
• Computational Efficiency: Despite the added complexity of integrating NMS during training, GrooMeD-NMS efficiently processes the training data, maintaining a competitive inference time comparable to traditional methods.
• Analytical Insights: The paper provides a detailed sensitivity analysis, demonstrating that GrooMeD-NMS is robust across various parameter settings, particularly the NMS threshold and grouping size. This robustness is crucial for operational deployment in varied environmental conditions.

Implications and Future Directions

The introduction of GrooMeD-NMS marks a considerable advancement in bridging the gap between training and inference for 3D object detection. The theoretical novelty lies in its ability to directly influence network training through differentiated NMS operations. Practically, this allows for more precise localization, crucial for applications in autonomous driving and augmented reality.

Looking forward, GrooMeD-NMS could serve as a foundation for further exploration into adaptable NMS techniques across different modalities beyond monocular vision, such as LiDAR-based detection and multi-sensor fusion strategies. Additionally, the approach sets the stage for further enhancements in network architectures and loss functions, explicitly accounting for 3D spatial reasoning during training.

This research offers a methodological shift that may inspire the integration of differentiable operations more broadly across machine learning tasks, encouraging seamless transitions between training and inference and leveraging end-to-end differentiable architectures for improved task-specific performance.