Inferring Spatial Uncertainty in Object Detection (2003.03644v2)

Abstract: The availability of real-world datasets is the prerequisite for developing object detection methods for autonomous driving. While ambiguity exists in object labels due to error-prone annotation process or sensor observation noises, current object detection datasets only provide deterministic annotations without considering their uncertainty. This precludes an in-depth evaluation among different object detection methods, especially for those that explicitly model predictive probability. In this work, we propose a generative model to estimate bounding box label uncertainties from LiDAR point clouds, and define a new representation of the probabilistic bounding box through spatial distribution. Comprehensive experiments show that the proposed model represents uncertainties commonly seen in driving scenarios. Based on the spatial distribution, we further propose an extension of IoU, called the Jaccard IoU (JIoU), as a new evaluation metric that incorporates label uncertainty. Experiments on the KITTI and the Waymo Open Datasets show that JIoU is superior to IoU when evaluating probabilistic object detectors.

• The paper introduces a generative model that infers spatial uncertainty from LiDAR data by modeling bounding box parameters as probability distributions.
• It visualizes label uncertainty through spatial distributions in 3D and BEV, offering a clearer depiction of sensor noise and annotation ambiguities.
• The novel JIoU metric refines detection evaluation by accounting for uncertainty in intersection measurements, enabling a more nuanced assessment of object localization.

Inferring Spatial Uncertainty in Object Detection

The paper "Inferring Spatial Uncertainty in Object Detection" addresses a critical aspect of object detection for autonomous driving: the inherent uncertainty in spatial labels derived from LiDAR data. The authors propose a generative model for estimating the uncertainty of bounding box labels and introduce a novel evaluation metric, the Jaccard IoU (JIoU), that incorporates this uncertainty into measuring object localization accuracy.

Key Contributions

The primary contributions of this research are threefold:

1. Generative Model for Label Uncertainty: The paper presents a method to infer the spatial uncertainty in bounding box labels using a generative model of LiDAR point clouds. This approach allows the incorporation of prior knowledge about sensor noises and annotation ambiguities directly into the uncertainty model. By treating bounding box parameters as a distribution, this model captures the inherent ambiguity that arises from partial observations typical in driving scenarios.
2. Representation and Visualization of Uncertainty: The authors propose a spatial distribution representation, transforming the parameters’ uncertainty into a probabilistic depiction in 3D or Bird's Eye View (BEV) spaces. This spatial distribution is crucial for visualization and offers insights into the quality of bounding box labels, better reflecting real-world sensor limitations and ambiguities.
3. JIoU Metric: By redefining the standard Intersection over Union (IoU) to account for probabilistic representations, the paper introduces JIoU. This metric adapts to the inherent uncertainties in both predictions and ground truths. The JIoU enables a more nuanced evaluation of probabilistic object detectors by considering variations in label quality and observation noises.

Methodology

The core methodology involves modelling the uncertainty of bounding box labels through a variational Bayes approach, leveraging LiDAR point data. This method provides a posterior distribution for bounding box parameters, accounting for correlations among different parameters due to shared observational data. The probabilistic model therefore translates annotation uncertainty into a comprehensive spatial distribution.

Furthermore, the representation of spatial distribution considers the likelihood of an object intersecting a given space, which guides the development of the JIoU metric. Compared to existing approaches, JIoU aligns better with the probabilistic nature of object detection outcomes, offering a more realistic and interpretable evaluation criterion.

Experimental Findings

In experiments conducted on datasets like KITTI and Waymo, the methodology exhibits improved performance in characterizing object detection uncertainty. Notably, the JIoU as a metric highlights both label imperfections and model prediction uncertainties more effectively than traditional IoU metrics. For instance, JIoU is able to delineate between detections that are uncertain due to intrinsic label inadequacies as opposed to model errors, providing a more informative feedback loop for improving model robustness.

Practical and Theoretical Implications

This work underscores the importance of considering label uncertainty in the training and evaluation of detection models in autonomous vehicles. By explicitly modelling and evaluating label uncertainty, the approach paves the way for developing more reliable and robust object detectors that better account for the inherent uncertainties present in LiDAR-based perception datasets.

Theoretically, the research bridges a gap in detection evaluation by proposing a metric rooted in probability theory, moving beyond deterministic approximations and fostering a deeper understanding of detection confidence.

Future Directions

The approach opens several avenues for future research in AI, particularly in improving detection model training using uncertain labels or extending the generative model to multi-object tracking scenarios. Moreover, refining JIoU to better accommodate different probabilistic output forms from emerging detection architectures might further enhance its applicability as a standardized evaluation tool.

In conclusion, this work provides a significant methodological advancement in the landscape of autonomous driving perception systems, offering tools to better understand, represent, and leverage the uncertainties inherently linked to real-world data.