On Calibration of Object Detectors: Pitfalls, Evaluation and Baselines (2405.20459v1)

Abstract: Reliable usage of object detectors require them to be calibrated -- a crucial problem that requires careful attention. Recent approaches towards this involve (1) designing new loss functions to obtain calibrated detectors by training them from scratch, and (2) post-hoc Temperature Scaling (TS) that learns to scale the likelihood of a trained detector to output calibrated predictions. These approaches are then evaluated based on a combination of Detection Expected Calibration Error (D-ECE) and Average Precision. In this work, via extensive analysis and insights, we highlight that these recent evaluation frameworks, evaluation metrics, and the use of TS have notable drawbacks leading to incorrect conclusions. As a step towards fixing these issues, we propose a principled evaluation framework to jointly measure calibration and accuracy of object detectors. We also tailor efficient and easy-to-use post-hoc calibration approaches such as Platt Scaling and Isotonic Regression specifically for object detection task. Contrary to the common notion, our experiments show that once designed and evaluated properly, post-hoc calibrators, which are extremely cheap to build and use, are much more powerful and effective than the recent train-time calibration methods. To illustrate, D-DETR with our post-hoc Isotonic Regression calibrator outperforms the recent train-time state-of-the-art calibration method Cal-DETR by more than 7 D-ECE on the COCO dataset. Additionally, we propose improved versions of the recently proposed Localization-aware ECE and show the efficacy of our method on these metrics as well. Code is available at: https://github.com/fiveai/detection_calibration.

• The paper exposes pitfalls in using fixed operating thresholds that bias the evaluation of object detector calibration.
• The paper introduces novel metrics, LaECE₀ and LaACE₀, to jointly assess classification and localization quality.
• The paper shows tailored post-hoc calibrators like Platt Scaling and Isotonic Regression can enhance reliability over train-time methods.

An Analysis of "On Calibration of Object Detectors: Pitfalls, Evaluation and Baselines"

The calibration of object detectors is critical for applications in fields such as autonomous vehicles and medical imaging, where reliability is paramount. The paper "On Calibration of Object Detectors: Pitfalls, Evaluation and Baselines" by Kuzucu et al. explores the challenges and solutions associated with calibrating object detectors. The authors present a thorough critique of existing calibration methods and propose their own approaches for improving both the calibration and evaluation of object detectors.

Critique and Analysis of Current Approaches

The paper highlights several limitations in currently popular evaluation frameworks for object detector calibration, particularly those that rely on measures such as Detection Expected Calibration Error (D-ECE) and Average Precision (AP). It notes that these measures often use fixed and inconsistent thresholds for accuracy and calibration, which do not reflect real-world deployments where models must perform consistent under various conditions. Furthermore, the paper argues that existing evaluations often neglect fine-grained confidence information and are restricted by unsuitable dataset configurations.

The primary critique is directed towards the evaluation framework, which tends to use fixed operating thresholds. This approach is claimed to introduce bias and misrepresent how object detectors are used in practice. The authors illustrate this with detailed examples showing that such a framework can unfairly rank detectors and lead to incorrect conclusions regarding their joint performance of accuracy and calibration.

Proposed Framework and Methods

To resolve these issues, the authors propose a comprehensive framework that considers both calibration and accuracy through the introduction of novel measures like $\mathrm{LaECE}_0$ (Localisation-aware Expected Calibration Error with zero threshold) and $\mathrm{LaACE}_0$ (Localisation-aware Adaptive Calibration Error), which account for both classification and localization qualities in calibration. These measures aim to provide more informative confidence scores, aligning them with localization performance on a finer scale.

The paper also champions the use of post-hoc calibrators tailored specifically for object detection, such as Platt Scaling (PS) and Isotonic Regression (IR). The authors present strong experimental evidence showing that post-hoc calibration techniques, when configured correctly, outperform the existing state-of-the-art train-time calibration methods. They argue that these post-hoc methods are easy to implement and can be applied to any pre-trained object detector, suggesting their use as strong baselines for the field moving forward.

Implications and Future Directions

The implications of this research are particularly salient for safety-critical systems employing object detectors. The proposed evaluation framework and calibration methods could enhance the reliability of detectors, ensuring that models not only predict accurate bounding boxes and classes but also provide confidence scores that genuinely reflect prediction uncertainties. This is vital for applications like autonomous vehicles where erroneous predictions can have severe consequences.

Furthermore, the authors' method of incorporating the calibration of long-tailed datasets like LVIS extends the utility of their approach to diverse real-world scenarios where object class distributions are severely uneven, providing a foundation for future work in improving generalization under such conditions.

Conclusion

By addressing fundamental flaws in current evaluation measures and proposing a robust alternative framework, this paper significantly contributes to the field of object detection calibration. It challenges practitioners to rethink model evaluation protocols and use more representative measures that align with practical applications. Future research will likely build on this framework, extending its application to real-time systems and exploring further improvements in post-hoc calibration techniques to ensure reliable performance across varied operational environments.