Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Evaluating Object (mis)Detection from a Safety and Reliability Perspective: Discussion and Measures (2203.02205v3)

Published 4 Mar 2022 in cs.LG, cs.AI, and cs.CV

Abstract: We argue that object detectors in the safety critical domain should prioritize detection of objects that are most likely to interfere with the actions of the autonomous actor. Especially, this applies to objects that can impact the actor's safety and reliability. To quantify the impact of object (mis)detection on safety and reliability in the context of autonomous driving, we propose new object detection measures that reward the correct identification of objects that are most dangerous and most likely to affect driving decisions. To achieve this, we build an object criticality model to reward the detection of the objects based on proximity, orientation, and relative velocity with respect to the subject vehicle. Then, we apply our model on the recent autonomous driving dataset nuScenes, and we compare nine object detectors. Results show that, in several settings, object detectors that perform best according to the nuScenes ranking are not the preferable ones when the focus is shifted on safety and reliability.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016.
  2. A survey on 3d object detection methods for autonomous driving applications. IEEE Transactions on Intelligent Transportation Systems, 20(10):3782–3795, 2019.
  3. Basic concepts and taxonomy of dependable and secure computing. IEEE transactions on dependable and secure computing, 1(1):11–33, 2004.
  4. Risk ranked recall: Collision safety metric for object detection systems in autonomous vehicles. In 2021 10th Mediterranean Conference on Embedded Computing (MECO), pages 1–4. IEEE, 2021.
  5. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020.
  6. On the evaluation of outlier detection: Measures, datasets, and an empirical study continued. Lernen, Wissen, Daten, Analysen 2016, 2016.
  7. library and examples used in this paper, https://github.com/AndreaCeccarelli/metrics_model.
  8. Chih-Hong Cheng. Safety-aware hardening of 3d object detection neural network systems. In International Conference on Computer Safety, Reliability, and Security, pages 213–227. Springer, 2020.
  9. Contributors. MMDetection3D: OpenMMLab next-generation platform for general 3D object detection. https://github.com/open-mmlab/mmdetection3d, 2020.
  10. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  11. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 764–773, 2017.
  12. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1):98–136, 2015.
  13. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303–338, 2010.
  14. Towards safe autonomous driving: Capture uncertainty in the deep neural network for lidar 3d vehicle detection. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pages 3266–3273. IEEE, 2018.
  15. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1231–1237, 2013.
  16. Understanding the properness of incorporating machine learning algorithms in safety-critical systems. In Proceedings of the 36th Annual ACM Symposium on Applied Computing, pages 232–234, 2021.
  17. A survey of deep learning techniques for autonomous driving. Journal of Field Robotics, 37(3):362–386, 2020.
  18. Bounding box regression with uncertainty for accurate object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  19. One thousand and one hours: Self-driving motion prediction dataset. In Jens Kober, Fabio Ramos, and Claire Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 409–418. PMLR, 16–18 Nov 2021.
  20. A survey of deep learning-based object detection. IEEE Access, 7:128837–128868, 2019.
  21. What uncertainties do we need in bayesian deep learning for computer vision? arXiv preprint arXiv:1703.04977, 2017.
  22. Autonomous vehicle safety: An interdisciplinary challenge. IEEE Intelligent Transportation Systems Magazine, 9(1):90–96, 2017.
  23. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12697–12705, 2019.
  24. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  25. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  26. Deep learning for generic object detection: A survey. International journal of computer vision, 128(2):261–318, 2020.
  27. A general framework for uncertainty estimation in deep learning. IEEE Robotics and Automation Letters, 5(2):3153–3160, 2020.
  28. From evaluation to verification: Towards task-oriented relevance metrics for pedestrian detection in safety-critical domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 38–45, 2021.
  29. nuScenes. nuscenes web-site, https://www.nuscenes.org/, 2022.
  30. A survey on performance metrics for object-detection algorithms. In 2020 International Conference on Systems, Signals and Image Processing (IWSSIP), pages 237–242. IEEE, 2020.
  31. David MW Powers. Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation. arXiv preprint arXiv:2010.16061, 2020.
  32. Rgb-d object classification for autonomous driving perception. In RGB-D Image Analysis and Processing, pages 377–395. Springer, 2019.
  33. Designing network design spaces. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10425–10433, 2020.
  34. Introduction to modern information retrieval. mcgraw-hill, 1983.
  35. Scalability in perception for autonomous driving: Waymo open dataset. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2443–2451, 2020.
  36. Interaction-dynamics-aware perception zones for obstacle detection safety evaluation. arXiv preprint arXiv:2206.12471, 2022.
  37. Kishor S. Trivedi. Probability and Statistics with Reliability, Queuing, and Computer Science Applications. Prentice Hall, 1982.
  38. On the safety of machine learning: Cyber-physical systems, decision sciences, and data products. Big data, 5(3):246–255, 2017.
  39. A comprehensive safety metric to evaluate perception in autonomous systems. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pages 1–8, 2020.
  40. Fcos3d: Fully convolutional one-stage monocular 3d object detection. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 913–922, Los Alamitos, CA, USA, oct 2021. IEEE Computer Society.
  41. Probabilistic and geometric depth: Detecting objects in perspective. In Conference on Robot Learning, pages 1475–1485. PMLR, 2022.
  42. Deep neural network with transfer learning in remote object detection from drone. In Proceedings of the 5th International ACM Mobicom Workshop on Drone Assisted Wireless Communications for 5G and Beyond, pages 121–126, 2022.
  43. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018.
  44. Object detection with deep learning: A review. IEEE transactions on neural networks and learning systems, 30(11):3212–3232, 2019.
  45. Ssn: Shape signature networks for multi-class object detection from point clouds. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 581–597, Cham, 2020. Springer International Publishing.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now