Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Decictor: Towards Evaluating the Robustness of Decision-Making in Autonomous Driving Systems (2402.18393v2)

Published 28 Feb 2024 in cs.AI, cs.NE, cs.RO, and cs.SE

Abstract: Autonomous Driving System (ADS) testing is crucial in ADS development, with the current primary focus being on safety. However, the evaluation of non-safety-critical performance, particularly the ADS's ability to make optimal decisions and produce optimal paths for autonomous vehicles (AVs), is also vital to ensure the intelligence and reduce risks of AVs. Currently, there is little work dedicated to assessing the robustness of ADSs' path-planning decisions (PPDs), i.e., whether an ADS can maintain the optimal PPD after an insignificant change in the environment. The key challenges include the lack of clear oracles for assessing PPD optimality and the difficulty in searching for scenarios that lead to non-optimal PPDs. To fill this gap, in this paper, we focus on evaluating the robustness of ADSs' PPDs and propose the first method, Decictor, for generating non-optimal decision scenarios (NoDSs), where the ADS does not plan optimal paths for AVs. Decictor comprises three main components: Non-invasive Mutation, Consistency Check, and Feedback. To overcome the oracle challenge, Non-invasive Mutation is devised to implement conservative modifications, ensuring the preservation of the original optimal path in the mutated scenarios. Subsequently, the Consistency Check is applied to determine the presence of non-optimal PPDs by comparing the driving paths in the original and mutated scenarios. To deal with the challenge of large environment space, we design Feedback metrics that integrate spatial and temporal dimensions of the AV's movement. These metrics are crucial for effectively steering the generation of NoDSs. We evaluate Decictor on Baidu Apollo, an open-source and production-grade ADS. The experimental results validate the effectiveness of Decictor in detecting non-optimal PPDs of ADSs.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Aldeida Aleti et al. 2022. Identifying Safety-critical Scenarios for Autonomous Vehicles via Key Features. arXiv preprint arXiv:2212.07566 (2022).
  2. Matthias Althoff and Sebastian Lutz. 2018. Automatic generation of safety-critical test scenarios for collision avoidance of road vehicles. In 2018 IEEE Intelligent Vehicles Symposium (IV). IEEE, Changshu, Suzhou, China, 1326–1333.
  3. Baidu. 2019. Apollo: Open Source Autonomous Driving. https://github.com/ApolloAuto/apollo
  4. DeepCrashTest: Turning Dashcam Videos into Virtual Crash Tests for Automated Driving Systems. In 2020 IEEE International Conference on Robotics and Automation ICRA. IEEE, Paris, France, 11353–11360.
  5. Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys (CSUR) 51, 1 (2018), 1–27.
  6. BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systems. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 488–500.
  7. comma.ai. 2022. OpenPilot: An open source driver assistance system. Retrieved Nov 7, 2022 from https://github.com/commaai/openpilot
  8. Scenario-based test reduction and prioritization for multi-module autonomous driving systems. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 82–93.
  9. A declarative metamorphic testing framework for autonomous driving. IEEE Transactions on Software Engineering (2022).
  10. Dense reinforcement learning for safety validation of autonomous vehicles. Nature 615, 7953 (2023), 620–627.
  11. Generating effective test cases for self-driving cars from police reports. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, Tallinn Estonia, 257–267.
  12. Automatically testing self-driving cars with search-based procedural content generation. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Beijing, China, 318–328.
  13. A comprehensive study of autonomous vehicle bugs. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. IEEE, Seoul, South Korea, 385–396.
  14. Jia Cheng Han and Zhi Quan Zhou. 2020. Metamorphic fuzz testing of autonomous vehicles. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops. 380–385.
  15. Preliminary evaluation of path-aware crossover operators for search-based test data generation for autonomous driving. In 2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST). IEEE, Madrid, Spain, 44–47.
  16. Efficient online testing for DNN-enabled systems using surrogate-assisted and many-objective optimization. In Proceedings of the 44th International Conference on Software Engineering. IEEE, Pittsburgh Pennsylvania, 811–822.
  17. Many-objective reinforcement learning for online testing of dnn-enabled systems. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1814–1826.
  18. PhysCov: Physical Test Coverage for Autonomous Vehicles. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 449–461.
  19. Planning-oriented Autonomous Driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  20. sceno RITA: Generating Diverse, Fully-Mutable, Test Scenarios for Autonomous Vehicle Planning. IEEE Transactions on Software Engineering (2023).
  21. Autoware on board: Enabling autonomous vehicles with embedded systems. In 2018 ACM/IEEE 9th International Conference on Cyber-Physical Systems (ICCPS). IEEE, 287–296.
  22. LG Electronics. [n. d.]. SVL Simulator Sunset. https://www.svlsimulator.com/news/2022-01-20-svl-simulator-sunset/.
  23. AV-FUZZER: Finding safety violations in autonomous driving systems. In 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE). IEEE, Coimbra, Portugal, 25–36.
  24. Testing of autonomous driving systems: where are we and where should we go?. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 31–43.
  25. Targeting requirements violations of autonomous driving systems by dynamic evolutionary search. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 279–291.
  26. Depiction of priority light-vehicle pre-crash scenarios for safety applications based on vehicle-to-vehicle communications. Technical Report DOT HS 811 732. National Highway Traffic Safety Administration, U.S. Department of Transportation, Washington, DC.
  27. Pre-crash scenarios at road junctions: A clustering method for car crash data. Accident Analysis & Prevention 107 (2017), 137–151.
  28. Automatic identification of critical scenarios in a public dataset of 6000 km of public-road driving. In 26th International Technical Conference on the Enhanced Safety of Vehicles (ESV). Mira Smart, Eindhoven, Netherlands.
  29. MDPFuzz: Testing Models Solving Markov Decision Processes. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (Virtual, South Korea) (ISSTA 2022). Association for Computing Machinery, New York, NY, USA, 378–390. https://doi.org/10.1145/3533767.3534388
  30. A scenario-based assessment approach for automated driving by using time series classification of human-driving behaviour. In 2016 IEEE 19th international conference on intelligent transportation systems (ITSC). IEEE, Rio de Janeiro, Brazil, 1360–1365.
  31. LGSVL simulator: A high fidelity simulator for autonomous driving. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC). IEEE, Rhodes, Greece, 1–6.
  32. Collision avoidance testing for autonomous driving systems on complete maps. In 2021 IEEE Intelligent Vehicles Symposium (IV). IEEE, Nagoya, Japan, 179–185.
  33. Route coverage testing for autonomous vehicles via map modeling. In 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, Xi’an, China, 11450–11456.
  34. Systematic testing of autonomous driving systems using map topology-based scenario classification. In Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, Melbourne, Australia, 1342–1346.
  35. A framework for automated driving system testable cases and scenarios. Technical Report. United States. Department of Transportation. National Highway Traffic Safety.
  36. Shuai Wang and Zhendong Su. 2021. Metamorphic object insertion for testing object detection systems. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (Virtual Event, Australia). Association for Computing Machinery, New York, NY, USA, 1053–1065.
  37. Towards the Robustness of Multiple Object Tracking Systems. In 2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE). IEEE, 402–413.
  38. Testing and validating machine learning classifiers by metamorphic testing. Journal of Systems and Software 84, 4 (2011), 544–558.
  39. DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 132–142.
  40. Xudong Zhang and Yan Cai. 2023. Building Critical Testing Scenarios for Autonomous Driving from Real Accidents. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 462–474.
  41. End-to-end urban driving by imitating a reinforcement learning coach. In Proceedings of the IEEE/CVF international conference on computer vision. 15222–15232.
  42. Neural network guided evolutionary fuzzing for finding traffic violations of autonomous vehicles. IEEE Transactions on Software Engineering 49, 4 (2023), 1860–1875.
  43. Specification-based Autonomous Driving System Testing. IEEE Transactions on Software Engineering (2023), 1–19.
  44. Zhi Quan Zhou and Liqun Sun. 2019. Metamorphic testing of driverless cars. Commun. ACM 62, 3 (2019), 61–67.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Mingfei Cheng (16 papers)
  2. Yuan Zhou (251 papers)
  3. Xiaofei Xie (104 papers)
  4. Junjie Wang (164 papers)
  5. Guozhu Meng (28 papers)
  6. Kairui Yang (7 papers)