Papers
Topics
Authors
Recent
Search
2000 character limit reached

A Diversity Analysis of Safety Metrics Comparing Vehicle Performance in the Lead-Vehicle Interaction Regime

Published 26 Jun 2023 in cs.RO, cs.SY, and eess.SY | (2306.14657v1)

Abstract: Vehicle performance metrics analyze data sets consisting of subject vehicle's interactions with other road users in a nominal driving environment and provide certain performance measures as outputs. To the best of the authors' knowledge, the vehicle safety performance metrics research dates back to at least 1967. To date, there still does not exist a community-wide accepted metric or a set of metrics for vehicle safety performance assessment and justification. This issue gets further amplified with the evolving interest in Advanced Driver Assistance Systems and Automated Driving Systems. In this paper, the authors seek to perform a unified study that facilitates an improved community-wide understanding of vehicle performance metrics using the lead-vehicle interaction operational design domain as a common means of performance comparison. In particular, the authors study the diversity (including constructive formulation discrepancies and empirical performance differences) among 33 base metrics with up to 51 metric variants (with different choices of hyper-parameters) in the existing literature, published between 1967 and 2022. Two data sets are adopted for the empirical performance diversity analysis, including vehicle trajectories from normal highway driving environment and relatively high-risk incidents with collisions and near-miss cases. The analysis further implies that (i) the conceptual acceptance of a safety metric proposal can be problematic if the assumptions, conditions, and types of outcome assurance are not justified properly, and (ii) the empirical performance justification of an acceptable metric can also be problematic as a dominant consensus is not observed among metrics empirically.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (90)
  1. Y. Lin and M. Althoff, “Commonroad-crime: A toolbox for criticality measures of autonomous vehicles,” in 2023 IEEE Intelligent Vehicles Symposium (IV), 2023.
  2. S. Shalev-Shwartz, S. Shammah, and A. Shashua, “On a formal model of safe and scalable self-driving cars,” arXiv preprint arXiv:1708.06374, 2017.
  3. D. Zhao, X. Huang, H. Peng, H. Lam, and D. J. LeBlanc, “Accelerated evaluation of automated vehicles in car-following maneuvers,” IEEE Transactions on Intelligent Transportation Systems, vol. 19, no. 3, pp. 733–744, 2018.
  4. C. Fan, B. Qi, and S. Mitra, “Data-driven formal reasoning and their applications in safety analysis of vehicle autonomy features,” IEEE Design & Test, vol. 35, no. 3, pp. 31–38, 2018.
  5. B. Weng, L. Capito, Ü. Özgüner, and K. Redmill, “A finite-sampling, operational domain specific, and provably unbiased connected and automated vehicle safety metric,” IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 6, pp. 6650–6662, 2023.
  6. S. R. Perkins and J. I. Harris, “Traffic conflict characteristics: Freeway curve and exit area F1, December, 1966,” General Motors Corporation, Tech. Rep., 1967.
  7. J. Hayward, “Near-miss determination through use of a scale of danger,” Highway Research Record, vol. 384, pp. 24–35, 1972.
  8. G. Seetharaman, A. Lakhotia, and E. P. Blasch, “Unmanned vehicles come of age: The DARPA grand challenge,” Computer, vol. 39, no. 12, pp. 26–29, 2006.
  9. H. Wakabayashi and K. Renge, “Traffic conflict analysis using vehicle tracking system with digital VCR and new conflict indicator under high speed and congested traffic environment,” in Proc. of the 19th Dresden Conference on Traffic and Transportation Sciences,” Mobility and Traffic Management in a Networked World”. CD-ROM (43.1-43.17), vol. 20, Citeseer.   Japan Society of Civil Engineers, 2003, pp. 949–956.
  10. K. Ozbay, H. Yang, B. Bartin, and S. Mudigonda, “Derivation and validation of new simulation-based surrogate safety measure,” Transportation Research Record, vol. 2083, no. 1, pp. 105–113, 2008. [Online]. Available: https://doi.org/10.3141/2083-12
  11. B. Weng, S. J. Rao, E. Deosthale, S. Schnelle, and F. Barickman, “Model predictive instantaneous safety metric for evaluation of automated driving systems,” in 2020 IEEE Intelligent Vehicles Symposium (IV).   IEEE, 2020, pp. 1899–1906.
  12. N. Uno, Y. Iida, S. Itsubo, and S. Yasuhara, “A microscopic analysis of traffic conflict caused by lane-changing vehicle at weaving section,” in Proceedings of the 13th Mini-EURO Conference-Handling Uncertainty in the Analysis of Traffic and Transportation Systems, Bari, Italy, 2002, pp. 10–13.
  13. M. Okamura, A. Fukuda, H. Morita, H. Suzuki, and M. Nakazawa, “Impact evaluation of a driving support system on traffic flow by microscopic traffic simulation,” Advances in Transportation Studies, no. Special Issue 2011, pp. 99–102, 2011.
  14. D. F. Cooper and N. Ferguson, “Traffic studies at T-Junctions. 2. A conflict simulation Record,” Traffic Engineering & Control, vol. 17, no. Analytic, pp. 306–309, 1976.
  15. C. Hupfer, “Deceleration to safety time (DST)-A useful figure to evaluate traffic safety,” in ICTCT Conference Proceedings of Seminar, vol. 3, 1997, pp. 5–7.
  16. J. Jansson, “Collision Avoidance Theory : With application to automotive collision mitigation,” Ph.D. dissertation, Linköping University Electronic Press, 2005.
  17. H. C. Chin, S. T. Quek, and R. Cheu, “Quantitative examination of traffic conflicts,” Transportation Research Record, vol. 1376, no. 7, pp. 67–74, 1992.
  18. F. J. C. Cunto, “Assessing safety performance of transportation systems using microscopic simulation,” Ph.D. dissertation, University of Waterloo, 2008.
  19. M. M. Minderhoud and P. H. Bovy, “Extended time-to-collision measures for road traffic safety assessment,” Accident Analysis & Prevention, vol. 33, no. 1, pp. 89–97, 2001.
  20. C. Oh, S. Park, and S. G. Ritchie, “A method for identifying rear-end collision risks using inductive loop detectors,” Accident Analysis & Prevention, vol. 38, no. 2, pp. 295–301, 2006.
  21. M. S. Rahman and M. Abdel-Aty, “Longitudinal safety evaluation of connected vehicles’ platooning on expressways,” Accident Analysis & Prevention, vol. 117, pp. 381–391, 2018.
  22. S. S. Mahmud, L. Ferreira, M. S. Hoque, and A. Tavassoli, “Application of proximal surrogate indicators for safety evaluation: A review of recent developments and research needs,” IATSS research, vol. 41, no. 4, pp. 153–163, 2017.
  23. K. Mattas, M. Makridis, G. Botzoris, A. Kriston, F. Minarini, B. Papadopoulos, F. Re, G. Rognelund, and B. Ciuffo, “Fuzzy surrogate safety metrics for real-time assessment of rear-end collision risk. A study based on empirical observations.” Accident Analysis & Prevention, vol. 148, p. 105794, 2020.
  24. UNECE, “Proposal for a new UN Regulation on uniform provisions concerning the approval of vehicles with regards to Automated Lane Keeping System,” United Nations Economic Commission for Europe (UNECE), Tech. Rep., 2020.
  25. V. Astarita and V. P. Giofré, “From traffic conflict simulation to traffic crash simulation: Introducing traffic safety indicators based on the explicit simulation of potential driver errors,” Simulation Modelling Practice and Theory, vol. 94, pp. 215–236, 2019.
  26. J. Wishart, S. Como, M. Elli, B. Russo, J. Weast, N. Altekar, E. James, and Y. Chen, “Driving safety performance assessment metrics for ADS-equipped vehicles,” SAE International Journal of Advances and Current Practices in Mobility, vol. 2, no. 2020-01-1206, pp. 2881–2899, 2020.
  27. D. Otte, C. Krettek, H. Brunner, and H. Zwipp, “Scientific approach and methodology of a new in-depth investigation study in Germany called GIDAS,” in Proceedings: International Technical Conference on the Enhanced Safety of Vehicles, vol. 2003.   National Highway Traffic Safety Administration, 2003, pp. 10–p.
  28. H. Bellem, B. Thiel, M. Schrauf, and J. F. Krems, “Comfort in automated driving: An analysis of preferences for different automated driving styles and their dependence on personality traits,” Transportation research part F: Traffic Psychology and Behaviour, vol. 55, pp. 90–100, 2018.
  29. K. Vogel, “What characterizes a “free vehicle” in an urban area?” Transportation Research Part F: Traffic Psychology and Behaviour, vol. 5, no. 1, pp. 15–29, 2002.
  30. J. Barceló Bugeda, A.-G. Dumont, L. Montero Mercadé, J. Perarnau, and A. Torday, “Safety indicators for microsimulation-based assessments,” in Transportation Research Board 82nd Annual Meeting.   TRB, 2003, pp. 1–18.
  31. N. Kalra and S. M. Paddock, “Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?” Transportation Research Part A: Policy and Practice, vol. 94, pp. 182–193, 2016.
  32. M. S. Elli, J. Wishart, S. Como, S. Dhakshinamoorthy, and J. Weast, “Evaluation of Operational Safety Assessment (OSA) metrics for automated vehicles in simulation,” SAE Technical Paper, pp. 01–0868, 2021.
  33. V. C. Jammula, J. Wishart, and Y. Yang, “Evaluation of Operational Safety Assessment (OSA) metrics for automated vehicles using real-world data,” SAE Technical Paper, pp. 01–0062, 2022.
  34. P. M. Junietz, “Microscopic and macroscopic risk metrics for the safety validation of automated driving,” Ph.D. dissertation, Technische Universität, 2019.
  35. E. R. Griffor, C. Greer, and D. A. Wollman, “Workshop report: Consensus safety measurement methodologies for automated driving system-equipped vehicles,” National Institute of Standards and Technology, Tech. Rep., 2019.
  36. N. Kidambi, J. Wishart, M. Elli, and S. Como, “Sensitivity of automated vehicle Operational Safety Assessment (OSA) metrics to measurement and parameter uncertainty,” SAE Technical Paper, 2022.
  37. K. Mattas, G. Albano, R. Donà, M. C. Galassi, R. Suarez-Bertoa, S. Vass, and B. Ciuffo, “Driver models for the definition of safety requirements of automated vehicles in international regulations. Application to motorway driving conditions,” Accident Analysis and Prevention, vol. 174, p. 106743, 2022.
  38. T. Saito and M. Rehmsmeier, “The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets,” PloS one, vol. 10, no. 3, p. e0118432, 2015.
  39. K. Czarnecki, “Operational design domain for automated driving systems,” Taxonomy of Basic Terms “, Waterloo Intelligent Systems Engineering (WISE) Lab, University of Waterloo, Canada, 2018.
  40. R. Krajewski, J. Bock, L. Kloeker, and L. Eckstein, “The HighD dataset: A drone dataset of naturalistic vehicle trajectories on German highways for validation of highly automated driving systems,” in 2018 21st International Conference on Intelligent Transportation Systems (ITSC).   IEEE, 2018, pp. 2118–2125.
  41. S. Ulbrich, T. Menzel, A. Reschka, F. Schuldt, and M. Maurer, “Defining and substantiating the terms scene, situation, and scenario for automated driving,” in 2015 IEEE 18th International Conference on Intelligent Transportation Systems.   IEEE, 2015, pp. 982–988.
  42. E. Thorn, S. C. Kimmel, M. Chaka, B. A. Hamilton et al., “A framework for automated driving system testable cases and scenarios,” United States. Department of Transportation. National Highway Traffic Safety, Tech. Rep., 2018.
  43. B. Weng, G. A. Castillo, W. Zhang, and A. Hereid, “On the comparability and optimal aggressiveness of the adversarial scenario-based safety testing of robots,” IEEE Transactions on Robotics, pp. 1–20, 2023.
  44. S. Feng, H. Sun, X. Yan, H. Zhu, Z. Zou, S. Shen, and H. X. Liu, “Dense reinforcement learning for safety validation of autonomous vehicles,” Nature, vol. 615, no. 7953, pp. 620–627, 2023.
  45. N. Euro, “Test protocol - AEB car-to-car systems,” Brussels, Belgium: Eur. New Car Assess. Programme (Euro NCAP), 2017.
  46. S. Feng, Y. Feng, C. Yu, Y. Zhang, and H. X. Liu, “Testing scenario library generation for connected and automated vehicles, part I: Methodology,” IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 3, pp. 1573–1582, 2020.
  47. B. Weng, “A class of model predictive safety performance metrics for driving behavior evaluation,” in 2021 IEEE International Intelligent Transportation Systems Conference (ITSC).   IEEE, 2021, pp. 180–187.
  48. P. Junietz, F. Bonakdar, B. Klamann, and H. Winner, “Criticality metric for the safety validation of automated driving using model predictive trajectory optimization,” in 2018 21st International Conference on Intelligent Transportation Systems (ITSC).   IEEE, 2018, pp. 60–65.
  49. C. Nowakowski, S. E. Shladover, C.-Y. Chan, and H.-S. Tan, “Development of California regulations to govern testing and operation of automated driving systems,” Transportation Research Record, vol. 2489, no. 1, pp. 137–144, 2015.
  50. H. He and E. A. Garcia, “Learning from imbalanced data,” IEEE Transactions on knowledge and data engineering, vol. 21, no. 9, pp. 1263–1284, 2009.
  51. A. P. Bradley, “The use of the area under the ROC curve in the evaluation of machine learning algorithms,” Pattern recognition, vol. 30, no. 7, pp. 1145–1159, 1997.
  52. B. W. Matthews, “Comparison of the predicted and observed secondary structure of T4 phage lysozyme,” Biochimica et Biophysica Acta (BBA)-Protein Structure, vol. 405, no. 2, pp. 442–451, 1975.
  53. M. L. McHugh, “Interrater reliability: The kappa statistic,” Biochemia medica, vol. 22, no. 3, pp. 276–282, 2012.
  54. C. J. Willmott, “On the validation of models,” Physical geography, vol. 2, no. 2, pp. 184–194, 1981.
  55. M. Buckland and F. Gey, “The relationship between recall and precision,” Journal of the American society for information science, vol. 45, no. 1, pp. 12–19, 1994.
  56. D. Powers, “Evaluation: From precision, recall and F-measure to ROC, informedness, markedness & correlation,” Journal of Machine Learning Technologies, vol. 2, no. 1, pp. 37–63, 2011.
  57. M. Grandini, E. Bagli, and G. Visani, “Metrics for multi-class classification: An overview,” arXiv preprint arXiv:2008.05756, 2020.
  58. P. Seiniger and J. Gail, “A methodology to derive precision requirements for automatic emergency braking (AEB) test procedures,” in 24th International Technical Conference on the Enhanced Safety of Vehicles (ESV) National Highway Traffic Safety Administration, no. 15-0315, 2015.
  59. F. Navin, M. Macnabb, and C. Nicolletti, “Vehicle traction experiments on snow and ice,” SAE transactions, vol. 105, no. 6, pp. 702–713, 1996.
  60. S. Hirst, “The format and presentation of collision warning,” Ergonomics and Safety of Intelligent Driver Interface, pp. 203–219, 1997.
  61. Y. Huang, M. S. Elli, J. Weast, Y. Lou, S. Lu, and Y. Chen, “RSS model calibration and evaluation for AV driving safety based on naturalistic driving data,” IFAC-PapersOnLine, vol. 54, no. 20, pp. 430–436, 2021.
  62. Y. Meng, D. Sun, Z. Qiu, M. T. B. Waez, and C. Fan, “Learning density distribution of reachable states for autonomous systems,” in Conference on Robot Learning.   PMLR, 2022, pp. 124–136.
  63. M. Althoff and J. M. Dolan, “Online verification of automated road vehicles using reachability analysis,” IEEE Transactions on Robotics, vol. 30, no. 4, pp. 903–918, 2014.
  64. M. Klischat and M. Althoff, “Generating critical test scenarios for automated vehicles with evolutionary algorithms,” in 2019 IEEE Intelligent Vehicles Symposium (IV).   IEEE, 2019, pp. 2352–2358.
  65. D. Nistér, H.-L. Lee, J. Ng, and Y. Wang, “An introduction to the Safety Force Field,” Nvidia Inc., Tech. Rep., 3 2019.
  66. R. Chen, R. Sherony, and H. C. Gabler, “Comparison of time to collision and enhanced time to collision at brake application during normal driving,” SAE Technical Paper, Tech. Rep., 2016.
  67. A. Eidehall, “Multi-target threat assessment for automotive applications,” in 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC).   IEEE, 2011, pp. 433–438.
  68. H. Suk, T. Kim, H. Park, P. Yadav, J. Lee, and S. Kim, “Rationale-aware autonomous driving policy utilizing Safety Force Field implemented on CARLA simulator,” arXiv preprint arXiv:2211.10237, 2022.
  69. B. L. Allen, B. T. Shin, and P. J. Cooper, “Analysis of traffic conflicts and collision,” Transportation Research Record, vol. 677, pp. 67–74, 1978.
  70. A. Laureshyn, Å. Svensson, and C. Hydén, “Evaluation of traffic safety, based on micro-level behavioural data: Theoretical framework and first implementation,” Accident Analysis & Prevention, vol. 42, no. 6, pp. 1637–1646, 2010.
  71. A. Hansson, “Studies in driver behaviour, with applications in traffic design and planning: Two examples,” Swedish National Road and Traffic Research Institute, Tech. Rep., 1975.
  72. S. G. Shelby et al., “Delta-V as a measure of traffic conflict severity,” in 3rd International Conference on Road Safety and Simulation. September, 2011, pp. 14–16.
  73. O. Bagdadi, “Estimation of the severity of safety critical events,” Accident Analysis & Prevention, vol. 50, pp. 167–174, 2013.
  74. W. K. Alhajyaseen, “The integration of conflict probability and severity for the safety assessment of intersections,” Arabian Journal for Science and Engineering, vol. 40, no. 2, pp. 421–430, 2015.
  75. J. Hillenbrand, A. M. Spieker, and K. Kroschel, “A multilevel collision mitigation approach — Its situation assessment, decision making, and performance tradeoffs,” IEEE Transactions on intelligent transportation systems, vol. 7, no. 4, pp. 528–540, 2006.
  76. A. Tamke, T. Dang, and G. Breuel, “A flexible method for criticality assessment in driver assistance systems,” in 2011 IEEE Intelligent Vehicles Symposium (IV).   IEEE, 2011, pp. 697–702.
  77. W. Wachenfeld, P. Junietz, R. Wenzel, and H. Winner, “The worst-time-to-collision metric for situation identification,” in 2016 IEEE Intelligent Vehicles Symposium (IV).   IEEE, 2016, pp. 729–734.
  78. F. Barickman, J. L. Every, B. Weng, S. Schnelle, and S. Rao, “Instantaneous Safety Metric,” National Highway Traffic Safety Administration, PowerPoint presentation, June, vol. 25, 2019.
  79. J. Dahl, G. R. de Campos, C. Olsson, and J. Fredriksson, “Collision avoidance: A literature review on threat-assessment techniques,” IEEE Transactions on Intelligent Vehicles, vol. 4, no. 1, pp. 101–113, 2018.
  80. T. Petzoldt, “On the relationship between pedestrian gap acceptance and time to arrival estimates,” Accident Analysis & Prevention, vol. 72, pp. 127–133, 2014.
  81. K. Ogawa, “An analysis of traffic conflict phenomenon of bicycles using Space Occupancy Index,” Journal of the Eastern Asia Society for Transportation Studies, vol. 7, pp. 1820–1827, 2007.
  82. S. Cafiso, A. G. Garcia, R. Cavarra, and M. Rojas, “Crosswalk safety evaluation using a Pedestrian Risk Index as traffic conflict measure,” in Proceedings of the 3rd International Conference on Road safety and Simulation, vol. 15, 2011.
  83. A. Varhelyi, “Drivers’ speed behaviour at a zebra crossing: A case study,” Accident Analysis & Prevention, vol. 30, no. 6, pp. 731–743, 1998.
  84. C. Wang, Y. Xie, H. Huang, and P. Liu, “A review of surrogate safety measures and their applications in connected and automated vehicles safety modeling,” Accident analysis and prevention, vol. 157, pp. 106 157–106 157, 2021.
  85. L. Westhofen, C. Neurohr, T. Koopmann, M. Butz, B. Schütt, F. Utesch, B. Neurohr, C. Gutenkunst, and E. Böde, “Criticality metrics for automated driving: A review and suitability analysis of the state of the art,” Archives of Computational Methods in Engineering, vol. 30, no. 1, pp. 1–35, 2022.
  86. I. IEEE VT, “White paper-literature review on kinematic properties of road users for use on safety-related models for automated driving systems,” IEEE Standards, pp. 1–35, 2022.
  87. S. Como and J. Wishart, “Evaluating automated vehicle scenario navigation using the Operational Safety Assessment (OSA) methodology,” SAE Technical Paper, Tech. Rep., 2023.
  88. T. A. Dingus, S. G. Klauer, V. L. Neale, A. Petersen, S. E. Lee, J. Sudweeks, M. A. Perez, J. Hankey, D. Ramsey, S. Gupta et al., “The 100-car naturalistic driving study: Phase II - Results of the 100-car field experiment,” United States. Department of Transportation. National Highway Traffic Safety, Tech. Rep., 2006.
  89. O. Bagdadi and A. Várhelyi, “Development of a method for detecting jerks in safety critical events,” Accident Analysis & Prevention, vol. 50, pp. 83–91, 2013.
  90. M. Althoff, G. Frehse, and A. Girard, “Set propagation techniques for reachability analysis,” Annual Review of Control, Robotics, and Autonomous Systems, vol. 4, pp. 369–395, 2021.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up