A Stochastic Approach to Classification Error Estimates in Convolutional Neural Networks (2401.06156v1)
Abstract: This technical report presents research results achieved in the field of verification of trained Convolutional Neural Network (CNN) used for image classification in safety-critical applications. As running example, we use the obstacle detection function needed in future autonomous freight trains with Grade of Automation (GoA) 4. It is shown that systems like GoA 4 freight trains are indeed certifiable today with new standards like ANSI/UL 4600 and ISO 21448 used in addition to the long-existing standards EN 50128 and EN 50129. Moreover, we present a quantitative analysis of the system-level hazard rate to be expected from an obstacle detection function. It is shown that using sensor/perceptor fusion, the fused detection system can meet the tolerable hazard rate deemed to be acceptable for the safety integrity level to be applied (SIL-3). A mathematical analysis of CNN models is performed which results in the identification of classification clusters and equivalence classes partitioning the image input space of the CNN. These clusters and classes are used to introduce a novel statistical testing method for determining the residual error probability of a trained CNN and an associated upper confidence limit. We argue that this greybox approach to CNN verification, taking into account the CNN model's internal structure, is essential for justifying that the statistical tests have covered the trained CNN with its neurons and inter-layer mappings in a comprehensive way.
- TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https://www.tensorflow.org/. Software available from tensorflow.org.
- Charu C. Aggarwal. Neural Networks and Deep Learning. Springer Nature Switzerland AG, Cham, Switzerland, 2018.
- Experimental stability analysis of neural networks in classification problems with confidence sets for persistence diagrams. Neural Networks, 143:42–51, 2021. ISSN 0893-6080. doi: https://doi.org/10.1016/j.neunet.2021.05.007. URL https://www.sciencedirect.com/science/article/pii/S0893608021001994.
- Finding and removing clever hans: Using explanation methods to debug and improve deep models. Information Fusion, 77:261–295, 2022. ISSN 1566-2535. doi: 10.1016/j.inffus.2021.07.015.
- Tom M. Apostol. Mathematical Analysis. Addison-Wesley Publishing Company, reading, Massachusetts, second edition, 1974.
- Testing, validation, and verification of robotic and autonomous systems: A systematic review. ACM Trans. Softw. Eng. Methodol., may 2022. ISSN 1049-331X. doi: 10.1145/3542945. URL https://doi.org/10.1145/3542945. Just Accepted.
- Strategy Synthesis for Autonomous Driving in a Moving Block Railway System with Uppaal Stratego. In Alexey Gotsman and Ana Sokolova, editors, Formal Techniques for Distributed Objects, Components, and Systems, Lecture Notes in Computer Science, pages 3–21, Cham, 2020. Springer International Publishing. ISBN 978-3-030-50086-3. doi: 10.1007/978-3-030-50086-3˙1.
- Météor: A successful application of B in a large project. In Jeannette M. Wing, Jim Woodcock, and Jim Davies, editors, FM’99 - Formal Methods, World Congress on Formal Methods in the Development of Computing Systems, Toulouse, France, September 20-24, 1999, Proceedings, Volume I, volume 1708 of Lecture Notes in Computer Science, pages 369–387. Springer, 1999. doi: 10.1007/3-540-48119-2_22. URL https://doi.org/10.1007/3-540-48119-2_22.
- A singular riemannian geometry approach to deep neural networks i. theoretical foundations. Neural Networks, 158:331–343, 2023a. ISSN 0893-6080. doi: https://doi.org/10.1016/j.neunet.2022.11.022. URL https://www.sciencedirect.com/science/article/pii/S0893608022004634.
- A singular riemannian geometry approach to deep neural networks ii. reconstruction of 1-d equivalence classes. Neural Networks, 158:344–358, 2023b. ISSN 0893-6080. doi: https://doi.org/10.1016/j.neunet.2022.11.026. URL https://www.sciencedirect.com/science/article/pii/S0893608022004671.
- A Runtime Risk Assessment Concept for Safe Reconfiguration in Open Adaptive Systems. In Stefano Tonetta, Erwin Schoitsch, and Friedemann Bitsch, editors, Computer Safety, Reliability, and Security, Lecture Notes in Computer Science, pages 309–316. Springer International Publishing, 2017. ISBN 978-3-319-66284-8.
- Programming multi-agent systems in AgentSpeak using Jason. John Wiley&Sons Ltd, West Sussex, England, 2007.
- Risk analyses for obstacle detection in automatic driving. 115(3):12–20, 2023.
- Identification of classification clusters in convolutional neural networks. In Martin Fränzle, Jürgen Niehaus, and Bernd Westphal, editors, Engineering Safe and Trustworthy Cyber Physical Systems – Essays Dedicated to Werner Damm on the Occasion of His 71st Birthday, Lecture Notes in Computer Science. Springer, 2024. to appear.
- CENELEC. EN 50128:2011 Railway applications - Communication, signalling and processing systems - Software for railway control and protection systems. 2011.
- CENELEC. EN 50129 Railway applications - Communication, signalling and processing systems - Safety related electronic systems for signalling. 2019.
- Quantitative Projection Coverage for Testing ML-enabled Autonomous Systems. In ATVA, LNCS, pages 126–142, Cham, 2018. Springer. ISBN 978-3-030-01090-4. doi: 10.1007/978-3-030-01090-4˙8. URL https://doi.org/10.1007/978-3-030-01090-4_8.
- Compositional explanations for image classifiers. CoRR, abs/2103.03622, 2021. URL https://arxiv.org/abs/2103.03622.
- François Chollet et al. Keras. https://keras.io, 2015.
- EN 50126-1. Railway Applications - The Specification and Demonstration of Reliability, Availability, Maintainability and Safety (RAMS) - Part 1: Generic RAMS Process. CENELEC, Brussels, 2017.
- EN 50126-2. Railway Applications - The Specification and Demonstration of Reliability, Availability, Maintainability and Safety (RAMS) - Part 2: Systems Approach to Safety. CENELEC, Brussels, 2017.
- Towards a framework for certification of reliable autonomous systems. Autonomous Agents and Multi-Agent Systems, 35(1):8, December 2020. ISSN 1573-7454. doi: 10.1007/s10458-020-09487-2. URL https://doi.org/10.1007/s10458-020-09487-2.
- Birthday paradox, coupon collectors, caching algorithms and self-organizing search. Discrete Applied Mathematics, 39(3):207–229, 1992. ISSN 0166-218X. doi: https://doi.org/10.1016/0166-218X(92)90177-C. URL https://www.sciencedirect.com/science/article/pii/0166218X9290177C.
- Safety integrity through self-adaptation for multi-sensor event detection: Methodology and case-study. Future Generation Computer Systems, 112:965–981, 2020. ISSN 0167-739X. doi: https://doi.org/10.1016/j.future.2020.06.036. URL https://www.sciencedirect.com/science/article/pii/S0167739X19333734.
- A vision of intelligent train control. In Simon Collart Dutilleul, Anne E. Haxthausen, and Thierry Lecomte, editors, Reliability, Safety, and Security of Railway Systems. Modelling, Analysis, Verification, and Certification - 4th International Conference, RSSRail 2022, Paris, France, June 1-2, 2022, Proceedings, volume 13294 of Lecture Notes in Computer Science, pages 192–208. Springer, 2022. doi: 10.1007/978-3-031-05814-1_14. URL https://doi.org/10.1007/978-3-031-05814-1_14.
- Riskstructures: A design algebra for risk-aware machines. Formal Aspects Comput., 33(4-5):763–802, 2021. doi: 10.1007/s00165-021-00545-4. URL https://doi.org/10.1007/s00165-021-00545-4.
- Probabilistic risk assessment of an obstacle detection system for goa 4 freight trains. In Proceedings of the 9th ACM SIGPLAN International Workshop on Formal Techniques for Safety-Critical Systems, FTSCS 2023, page 26–36, New York, NY, USA, 2023a. Association for Computing Machinery. ISBN 9798400703980. doi: 10.1145/3623503.3623533.
- Probabilistic risk assessment of an obstacle detection system for GoA 4 freight trains. CoRR, abs/2306.14814, 2023b. doi: 10.48550/arXiv.2306.14814. URL https://doi.org/10.48550/arXiv.2306.14814.
- Test and training data generation for object recognition in the railway domain. In Paolo Masci, Cinzia Bernardeschi, Pierluigi Graziani, Mario Koddenbrock, and Maurizio Palmieri, editors, Software Engineering and Formal Methods. SEFM 2022 Collocated Workshops - AI4EA, F-IDE, CoSim-CPS, CIFMA, Berlin, Germany, September 26-30, 2022, Revised Selected Papers, volume 13765 of Lecture Notes in Computer Science, pages 5–16. Springer, 2022. doi: 10.1007/978-3-031-26236-4_1. URL https://doi.org/10.1007/978-3-031-26236-4_1.
- A formal model of train control with AI-based obstacle detection. In RSSRail, volume 0 of LNCS, pages 1–16, Berlin, DE, 2023. Springer. In press.
- Did we test all scenarios for automated and autonomous driving systems? In 2019 IEEE Intelligent Transportation Systems Conference, ITSC 2019, Auckland, New Zealand, October 27-30, 2019, pages 2950–2955. IEEE, 2019. doi: 10.1109/ITSC.2019.8917326. URL https://doi.org/10.1109/ITSC.2019.8917326.
- A. E. Haxthausen and J. Peleska. Formal Development and Verification of a Distributed Railway Control System. IEEE Transaction on Software Engineering, 26(8):687–701, 2000.
- Standardisation Considerations for Autonomous Train Control - Technical Report. Technical report, Zenodo, February 2022. URL https://zenodo.org/record/6185229.
- Complete model-based equivalence class testing. Software Tools for Technology Transfer, 18(3):265–283, 2016. doi: 10.1007/s10009-014-0356-8. URL http://dx.doi.org/10.1007/s10009-014-0356-8.
- ISO. ISO/DIS 21448: Road vehicles — Safety of the intended functionality. European Committee for Electronic Standardization, 2021. ICS: 43.040.10, Draft International Standard.
- Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability? RAND Corporation, Santa Monica, CA, 2016. doi: 10.7249/RR1478.
- The vision of autonomic computing. Computer, 36(1):41–50, 2003. doi: 10.1109/MC.2003.1160055. URL https://doi.org/10.1109/MC.2003.1160055.
- Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980.
- Toward a Framework for Highly Automated Vehicle Safety Validation. In Proceedings of the 2018 SAE World Congress / SAE 2018-01-1071, 2018. URL https://users.ece.cmu.edu/~koopman/pubs/koopman18_av_safety_validation.pdf.
- Autonomous vehicle safety: An interdisciplinary challenge. IEEE Intell. Transp. Syst. Mag., 9(1):90–96, 2017. doi: 10.1109/MITS.2016.2583491. URL https://doi.org/10.1109/MITS.2016.2583491.
- Credible autonomy safety argumentation. In Proceedings of the 27th Safety-Critical Systems Symposium, Feb. 2019., 2019. URL https://users.ece.cmu.edu/~koopman/pubs/Koopman19_SSS_CredibleSafetyArgumentation.pdf.
- Demir N. Kupeli. Singular Semi-Riemannian Geometry. Springer Science+Business Media Dordrecht, 1996. ISBN 978-90-481-4689-5. doi: 10.1007/978-94-015-8761-7.
- PRISM 4.0: Verification of probabilistic real-time systems. In CAV, volume 6806 of LNCS, pages 585–591, Snowbird, UT, 2011. Springer. doi: 10.1007/978-3-642-22110-1˙47. URL https://doi.org/10.1007/978-3-642-22110-1_47.
- Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2, 2010.
- Railway sign detection and classification. In 2006 IEEE Intelligent Transportation Systems Conference, pages 1358–1363, 2006. doi: 10.1109/ITSC.2006.1707412.
- Standardisation considerations for autonomous train control. In Tiziana Margaria and Bernhard Steffen, editors, Leveraging Applications of Formal Methods, Verification and Validation. Practice - 11th International Symposium, ISoLA 2022, Rhodes, Greece, October 22-30, 2022, Proceedings, Part IV, volume 13704 of Lecture Notes in Computer Science, pages 286–307. Springer, 2022. doi: 10.1007/978-3-031-19762-8_22. URL https://doi.org/10.1007/978-3-031-19762-8_22.
- Risk and safety analysis of main line autonomous train operation: Context, challenges and solutions. In Lambda Mu 21, pages 1–11, Reims, France, 2018. HAL. URL https://hal.archives-ouvertes.fr/hal-02073235.
- A Review of Vision-Based On-Board Obstacle Detection and Distance Estimation in Railways. Sensors (Basel, Switzerland), 21(10):3452, May 2021. ISSN 1424-8220. doi: 10.3390/s21103452. URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8156009/.
- Lothar Sachs. Applied Statistics – A Handbook of Techniques. Springer, New York, Berlin, Heidelberg, Tokyo, 1984.
- Towards explainable artificial intelligence. In Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller, editors, Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, volume 11700 of Lecture Notes in Computer Science, pages 5–22. Springer, 2019. doi: 10.1007/978-3-030-28954-6_1. URL https://doi.org/10.1007/978-3-030-28954-6_1.
- Claudio Filipi Gonçalves Dos Santos and João Paulo Papa. Avoiding overfitting: A survey on regularization methods for convolutional neural networks. ACM Comput. Surv., 54(10s), sep 2022. ISSN 0360-0300. doi: 10.1145/3510413. URL https://doi.org/10.1145/3510413.
- Misclassification risk and uncertainty quantification in deep classifiers. In IEEE Winter Conference on Applications of Computer Vision, WACV 2021, Waikoloa, HI, USA, January 3-8, 2021, pages 2483–2491. IEEE, 2021. doi: 10.1109/WACV48630.2021.00253. URL https://doi.org/10.1109/WACV48630.2021.00253.
- Jun Shao. Mathematical Statistics – Second Edition. Springer Science+Business Media, LLC., New York, 2003.
- An Analysis Of Convolutional Neural Networks For Image Classification. Procedia Computer Science, 132:377–384, January 2018. ISSN 1877-0509. doi: 10.1016/j.procs.2018.05.198. URL https://www.sciencedirect.com/science/article/pii/S1877050918309335.
- Efficient statistical validation of machine learning systems for autonomous driving. In Frank Liu, editor, Proceedings of the 35th International Conference on Computer-Aided Design, ICCAD 2016, Austin, TX, USA, November 7-10, 2016, page 36. ACM, 2016. doi: 10.1145/2966986.2980077. URL https://doi.org/10.1145/2966986.2980077.
- Siemens Mobility GmbH. World premiere: DB and Siemens present the first automatic train, October 2021. URL https://press.siemens.com/global/en/pressrelease/world-premiere-db-and-siemens-present-first-self-driving-train. Press release.
- Concolic testing for deep neural networks. In Marianne Huchard, Christian Kästner, and Gordon Fraser, editors, Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018, pages 109–119. ACM, 2018. doi: 10.1145/3238147.3238172. URL https://doi.org/10.1145/3238147.3238172.
- Explaining image classifiers using statistical fault localization. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII, volume 12373 of Lecture Notes in Computer Science, pages 391–406. Springer, 2020. doi: 10.1007/978-3-030-58604-1_24. URL https://doi.org/10.1007/978-3-030-58604-1_24.
- Towards a Specified Operational Design Domain for a Safe Remote Driving of Trains. In ESREL 2021, 31st European Safety And Reliability Conference, page 8p, Angers, France, September 2021. URL https://hal.archives-ouvertes.fr/hal-03328878. ESREL 2021, 31st European Safety And Reliability Conference, Angers, FRANCE, 19-/09/2021 - 23/09/2021. Trentesaux et al. [2018] Damien Trentesaux, Rudy Dahyot, Abel Ouedraogo, Diego Arenas, Sébastien Lefebvre, Walter Schön, Benjamin Lussier, and Hugues Chéritel. The Autonomous Train. In 2018 13th Annual Conference on System of Systems Engineering (SoSE), pages 514–520, June 2018. doi: 10.1109/SYSOSE.2018.8428771. Underwriters Laboratories Inc. [2020] Underwriters Laboratories Inc. ANSI/UL 4600-2020 Standard for Evaluation of Autonomous Products – First Edition. Underwriters Laboratories Inc., 333 Pfingsten Road, Northbrook, Illinois 60062-2096, 847.272.8800, April 2020. UNISIG [2006a] UNISIG, editor. ERTMS/ETCS – Class 1 System Requirements Specification, volume Subset-026. February 2006a. Issue 2.3.0. UNISIG [2006b] UNISIG. Basic System Description, chapter 2. Volume Subset-026-2 of UNISIG [63], February 2006b. Issue 2.3.0. UNISIG [2012] UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. Damien Trentesaux, Rudy Dahyot, Abel Ouedraogo, Diego Arenas, Sébastien Lefebvre, Walter Schön, Benjamin Lussier, and Hugues Chéritel. The Autonomous Train. In 2018 13th Annual Conference on System of Systems Engineering (SoSE), pages 514–520, June 2018. doi: 10.1109/SYSOSE.2018.8428771. Underwriters Laboratories Inc. [2020] Underwriters Laboratories Inc. ANSI/UL 4600-2020 Standard for Evaluation of Autonomous Products – First Edition. Underwriters Laboratories Inc., 333 Pfingsten Road, Northbrook, Illinois 60062-2096, 847.272.8800, April 2020. UNISIG [2006a] UNISIG, editor. ERTMS/ETCS – Class 1 System Requirements Specification, volume Subset-026. February 2006a. Issue 2.3.0. UNISIG [2006b] UNISIG. Basic System Description, chapter 2. Volume Subset-026-2 of UNISIG [63], February 2006b. Issue 2.3.0. UNISIG [2012] UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. Underwriters Laboratories Inc. ANSI/UL 4600-2020 Standard for Evaluation of Autonomous Products – First Edition. Underwriters Laboratories Inc., 333 Pfingsten Road, Northbrook, Illinois 60062-2096, 847.272.8800, April 2020. UNISIG [2006a] UNISIG, editor. ERTMS/ETCS – Class 1 System Requirements Specification, volume Subset-026. February 2006a. Issue 2.3.0. UNISIG [2006b] UNISIG. Basic System Description, chapter 2. Volume Subset-026-2 of UNISIG [63], February 2006b. Issue 2.3.0. UNISIG [2012] UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. UNISIG, editor. ERTMS/ETCS – Class 1 System Requirements Specification, volume Subset-026. February 2006a. Issue 2.3.0. UNISIG [2006b] UNISIG. Basic System Description, chapter 2. Volume Subset-026-2 of UNISIG [63], February 2006b. Issue 2.3.0. UNISIG [2012] UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. UNISIG. Basic System Description, chapter 2. Volume Subset-026-2 of UNISIG [63], February 2006b. Issue 2.3.0. UNISIG [2012] UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272.
- The Autonomous Train. In 2018 13th Annual Conference on System of Systems Engineering (SoSE), pages 514–520, June 2018. doi: 10.1109/SYSOSE.2018.8428771. Underwriters Laboratories Inc. [2020] Underwriters Laboratories Inc. ANSI/UL 4600-2020 Standard for Evaluation of Autonomous Products – First Edition. Underwriters Laboratories Inc., 333 Pfingsten Road, Northbrook, Illinois 60062-2096, 847.272.8800, April 2020. UNISIG [2006a] UNISIG, editor. ERTMS/ETCS – Class 1 System Requirements Specification, volume Subset-026. February 2006a. Issue 2.3.0. UNISIG [2006b] UNISIG. Basic System Description, chapter 2. Volume Subset-026-2 of UNISIG [63], February 2006b. Issue 2.3.0. UNISIG [2012] UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. Underwriters Laboratories Inc. ANSI/UL 4600-2020 Standard for Evaluation of Autonomous Products – First Edition. Underwriters Laboratories Inc., 333 Pfingsten Road, Northbrook, Illinois 60062-2096, 847.272.8800, April 2020. UNISIG [2006a] UNISIG, editor. ERTMS/ETCS – Class 1 System Requirements Specification, volume Subset-026. February 2006a. Issue 2.3.0. UNISIG [2006b] UNISIG. Basic System Description, chapter 2. Volume Subset-026-2 of UNISIG [63], February 2006b. Issue 2.3.0. UNISIG [2012] UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. UNISIG, editor. ERTMS/ETCS – Class 1 System Requirements Specification, volume Subset-026. February 2006a. Issue 2.3.0. UNISIG [2006b] UNISIG. Basic System Description, chapter 2. Volume Subset-026-2 of UNISIG [63], February 2006b. Issue 2.3.0. UNISIG [2012] UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. UNISIG. Basic System Description, chapter 2. Volume Subset-026-2 of UNISIG [63], February 2006b. Issue 2.3.0. UNISIG [2012] UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272.
- Underwriters Laboratories Inc. ANSI/UL 4600-2020 Standard for Evaluation of Autonomous Products – First Edition. Underwriters Laboratories Inc., 333 Pfingsten Road, Northbrook, Illinois 60062-2096, 847.272.8800, April 2020. UNISIG [2006a] UNISIG, editor. ERTMS/ETCS – Class 1 System Requirements Specification, volume Subset-026. February 2006a. Issue 2.3.0. UNISIG [2006b] UNISIG. Basic System Description, chapter 2. Volume Subset-026-2 of UNISIG [63], February 2006b. Issue 2.3.0. UNISIG [2012] UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. UNISIG, editor. ERTMS/ETCS – Class 1 System Requirements Specification, volume Subset-026. February 2006a. Issue 2.3.0. UNISIG [2006b] UNISIG. Basic System Description, chapter 2. Volume Subset-026-2 of UNISIG [63], February 2006b. Issue 2.3.0. UNISIG [2012] UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. UNISIG. Basic System Description, chapter 2. Volume Subset-026-2 of UNISIG [63], February 2006b. Issue 2.3.0. UNISIG [2012] UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272.
- UNISIG, editor. ERTMS/ETCS – Class 1 System Requirements Specification, volume Subset-026. February 2006a. Issue 2.3.0. UNISIG [2006b] UNISIG. Basic System Description, chapter 2. Volume Subset-026-2 of UNISIG [63], February 2006b. Issue 2.3.0. UNISIG [2012] UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. UNISIG. Basic System Description, chapter 2. Volume Subset-026-2 of UNISIG [63], February 2006b. Issue 2.3.0. UNISIG [2012] UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272.
- UNISIG. Basic System Description, chapter 2. Volume Subset-026-2 of UNISIG [63], February 2006b. Issue 2.3.0. UNISIG [2012] UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272.
- UNISIG. ERTMS/ETCS System Requirements Specification, Chapter 3, Principles, chapter 3. Volume Subset-026-3 of UNISIG [63], February 2012. Issue 3.3.0. Withers and Stoehr [2020] Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. Jared Withers and Nate Stoehr. Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272.
- Automated Train Operations (ATO) Safety and Sensor Development. Technical Report RR 20-21, U.S. Department of Transportation – Federal Railroad Administration, November 2020. URL https://railroads.dot.gov/elibrary/automated-train-operations-ato-safety-and-sensor-development. Zhang et al. [2012] Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272. Zhen Zhang, Yifei Wang, Jason Brand, and Naim Dahnoun. Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272.
- Real-time obstacle detection based on stereo vision for automotive applications. In 2012 5th European DSP Education and Research Conference (EDERC), pages 281–285, 2012. doi: 10.1109/EDERC.2012.6532272.