Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

BEACON: A Bayesian Evolutionary Approach for Counterexample Generation of Control Systems (2403.05925v2)

Published 9 Mar 2024 in eess.SY and cs.SY

Abstract: The rigorous safety verification of control systems in critical applications is essential, given their increasing complexity and integration into everyday life. Simulation-based falsification approaches play a pivotal role in the safety verification of control systems, particularly within critical applications. These methods systematically explore the operational space of systems to identify configurations that result in violations of safety specifications. However, the effectiveness of traditional simulation-based falsification is frequently limited by the high dimensionality of the search space and the substantial computational resources required for exhaustive exploration. This paper presents BEACON, a novel framework that enhances the falsification process through a combination of Bayesian optimization and covariance matrix adaptation evolutionary strategy. By exploiting quantitative metrics to evaluate how closely a system adheres to safety specifications, BEACON advances the state-of-the-art in testing methodologies. It employs a model-based test point selection approach, designed to facilitate exploration across dynamically evolving search zones to efficiently uncover safety violations. Our findings demonstrate that BEACON not only locates a higher percentage of counterexamples compared to standalone BO but also achieves this with significantly fewer simulations than required by CMA-ES, highlighting its potential to optimize the verification process of control systems. This framework offers a promising direction for achieving thorough and resource-efficient safety evaluations, ensuring the reliability of control systems in critical applications.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. E. M. Clarke, “Model checking,” in Foundations of Software Technology and Theoretical Computer Science: 17th Conference Kharagpur, India, December 18–20, 1997 Proceedings 17.   Springer, 1997, pp. 54–56.
  2. E. Plaku, L. E. Kavraki, and M. Y. Vardi, “Falsification of LTL safety properties in hybrid systems,” International Journal on Software Tools for Technology Transfer, vol. 15, no. 4, pp. 305–320, 2013.
  3. J. Kapinski, J. Deshmukh, X. Jin, H. Ito, and K. Butts, “Simulation-guided approaches for verification of automotive powertrain control systems,” in 2015 American Control Conference (ACC).   IEEE, 2015, pp. 4086–4095.
  4. A. Baheri, “Exploring the role of simulator fidelity in the safety validation of learning-enabled autonomous systems,” AI Magazine, vol. 44, no. 4, pp. 453–459, 2023.
  5. A. Baheri, “Safety validation of learning-based autonomous systems: A multi-fidelity approach,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 13, pp. 15 432–15 432, 2023.
  6. G. E. Fainekos, S. Sankaranarayanan, K. Ueda, and H. Yazarel, “Verification of automotive control applications using S-TaLiRo,” in 2012 American Control Conference (ACC).   IEEE, 2012, pp. 3567–3572.
  7. Y. Annpureddy, C. Liu, G. Fainekos, and S. Sankaranarayanan, “S-TaLiRo: A tool for temporal logic falsification for hybrid systems,” in International Conference on Tools and Algorithms for the Construction and Analysis of Systems.   Springer, 2011, pp. 254–257.
  8. A. Donzé, “Breach, a toolbox for verification and parameter synthesis of hybrid systems,” in Computer Aided Verification: 22nd International Conference, CAV 2010, Edinburgh, UK, July 15-19, 2010. Proceedings 22.   Springer, 2010, pp. 167–170.
  9. P. S. Duggirala, S. Mitra, M. Viswanathan, and M. Potok, “C2E2: A verification tool for stateflow models,” in Tools and Algorithms for the Construction and Analysis of Systems: 21st International Conference, TACAS 2015, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2015, London, UK, April 11-18, 2015, Proceedings 21.   Springer, 2015, pp. 68–82.
  10. B. Qi, C. Fan, M. Jiang, and S. Mitra, “DryVR 2.0: a tool for verification and controller synthesis of black-box cyber-physical systems,” in Proceedings of the 21st International Conference on Hybrid Systems: Computation and Control (part of CPS Week), 2018, pp. 269–270.
  11. Z. Zhang, G. Ernst, S. Sedwards, P. Arcaini, and I. Hasuo, “Two-layered falsification of hybrid systems guided by monte carlo tree search,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 37, no. 11, pp. 2894–2905, 2018.
  12. Z. Ramezani, K. Claessen, N. Smallbone, M. Fabian, and K. Åkesson, “Testing cyber–physical systems using a line-search falsification method,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 41, no. 8, pp. 2393–2406, 2021.
  13. Z. Zhang, D. Lyu, P. Arcaini, L. Ma, I. Hasuo, and J. Zhao, “Effective hybrid system falsification using Monte Carlo tree search guided by QB-robustness,” in International Conference on Computer Aided Verification.   Springer, 2021, pp. 595–618.
  14. M. Hekmatnejad, B. Hoxha, and G. Fainekos, “Search-based test-case generation by monitoring responsibility safety rules,” in 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC).   IEEE, 2020, pp. 1–8.
  15. Z. Zhang, P. Arcaini, and I. Hasuo, “Hybrid system falsification under (in)equality constraints via search space transformation,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 39, no. 11, pp. 3674–3685, 2020.
  16. J. Deshmukh, X. Jin, J. Kapinski, and O. Maler, “Stochastic local search for falsification of hybrid systems,” in International Symposium on Automated Technology for Verification and Analysis.   Springer, 2015, pp. 500–517.
  17. G. Ernst, S. Sedwards, Z. Zhang, and I. Hasuo, “Falsification of hybrid systems using adaptive probabilistic search,” ACM Transactions on Modeling and Computer Simulation (TOMACS), vol. 31, no. 3, pp. 1–22, 2021.
  18. Z. Ramezani, J. L. Eddeland, K. Claessen, M. Fabian, and K. Åkesson, “Multiple objective functions for falsification of cyber-physical systems,” IFAC-PapersOnLine, vol. 53, no. 4, pp. 417–422, 2020.
  19. L. Mathesen, G. Pedrielli, and G. Fainekos, “Efficient optimization-based falsification of cyber-physical systems with multiple conjunctive requirements,” in 2021 IEEE 17th International Conference on Automation Science and Engineering (CASE).   IEEE, 2021, pp. 732–737.
  20. J. Deshmukh, M. Horvat, X. Jin, R. Majumdar, and V. S. Prabhu, “Testing cyber-physical systems through Bayesian optimization,” ACM Transactions on Embedded Computing Systems (TECS), vol. 16, no. 5s, pp. 1–18, 2017.
  21. H. Abbas, M. O’Kelly, A. Rodionova, and R. Mangharam, “Safe at any speed: A simulation-based test harness for autonomous vehicles,” in Cyber Physical Systems. Design, Modeling, and Evaluation: 7th International Workshop, CyPhy 2017, Seoul, South Korea, October 15-20, 2017, Revised Selected Papers 7.   Springer, 2019, pp. 94–106.
  22. A. Aerts, B. T. Minh, M. R. Mousavi, and M. A. Reniers, “Temporal logic falsification of cyber-physical systems: An input-signal-space optimization approach,” in 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).   IEEE, 2018, pp. 214–223.
  23. Y. S. R. Annapureddy and G. E. Fainekos, “Ant colonies for temporal logic falsification of hybrid systems,” in IECON 2010-36th Annual Conference on IEEE Industrial Electronics Society.   IEEE, 2010, pp. 91–96.
  24. T. Akazaki, S. Liu, Y. Yamagata, Y. Duan, and J. Hao, “Falsification of cyber-physical systems using deep reinforcement learning,” in Formal Methods: 22nd International Symposium, FM 2018, Held as Part of the Federated Logic Conference, FloC 2018, Oxford, UK, July 15-17, 2018, Proceedings 22.   Springer, 2018, pp. 456–465.
  25. Z. Zhang, I. Hasuo, and P. Arcaini, “Multi-armed bandits for Boolean connectives in hybrid system falsification,” in Computer Aided Verification: 31st International Conference, CAV 2019, New York City, NY, USA, July 15-18, 2019, Proceedings, Part I 31.   Springer, 2019, pp. 401–420.
  26. X. Qin, N. Aréchiga, A. Best, and J. Deshmukh, “Automatic testing and falsification with dynamically constrained reinforcement learning,” arXiv preprint arXiv:1910.13645, 2019.
  27. J. J. Beard and A. Baheri, “Safety verification of autonomous systems: A multi-fidelity reinforcement learning approach,” arXiv preprint arXiv:2203.03451, 2022.
  28. G. Frehse, C. Le Guernic, A. Donzé, S. Cotton, R. Ray, O. Lebeltel, R. Ripado, A. Girard, T. Dang, and O. Maler, “Spaceex: Scalable verification of hybrid systems,” in Computer Aided Verification: 23rd International Conference, CAV 2011, Snowbird, UT, USA, July 14-20, 2011. Proceedings 23.   Springer, 2011, pp. 379–395.
  29. T. Dreossi, A. Donzé, and S. A. Seshia, “Compositional falsification of cyber-physical systems with machine learning components,” Journal of Automated Reasoning, vol. 63, pp. 1031–1053, 2019.
  30. M. Koren, S. Alsaif, R. Lee, and M. J. Kochenderfer, “Adaptive stress testing for autonomous vehicles,” in 2018 IEEE Intelligent Vehicles Symposium (IV).   IEEE, 2018, pp. 1–7.
  31. E. Bartocci, R. Bloem, B. Maderbacher, N. Manjunath, and D. Ničković, “Adaptive testing for specification coverage in CPS models,” IFAC-PapersOnLine, vol. 54, no. 5, pp. 229–234, 2021.
  32. O. Maler and D. Ničković, “Monitoring properties of analog and mixed-signal circuits,” International Journal on Software Tools for Technology Transfer, vol. 15, pp. 247–268, 2013.
  33. A. Donzé and O. Maler, “Robust satisfaction of temporal logic over real-valued signals,” in International Conference on Formal Modeling and Analysis of Timed Systems.   Springer, 2010, pp. 92–106.
  34. J. Wu, X.-Y. Chen, H. Zhang, L.-D. Xiong, H. Lei, and S.-H. Deng, “Hyperparameter optimization for machine learning models based on Bayesian optimization,” Journal of Electronic Science and Technology, vol. 17, no. 1, pp. 26–40, 2019.
  35. F. Berkenkamp, A. Krause, and A. P. Schoellig, “Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics,” Machine Learning, vol. 112, no. 10, pp. 3713–3747, 2023.
  36. A. Baheri, S. Bin-Karim, A. Bafandeh, and C. Vermillion, “Real-time control using Bayesian optimization: A case study in airborne wind energy systems,” Control Engineering Practice, vol. 69, pp. 131–140, 2017.
  37. A. Baheri, P. Ramaprabhu, and C. Vermillion, “Iterative in-situ 3D layout optimization of a reconfigurable ocean current turbine array using Bayesian optimization,” in ASME 2017 Dynamic Systems and Control Conference.   American Society of Mechanical Engineers, 2017, pp. V003T40A002–V003T40A002.
  38. A. Baheri and C. Vermillion, “Waypoint optimization using bayesian optimization: A case study in airborne wind energy systems,” in 2020 American Control Conference (ACC).   IEEE, 2020, pp. 5102–5017.
  39. S. Ghosh, F. Berkenkamp, G. Ranade, S. Qadeer, and A. Kapoor, “Verifying controllers against adversarial examples with Bayesian optimization,” in 2018 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2018, pp. 7306–7313.
  40. Z. Shahrooei, M. J. Kochenderfer, and A. Baheri, “Falsification of learning-based controllers through multi-fidelity Bayesian optimization,” in 2023 European Control Conference (ECC).   IEEE, 2023, pp. 1–6.
  41. N. Hansen, “The CMA evolution strategy: A tutorial,” arXiv preprint arXiv:1604.00772, 2016.
  42. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971, 2015.
  43. MathWorks, “Design NARMA-L2 neural controller in simulink,” 2020, online: accessed 22 May 2023. [Online]. Available: https://au.mathworks.com/help/deeplearning/ug/design-narma-l2-neural-controller-in-simulink.html
  44. P. Heidlauf, A. Collins, M. Bolender, and S. Bak, “Verification challenges in f-16 ground collision avoidance and other automated maneuvers,” in ARCH18. 5th International Workshop on Applied Verification of Continuous and Hybrid Systems, ser. EPiC Series in Computing, G. Frehse, Ed., vol. 54.   EasyChair, 2018, pp. 208–217.
  45. X. Jin, J. V. Deshmukh, J. Kapinski, K. Ueda, and K. Butts, “Powertrain control verification benchmark,” in Proceedings of the 17th international conference on Hybrid systems: computation and control, 2014, pp. 253–262.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now