Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Discovering Closed-Loop Failures of Vision-Based Controllers via Reachability Analysis (2211.02736v4)

Published 4 Nov 2022 in cs.RO, cs.AI, cs.CV, cs.LG, cs.SY, and eess.SY

Abstract: Machine learning driven image-based controllers allow robotic systems to take intelligent actions based on the visual feedback from their environment. Understanding when these controllers might lead to system safety violations is important for their integration in safety-critical applications and engineering corrective safety measures for the system. Existing methods leverage simulation-based testing (or falsification) to find the failures of vision-based controllers, i.e., the visual inputs that lead to closed-loop safety violations. However, these techniques do not scale well to the scenarios involving high-dimensional and complex visual inputs, such as RGB images. In this work, we cast the problem of finding closed-loop vision failures as a Hamilton-Jacobi (HJ) reachability problem. Our approach blends simulation-based analysis with HJ reachability methods to compute an approximation of the backward reachable tube (BRT) of the system, i.e., the set of unsafe states for the system under vision-based controllers. Utilizing the BRT, we can tractably and systematically find the system states and corresponding visual inputs that lead to closed-loop failures. These visual inputs can be subsequently analyzed to find the input characteristics that might have caused the failure. Besides its scalability to high-dimensional visual inputs, an explicit computation of BRT allows the proposed approach to capture non-trivial system failures that are difficult to expose via random simulations. We demonstrate our framework on two case studies involving an RGB image-based neural network controller for (a) autonomous indoor navigation, and (b) autonomous aircraft taxiing.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (27)
  1. A. Loquercio, E. Kaufmann, R. Ranftl, A. Dosovitskiy, V. Koltun, and D. Scaramuzza, “Deep drone racing: From simulation to reality with domain randomization,” T-RO, vol. 36, no. 1, pp. 1–14, 2019.
  2. A. Wang, T. Kurutach, K. Liu, P. Abbeel, and A. Tamar, “Learning robotic manipulation through visual planning and acting,” arXiv preprint arXiv:1905.04411, 2019.
  3. S. Grigorescu, B. Trasnea, T. Cocias, and G. Macesanu, “A survey of deep learning techniques for autonomous driving,” JFR, vol. 37, no. 3, pp. 362–386, 2020.
  4. S. M. Katz, A. L. Corso, C. A. Strong, and M. J. Kochenderfer, “Verification of image-based neural network controllers using generative models,” in DASC.   IEEE, 2021, pp. 1–10.
  5. V. Tjeng, K. Xiao, and R. Tedrake, “Evaluating robustness of neural networks with mixed integer programming,” arXiv preprint arXiv:1711.07356, 2017.
  6. G. Katz, C. Barrett, D. L. Dill, K. Julian, and M. J. Kochenderfer, “Reluplex: An efficient smt solver for verifying deep neural networks,” in ICCAV.   Springer, 2017, pp. 97–117.
  7. R. A. Brown, E. Schmerling, N. Azizan, and M. Pavone, “A unified view of sdp-based neural network verification through completely positive programming,” in AISTATS, vol. 151.   PMLR, 2022, pp. 9334–9355.
  8. X. Huang, M. Kwiatkowska, S. Wang, and M. Wu, “Safety verification of deep neural networks,” in ICCAV.   Springer, 2017, pp. 3–29.
  9. K. Pei, Y. Cao, J. Yang, and S. Jana, “Deepxplore: Automated whitebox testing of deep learning systems,” in SOSP, 2017, pp. 1–18.
  10. C. Huang, J. Fan, W. Li, X. Chen, and Q. Zhu, “Reachnn: Reachability analysis of neural-network controlled systems,” ACM TECS, vol. 18, no. 5s, pp. 1–22, 2019.
  11. W. Xiang, H.-D. Tran, and T. T. Johnson, “Output reachable set estimation and verification for multilayer neural networks,” IEEE TNNLS, vol. 29, no. 11, pp. 5777–5783, 2018.
  12. K. D. Julian and M. J. Kochenderfer, “Guaranteeing safety for neural network-based aircraft collision avoidance systems,” in DASC, 2019.
  13. C. Hsieh, K. Joshi, S. Misailovic, and S. Mitra, “Verifying controllers with convolutional neural network-based perception: a case for intelligible, safe, and precise abstractions,” arXiv preprint arXiv:2111.05534, 2021.
  14. U. Santa Cruz and Y. Shoukry, “Nnlander-verif: A neural network formal verification framework for vision-based autonomous aircraft landing,” in NASA Formal Methods.   Springer, 2022, pp. 213–230.
  15. “Laminar Research: X-Plane 11 (2019),” https://www.x-plane.com/.
  16. “Matterport,” https://matterport.com/.
  17. I. Armeni, A. Sax, A. R. Zamir, and S. Savarese, “Joint 2D-3D-Semantic Data for Indoor Scene Understanding,” ArXiv e-prints, Feb. 2017.
  18. F. Indaheng, E. Kim, K. Viswanadha, J. Shenoy, J. Kim, D. J. Fremont, and S. A. Seshia, “A scenario-based platform for testing autonomous vehicle behavior prediction models in simulation,” arXiv preprint arXiv:2110.14870, 2021.
  19. D. J. Fremont, E. Kim, Y. V. Pant, S. A. Seshia, A. Acharya, X. Bruso, P. Wells, S. Lemke, Q. Lu, and S. Mehta, “Formal scenario-based testing of autonomous vehicles: From simulation to the real world,” in ITSC.   IEEE, 2020, pp. 1–8.
  20. T. Dreossi, S. Ghosh, A. Sangiovanni-Vincentelli, and S. A. Seshia, “Systematic testing of convolutional neural networks for autonomous driving,” arXiv preprint arXiv:1708.03309, 2017.
  21. S. Ghosh, Y. V. Pant, H. Ravanbakhsh, and S. A. Seshia, “Counterexample-guided synthesis of perception models and control,” in ACC.   IEEE, 2021, pp. 3447–3454.
  22. L. Yang and N. Ozay, “Synthesis-guided adversarial scenario generation for gray-box feedback control systems with sensing imperfections,” ACM TECS, vol. 20, no. 5s, pp. 1–25, 2021.
  23. I. Mitchell, A. Bayen, and C. J. Tomlin, “A time-dependent hamilton-jacobi formulation of reachable sets for continuous dynamic games,” TAC, vol. 50, no. 7, pp. 947–957, 2005.
  24. I. Mitchell and C. J. Tomlin, “Level set methods for computation in hybrid systems,” in HSCC.   Springer, 2002, pp. 310–323.
  25. I. M. Mitchell et al., “A toolbox of level set methods,” UBC Department of Computer Science Technical Report TR-2007-11, p. 31, 2007.
  26. S. Bansal, V. Tolani, S. Gupta, J. Malik, and C. Tomlin, “Combining optimal control and learning for visual navigation in novel environments,” in CoRL.   PMLR, 2020, pp. 420–429.
  27. S. Bansal and C. J. Tomlin, “Deepreach: A deep learning approach to high-dimensional reachability,” in ICRA.   IEEE, 2021, pp. 1817–1824.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Kaustav Chakraborty (23 papers)
  2. Somil Bansal (49 papers)
Citations (11)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now