Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Formal Verification of Robustness and Resilience of Learning-Enabled State Estimation Systems (2010.08311v4)

Published 16 Oct 2020 in cs.RO and cs.CR

Abstract: This paper presents a formal verification guided approach for a principled design and implementation of robust and resilient learning-enabled systems. We focus on learning-enabled state estimation systems (LE-SESs), which have been widely used in robotics applications to determine the current state (e.g., location, speed, direction, etc.) of a complex system. The LE-SESs are networked systems, composed of a set of connected components including: Bayes filters for state estimation, and neural networks for processing sensory input. We study LE-SESs from the perspective of formal verification, which determines the satisfiabilty of a system model against the specified properties. Over LE-SESs, we investigate two key properties -- robustness and resilience -- and provide their formal definitions. To enable formal verification, we reduce the LE-SESs to a novel class of labelled transition systems, named {PO}2-LTS in the paper, and formally express the properties as constrained optimisation objectives. We prove that the verification problems are NP-complete. Based on {PO}2-LTS and the optimisation objectives, practical verification algorithms are developed to check the satisfiability of the properties on the LE-SESs. As a major case study, we interrogate a real-world dynamic tracking system which uses a single Kalman Filter (KF) -- a special case of Bayes filter -- to localise and track a ground vehicle. Its perception system, based on convolutional neural networks, processes a high-resolution Wide Area Motion Imagery (WAMI) data stream. Experimental results show that our algorithms can not only verify the properties of the WAMI tracking system but also provide representative examples, the latter of which inspired us to take an enhanced LE-SESs design where runtime monitors or joint-KFs are required. Experimental results confirm the improvement in the robustness of the enhanced design.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (76)
  1. Iso/iec/ieee international standard - systems and software engineering–vocabulary. ISO/IEC/IEEE 24765:2017(E), pages 1–541, 2017.
  2. SURF: speeded up robust features. In Computer Vision - ECCV 2006, 9th European Conference on Computer Vision, Graz, Austria, May 7-13, 2006, Proceedings, Part I, volume 3951 of Lecture Notes in Computer Science, pages 404–417. Springer, 2006.
  3. What, indeed, is an achievable provable guarantee for learning-enabled safety-critical systems. In Bernhard Steffen, editor, Bridging the Gap Between AI and Reality, pages 55–76, Cham, 2024. Springer Nature Switzerland.
  4. Formal specification for learning-enabled autonomous systems. In Omri Isac, Radoslav Ivanov, Guy Katz, Nina Narodytska, and Laura Nenzi, editors, Software Verification and Formal Methods for ML-Enabled Autonomous Systems, pages 131–143, Cham, 2022. Springer International Publishing.
  5. Bridging formal methods and machine learning with model checking and global optimisation. Journal of Logical and Algebraic Methods in Programming, 137:100941, 2024.
  6. A conceptual review of family resilience factors. Journal of Family Nursing, 14(1):33–55, 2008.
  7. End to end learning for self-driving cars. CoRR, abs/1604.07316, 2016.
  8. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy, S&P, pages 39–57. IEEE Computer Society, 2017.
  9. Model checking. The MIT Press, 2018.
  10. Camera models for the wright patterson air force base (wpafb) 2009 wide-area motion imagery (wami) data set. IEEE Aerospace and Electronic Systems Magazine, 30(6):4–15, 2015.
  11. Detecting black-box adversarial examples through nonlinear dimensionality reduction. In 27th European Symposium on Artificial Neural Networks, ESANN, 2019.
  12. Community vitality: The role of community-level resilience adaptation and innovation in sustainable development. Sustainability, 2(1):215–231, 2010.
  13. The complexity of constrained min-max optimization. In STOC ’21: 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 1466–1478. ACM, 2021.
  14. Compositional falsification of cyber-physical systems with machine learning components. Journal of Automated Reasoning, 63(4):1031–1053, 2019.
  15. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 24(6):381–395, June 1981.
  16. Regime shifts, resilience, and biodiversity in ecosystem management. Annual Review of Ecology, Evolution, and Systematics, 35(1):557–581, 2004.
  17. A metric and frameworks for resilience analysis of engineered and infrastructure systems. Reliability Engineering & System Safety, 121:90–103, 2014.
  18. AI2: safety and robustness certification of neural networks with abstract interpretation. In 2018 IEEE Symposium on Security and Privacy, S&P, pages 3–18. IEEE Computer Society, 2018.
  19. Biochemical clocks and molecular noise: Theoretical study of robustness factors. The Journal of chemical physics, 116(24):10997–11010, 2002.
  20. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR, 2015.
  21. Bayesian state estimation for tracking and guidance using the bootstrap filter. Journal of Guidance, Control, and Dynamics, 18(6):1434–1443, 1995.
  22. Joseph Y. Halpern. Reasoning about Uncertainty. The MIT Press, 2017.
  23. Adaptive fuzzy neural network control for a constrained robot using impedance learning. IEEE Transactions on Neural Networks and Learning Systems, 29(4):1174–1186, 2018.
  24. Coverage-guided testing for recurrent neural networks. IEEE Transactions on Reliability, 71(3):1191–1206, 2022.
  25. Formal verification of robustness and resilience of learning-enabled state estimation systems for robotics. arXiv preprint arXiv:2010.08311, 2020.
  26. Practical verification of neural network enabled state estimation system for robotics. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7336–7343, 2020.
  27. A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability. Computer Science Review, 37:100270, 2020.
  28. Safety verification of deep neural networks. In Computer Aided Verification - 29th International Conference, CAV, volume 10426, pages 3–29. Springer, 2017.
  29. Verisig: Verifying safety properties of hybrid systems with neural network controllers. In Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control, page 169–178. Association for Computing Machinery, 2019.
  30. Enhancing adversarial training with second-order statistics of weights. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15273–15283, 2022.
  31. Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 82(1):35–45, 1960.
  32. Reluplex: An efficient SMT solver for verifying deep neural networks. In International Conference on Computer Aided Verification, pages 97–117. Springer, 2017.
  33. Dexter C Kozen. Depth-first and breadth-first search. In The Design and Analysis of Algorithms, pages 19–24. Springer, 1992.
  34. Clusternet: Detecting small objects in large scenes by exploiting spatio-temporal information. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, pages 4003–4012. Computer Vision Foundation / IEEE Computer Society, 2018.
  35. Analyzing deep neural networks with symbolic propagation: Towards higher precision and faster verification. In Static Analysis: 26th International Symposium, SAS, volume 11822, pages 296–319. Springer, 2019.
  36. An approach to reachability analysis for feed-forward ReLU neural networks. arXiv preprint arXiv:1706.07351, 2017.
  37. David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91–110, 2004.
  38. Deepgauge: multi-granularity testing criteria for deep learning systems. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE, pages 120–131. ACM, 2018.
  39. On detecting adversarial perturbations. In 5th International Conference on Learning Representations, ICLR. OpenReview.net, 2017.
  40. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2574–2582, 2016.
  41. Analysis of divide-and-conquer strategies for the 0–1 minimization knapsack problem. Journal of Combinatorial Optimization, 40(1):234–278, 2020.
  42. Resilience methods within the software development cycle. In Proceedings of the International Conference on Software Engineering Research and Practice (SERP), pages 62–65. The Steering Committee of The World Congress in Computer Science, 2017.
  43. System state estimation in the presence of false information injection. In IEEE Statistical Signal Processing Workshop, SSP, pages 385–388. IEEE, 2012.
  44. Hydiff: hybrid differential software analysis. In ICSE ’20: 42nd International Conference on Software Engineering, pages 1273–1285. ACM, 2020.
  45. Cale Palmer. A theory of risk and resilience factors in military families. Military Psychology, 20(3):205–217, 2008.
  46. Cooperative localization of marine vehicles using nonlinear state estimation. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4874–4879. IEEE, 2010.
  47. Distillation as a defense to adversarial perturbations against deep neural networks. In IEEE Symposium on Security and Privacy, S&P, pages 582–597. IEEE Computer Society, 2016.
  48. DeepXplore: Automated whitebox testing of deep learning systems. In Proceedings of the 26th Symposium on Operating Systems Principles, pages 1–18. ACM, 2017.
  49. Dwight Read. Some observations on resilience and robustness in human systems. Cybernetics and Systems: An International Journal, 36(8):773–802, 2005.
  50. Craig G Rieger. Resilient control systems practical metrics basis for defining mission impact. In 2014 7th International Symposium on Resilient Control Systems (ISRCS), pages 1–10. IEEE, 2014.
  51. Reachability analysis of deep neural networks with provable guarantees. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI, pages 2651–2659. ijcai.org, 2018.
  52. Global robustness evaluation of deep neural networks with provable guarantees for the hamming distance. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI, pages 5944–5952. ijcai.org, 2019.
  53. Orb: An efficient alternative to sift or surf. In International Conference on Computer Vision, pages 2564–2571, Nov 2011.
  54. John Ruscio. A probability-based measure of effect size: Robustness to base rates and other factors. Psychological methods, 13(1):19, 2008.
  55. Risk, vulnerability, robustness, and resilience from a decision-theoretic perspective. Journal of Risk Research, 15(3):313–330, 2012.
  56. Joseph Sifakis. Autonomous systems - an architectural characterization. In Models, Languages, and Tools for Concurrent and Distributed Programming - Essays Dedicated to Rocco De Nicola on the Occasion of His 65th Birthday, volume 11665, pages 388–410. Springer, 2019.
  57. The brief resilience scale: assessing the ability to bounce back. International journal of behavioral medicine, 15(3):194–200, 2008.
  58. A survey on moving object detection for wide area motion imagery. In IEEE Winter Conference on Applications of Computer Vision, WACV, pages 1–9, 2016.
  59. Finite-time adaptive neural resilient dsc for fractional-order nonlinear large-scale systems against sensor-actuator faults. Nonlinear Dynamics, 111:1–16, 2023.
  60. Switching-like event-triggered state estimation for reaction–diffusion neural networks against dos attacks. Neural Processing Letters, 55(7):8997–9018, 2023.
  61. DeepConcolic: testing and debugging deep neural networks. In 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pages 111–114. IEEE / ACM, 2019.
  62. Structural test coverage criteria for deep neural networks. ACM Transactions on Embedded Computing Systems (TECS), 18(5s):94:1–94:23, 2019.
  63. Concolic testing for deep neural networks. In Marianne Huchard, Christian Kästner, and Gordon Fraser, editors, Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE, pages 109–119. ACM, 2018.
  64. Reliability validation of learning enabled vehicle tracking. In 2020 IEEE International Conference on Robotics and Automation, ICRA, pages 9390–9396. IEEE, 2020.
  65. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, ICLR, 2014.
  66. Reasoning about safety of learning-enabled components in autonomous cyber-physical systems. In Proceedings of the 55th Annual Design Automation Conference, DAC, pages 30:1–30:6. ACM, 2018.
  67. Guidance for robustness/ruggedness tests in method validation. Journal of pharmaceutical and biomedical analysis, 24(5):723–753, 2001.
  68. Emmy E Werner. Risk, resilience, and recovery: Perspectives from the kauai longitudinal study. Development and psychopathology, 5(4):503–515, 1993.
  69. Feature-guided black-box safety testing of deep neural networks. In Tools and Algorithms for the Construction and Analysis of Systems - 24th International Conference, TACAS 2018, volume 10805, pages 408–426. Springer, 2018.
  70. A game-based approximate verification of deep neural networks with provable guarantees. Theoretical Computer Science, 807:298–329, 2020.
  71. Output reachable set estimation and verification for multilayer neural networks. IEEE Transactions on Neural Networks and Learning Systems, 29(11):5777–5783, 2018.
  72. Npc: Neuron path coverage via characterizing decision logic of deep neural networks. ACM Transactions on Software Engineering and Methodology (TOSEM), 31(3), 2022.
  73. Gray-box adversarial testing for control systems with machine learning components. In Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control, page 179–184. Association for Computing Machinery, 2019.
  74. On false data injection attacks against kalman filtering in power system dynamic state estimation. Security and Communication Networks, 9(9):833–849, 2016.
  75. Robust point-to-point iterative learning control for constrained systems: A minimum energy approach. International Journal of Robust and Nonlinear Control, 32(18):10139–10161, 2022.
  76. Detecting and tracking small moving objects in wide area motion imagery (wami) using convolutional neural networks (cnns). In 2019 22th International Conference on Information Fusion (FUSION), pages 1–8. IEEE, 2019.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now