Compositional Falsification of Cyber-Physical Systems with Machine Learning Components (1703.00978v3)

Abstract: Cyber-physical systems (CPS), such as automotive systems, are starting to include sophisticated ML components. Their correctness, therefore, depends on properties of the inner ML modules. While learning algorithms aim to generalize from examples, they are only as good as the examples provided, and recent efforts have shown that they can produce inconsistent output under small adversarial perturbations. This raises the question: can the output from learning components can lead to a failure of the entire CPS? In this work, we address this question by formulating it as a problem of falsifying signal temporal logic (STL) specifications for CPS with ML components. We propose a compositional falsification framework where a temporal logic falsifier and a machine learning analyzer cooperate with the aim of finding falsifying executions of the considered model. The efficacy of the proposed technique is shown on an automatic emergency braking system model with a perception component based on deep neural networks.

• The paper introduces a compositional falsification approach that splits the verification task into analyzing an ideal CPS model and pinpointing critical ML misclassifications.
• It employs an innovative ML analyzer and abstraction techniques to efficiently explore high-risk regions in the ML input space.
• The framework is validated through AEBS case studies, demonstrating its potential to improve safety in autonomous systems.

Compositional Falsification of Cyber-Physical Systems with Machine Learning Components

The paper under discussion presents a methodological framework for addressing the verification challenges introduced by the integration of ML components within cyber-physical systems (CPS). With the emergence of complex CPS such as autonomous vehicles, the inclusion of ML modules for perception and decision-making necessitates robust verification techniques to ensure system safety and correctness. This paper proposes a novel compositional falsification approach that utilizes both temporal logic-based verification and ML analysis to identify potential failures in CPS with embedded ML components.

Summary of Contributions

The central contribution of this work is a two-pronged framework enabling the effective falsification of signal temporal logic (STL) specifications in CPS models with ML components. This framework consists of the following key components:

• Compositional Falsification Technique: The framework introduces a compositional strategy to partition the verification problem into two distinct sub-problems: analyzing the CPS model with an "ideal" ML component and separately examining the ML component to identify misclassifications critical to the overall system's correctness. This decomposition leverages the strengths of conventional temporal logic falsification tools and a specialized ML analyzer.
• ML Analyzer: An innovative aspect of this research is the ML analyzer that abstracts the high-dimensional feature space of ML components, enabling more feasible analysis. This abstraction facilitates the systematic exploration of potentially problematic regions in the ML input space, focusing on "realistic and meaningful modifications" rather than an exhaustive search of all possible inputs.
• Abstraction-Based Approach: The framework employs optimistic and pessimistic abstractions of the CPS model, simulating scenarios with perfectly accurate and completely flawed ML components respectively. This dichotomy aids in identifying the region of uncertainty (ROU) where the correctness of the ML component directly affects the system behavior, thus bounding the input space exploration.

Strong Numerical Results

The applicability of this framework is demonstrated using two case studies involving an Automatic Emergency Braking System (AEBS). The test cases illustrate the framework's capacity to pinpoint inputs that lead to property violations due to ML component misclassifications. The framework efficiently discovered falsifying inputs even within large and complex input spaces, showcasing its practical utility and effectiveness in identifying high-risk scenarios.

Implications and Future Directions

The proposed approach significantly advances the domain of CPS verification by incorporating ML-specific analysis techniques. The implications of this work are vast, offering a pathway to more reliable integration of ML into safety-critical systems. From a theoretical standpoint, the research discusses abstract notions such as abstraction and compositionality, underscoring their significance in managing verification complexity.

Future research directions could include expanding the framework to handle continuous streams of sensor data in autonomous systems, improving abstraction techniques for ML components to minimize false negatives, and exploring the integration of this falsification framework with other verification methodologies such as formal specification mining or automated controller synthesis.

Moreover, beyond autonomous driving, the compositional falsification strategy holds potential for a broader range of applications where CPS are used in conjunction with ML components, including robotics and industrial automation. The paper paves the way for creating more dependable intelligent systems, addressing both contemporary verification challenges and setting the stage for future innovation in the field.