Verisig: verifying safety properties of hybrid systems with neural network controllers (1811.01828v1)

Abstract: This paper presents Verisig, a hybrid system approach to verifying safety properties of closed-loop systems using neural networks as controllers. Although techniques exist for verifying input/output properties of the neural network itself, these methods cannot be used to verify properties of the closed-loop system (since they work with piecewise-linear constraints that do not capture non-linear plant dynamics). To overcome this challenge, we focus on sigmoid-based networks and exploit the fact that the sigmoid is the solution to a quadratic differential equation, which allows us to transform the neural network into an equivalent hybrid system. By composing the network's hybrid system with the plant's, we transform the problem into a hybrid system verification problem which can be solved using state-of-the-art reachability tools. We show that reachability is decidable for networks with one hidden layer and decidable for general networks if Schanuel's conjecture is true. We evaluate the applicability and scalability of Verisig in two case studies, one from reinforcement learning and one in which the neural network is used to approximate a model predictive controller.

• The paper introduces Verisig, which transforms neural network controller verification for hybrid systems into a hybrid system reachability problem solvable with standard tools.
• The method shows reachability decidability for one hidden layer NNs and verifies safety in mountain car and quadcopter examples, demonstrating scalability.
• This work enables verification of neural networks within dynamic systems, paving the way for increased trust and deployment of AI in safety-critical applications.

Verification of Safety Properties in Hybrid Systems with Neural Network Controllers: The Verisig Approach

The paper "Verisig: verifying safety properties of hybrid systems with neural network controllers" presents a comprehensive methodology for addressing the challenges of verifying safety properties in hybrid systems where the controller is implemented as a neural network (NN). Unlike traditional verification approaches that focus on the neural network alone, Verisig transforms the overall verification problem into a hybrid system problem, thereby enabling the use of existing reachability tools for hybrid systems.

Problem and Approach

The task of verifying the safety of closed-loop systems with NN controllers is non-trivial due to the non-linear dynamics of these systems, which are typically represented by piecewise-linear constraints in purely NN verification methods. In Verisig, the authors concentrate on networks utilizing sigmoid activation functions, capitalizing on the inherent solvability of the sigmoid's quadratic differential equations. This innovative choice allows the transformation of the neural network into an equivalent hybrid system representation. By composing this network's hybrid system with that of the plant model, they convert the system into a hybrid system verification problem, solvable using advanced reachability tools like dReach and Flow*.

Decidability and Case Studies

The decidability of the reachability problem is a pivotal aspect of the Verisig methodology. The authors demonstrate that reachability is decidable for networks with one hidden layer, and potentially for more complex networks should Schanuel's conjecture hold true. They also establish $\delta$-decidability by framing the problem within the dReach framework.

The applicability and scalability of this verification approach are tested through two diverse case studies: a mountain car reinforcement learning (RL) scenario and a model predictive control (MPC) approximation for quadcopter dynamics. For the mountain car problem, they validate the control policy's performance over a range of starting conditions, ensuring a guaranteed reward threshold. In the MPC case, a neural network approximates the controller and verifies that the quadcopter remains within a safe operational envelope as it follows a designated path, avoiding obstacles.

Numerical Results and Implications

The paper provides strong numerical results, demonstrating that Verisig, combined with Flow*, efficiently verifies the safety properties in systems composed of neural network controllers. Particularly noteworthy is the scalability observed in the verification process, where the hybrid system representation permits linear scaling with the network layers, as opposed to the exponential scaling often cited in MILP-based approaches. This alignment with the efficient learning capabilities of deeper networks underscores Verisig's practical relevance in real-world applications.

Future Directions

Verisig opens several avenues for further research and practical improvements. The potential extension to general neural networks through approximations and transformations into sigmoid-based systems holds the promise of broader applicability. Enhancements in computational efficiency could be achieved by developing reachability tools tailored specifically for the quadratic and monotone nature of sigmoid dynamics. There is also a need to precisely quantify the approximation errors associated with using Flow*, which could validate the observed empirical reliability of results.

In conclusion, Verisig offers a structured approach to the verification of safety properties in hybrid systems controlled by neural networks, laying the groundwork for both theoretical advancements and practical applications in AI safety and cyber-physical systems. This work demonstrates the practicality of verifying NNs embedded in dynamic systems, which could significantly impact the deployment of AI in safety-critical sectors.