Lyapunov-stable neural-network control (2109.14152v1)

Abstract: Deep learning has had a far reaching impact in robotics. Specifically, deep reinforcement learning algorithms have been highly effective in synthesizing neural-network controllers for a wide range of tasks. However, despite this empirical success, these controllers still lack theoretical guarantees on their performance, such as Lyapunov stability (i.e., all trajectories of the closed-loop system are guaranteed to converge to a goal state under the control policy). This is in stark contrast to traditional model-based controller design, where principled approaches (like LQR) can synthesize stable controllers with provable guarantees. To address this gap, we propose a generic method to synthesize a Lyapunov-stable neural-network controller, together with a neural-network Lyapunov function to simultaneously certify its stability. Our approach formulates the Lyapunov condition verification as a mixed-integer linear program (MIP). Our MIP verifier either certifies the Lyapunov condition, or generates counter examples that can help improve the candidate controller and the Lyapunov function. We also present an optimization program to compute an inner approximation of the region of attraction for the closed-loop system. We apply our approach to robots including an inverted pendulum, a 2D and a 3D quadrotor, and showcase that our neural-network controller outperforms a baseline LQR controller. The code is open sourced at \url{https://github.com/StanfordASL/neural-network-lyapunov}.

• The paper presents a novel framework synthesizing neural-network controllers with Lyapunov stability to ensure system trajectories converge to desired goals.
• It employs a mixed-integer linear programming formulation to verify Lyapunov conditions and iteratively refine the controller based on counterexamples.
• Comparative results show the approach stabilizes up to 20% more states than LQR in simulated robotic systems, highlighting its advantage in handling input constraints.

Overview of "Lyapunov-stable neural-network control"

The paper "Lyapunov-stable neural-network control" by Hongkai Dai et al. addresses a critical shortcoming in the application of deep reinforcement learning (RL) algorithms for robotic control tasks: the lack of theoretical guarantees on the stability of neural-network controllers. Specifically, the authors focus on achieving Lyapunov stability, which ensures that all trajectories of a closed-loop system will converge to a specified goal state. This stability is vital for adopting such controllers in safety-critical applications where reliability and predictability are paramount.

The authors propose a novel framework to synthesize Lyapunov-stable neural-network controllers along with neural-network-based Lyapunov functions to certify their stability. Unlike traditional methods which are often limited to linear or polynomial controllers, this approach extends the capacity to stabilize complex, nonlinear systems that have their dynamics formulated or approximated with neural networks.

To certify the Lyapunov stability of the proposed controller, the authors adopt a mixed-integer linear programming (MIP) formulation to verify Lyapunov conditions. This verification process either certifies the stability of the proposed controller or identifies counterexamples indicating failure of the candidate function in satisfying stability constraints. The system is then improved iteratively based on these counterexamples. The authors also introduce an optimization program to compute an inner approximation of the region of attraction, which denotes the states from which the system will eventually converge to the equilibrium point under the proposed controller.

Furthermore, the paper demonstrates the efficacy of the proposed approach on various robotic systems, including an inverted pendulum and both 2D and 3D quadrotors. These examples illustrate the capability of neural-network controllers synthesized using their method to outperform baseline linear quadratic regulators (LQRs), particularly in accounting for system input constraints which LQR controllers may neglect.

Key Numerical Results and Comparisons

The results are compelling, especially in terms of performance gains over LQR controllers. In a 2D quadrotor simulation, a neural-network controller stabilized a broader set of initial states to the goal compared to an LQR controller. Specifically, from 10,000 sampled initial states, the neural-network controller successfully stabilized approximately 20% more states than the LQR. This outcome underscores the advantage of leveraging neural networks' capacity for piecewise-linear control formulation over the strictly linear H_{\infty} control method of LQR.

Importantly, the synthesized controllers manage input constraints more effectively, avoiding scenarios where the LQR controller would require impractically high control inputs.

Implications and Future Research

This research has significant implications for the development of safer and more reliable robotics systems that rely on neural-network-based controllers. By providing a method to ensure Lyapunov stability, the authors' approach could potentially pave the way for implementing deep learning controllers in high-stakes fields like autonomous driving and robotics surgery, which demand rigorous stability guarantees.

For future developments, the authors indicate an intention to extend the method to continuous-time systems and other complex robot dynamics approximable by neural networks. Moreover, the scalability challenges inherent in solving MIPs efficiently, especially within large-scale systems, merit further exploration. A promising research direction lies in optimizing verification techniques, potentially by employing tighter formulation strategies or more sophisticated solver techniques tailored to neural-network outputs.

Overall, this work advances the field by coupling empirical success in training neural-network controllers with theoretical guarantees, a significant step towards their broader application and acceptance in autonomous systems.