Discrete-Time Stability Analysis of ReLU Feedback Systems via Integral Quadratic Constraints

Abstract: This paper analyzes internal stability of a discrete-time feedback system with a ReLU nonlinearity. This feedback system is motivated by recurrent neural networks. We first review existing static quadratic constraints (QCs) for slope-restricted nonlinearities. Next, we derive hard integral quadratic constraints (IQCs) for scalar ReLU by using finite impulse filters and structured matrices. These IQCs are combined with a dissipation inequality leading to an LMI condition that certifies internal stability. We show that our new dynamic IQCs for ReLU are a superset of the well-known Zames-Falb IQCs specified for slope-restricted nonlinearities. Numerical results show that the proposed hard IQCs give less conservative stability margins than Zames-Falb multipliers and prior static QC methods, sometimes dramatically so.