Neural Network Verification with Branch-and-Bound for General Nonlinearities (2405.21063v2)

Abstract: Branch-and-bound (BaB) is among the most effective techniques for neural network (NN) verification. However, existing works on BaB for NN verification have mostly focused on NNs with piecewise linear activations, especially ReLU networks. In this paper, we develop a general framework, named GenBaB, to conduct BaB on general nonlinearities to verify NNs with general architectures, based on linear bound propagation for NN verification. To decide which neuron to branch, we design a new branching heuristic which leverages linear bounds as shortcuts to efficiently estimate the potential improvement after branching. To decide nontrivial branching points for general nonlinear functions, we propose to pre-optimize branching points, which can be efficiently leveraged during verification with a lookup table. We demonstrate the effectiveness of our GenBaB on verifying a wide range of NNs, including NNs with activation functions such as Sigmoid, Tanh, Sine and GeLU, as well as NNs involving multi-dimensional nonlinear operations such as multiplications in LSTMs and Vision Transformers. Our framework also allows the verification of general nonlinear computation graphs and enables verification applications beyond simple NNs, particularly for AC Optimal Power Flow (ACOPF). GenBaB is part of the latest $\alpha,!\beta$-CROWN, the winner of the 4th and the 5th International Verification of Neural Networks Competition (VNN-COMP 2023 and 2024).

• The paper introduces the GenBaB framework that extends branch-and-bound verification to handle diverse nonlinear activations like Sigmoid, Tanh, Sine, and GeLU.
• It details a novel branching heuristic (BBPS) that uses pre-computed linear bounds to optimize branching decisions and improve verification performance.
• Experimental results demonstrate significant gains, including up to 60% verification rate improvements in feedforward networks and successful verifications in LSTMs, ViTs, and ACOPF applications.

Neural Network Verification with Branch-and-Bound for General Nonlinearities: Summary and Contributions

The paper "Neural Network Verification with Branch-and-Bound for General Nonlinearities" introduces the GenBaB framework, advancing neural network (NN) verification by utilizing branch-and-bound (BaB) methodologies for neural networks featuring general nonlinearities. Traditionally, verification efforts have emphasized networks with piecewise linear functions like ReLU due to simpler branching and verification processes. However, many state-of-the-art models incorporate a variety of nonlinear components such as Sigmoid, Tanh, Sine, and GeLU activations, as well as complex computational elements like those found in LSTMs and Vision Transformers (ViTs).

Key Contributions

GenBaB Framework:

GenBaB extends the BaB approach to handle general nonlinear activation functions found in neural networks, overcoming the limitations seen with existing methods that predominantly cater to ReLU networks. The authors emphasize linear bound propagation techniques to guide the branching process. This involves linearizing nonlinear activations for efficient verification.

Branching Heuristic - BBPS:

A novel branching heuristic named "Bound Propagation with Shortcuts" (BBPS) is introduced, enhancing the branching decision process by utilizing pre-computed linear bounds for each neuron to assess the potential improvability of bounds post-branching. This strategy is noted to be more effective than previous methodologies due to better utilization of propagated linear terms up to the input layer of the network.

Offline Optimization of Branching Points:

GenBaB's efficiency is further boosted through the use of optimized branching points, pre-computed and stored in a lookup table to allow quick access during network verification. This pre-optimization aims at achieving the most effective linear relaxation by minimizing the tightness loss associated with different branching points.

Experimental Results

The GenBaB framework showcases significant improvements compared to existing tools and methods. The authors present an extensive empirical evaluation across numerous networks, including feedforward networks, LSTMs, ViTs, and those used in AC Optimal Power Flow (ACOPF) applications:

• Feedforward networks with Sine activations: Verification rates significantly increased from 4% to 60% on certain network configurations, underscoring GenBaB's effectiveness in managing strong nonlinear functions.
• LCSTMs and ViTs: Substantial gains were made over specialized RNN and Transformer verifiers. The proposed method demonstrated superior performance over baselines like PROVER for RNNs and DeepT for Transformers.
• ML4ACOPF problem: GenBaB effectively verified 22 out of 23 instances, exposing practical applicability beyond theoretical models.

Implications and Future Work

GenBaB's versatile approach opens new opportunities by providing a more generalized framework for NN verification, particularly in safety-critical domains where neural networks incorporating nonlinear functions are increasingly employed. It effectively shifts the landscape of network verification from rigid, ReLU-centric methods to a more flexible paradigm. However, the authors acknowledge that while current experiments are promising, further scaling to larger models and diversifying application scenarios is necessary. This may involve refining the framework's capabilities and introducing additional heuristics to manage non-typical NN topologies.

In conclusion, the advancements offered by GenBaB suggest promising future avenues for the verification of complex neural network systems. By broadening the verification capability beyond simplistic nonlinear structures, GenBaB could greatly enhance the deployment safety of neural networks in real-world applications.