Towards Stable and Efficient Training of Verifiably Robust Neural Networks (1906.06316v2)

Abstract: Training neural networks with verifiable robustness guarantees is challenging. Several existing approaches utilize linear relaxation based neural network output bounds under perturbation, but they can slow down training by a factor of hundreds depending on the underlying network architectures. Meanwhile, interval bound propagation (IBP) based training is efficient and significantly outperforms linear relaxation based methods on many tasks, yet it may suffer from stability issues since the bounds are much looser especially at the beginning of training. In this paper, we propose a new certified adversarial training method, CROWN-IBP, by combining the fast IBP bounds in a forward bounding pass and a tight linear relaxation based bound, CROWN, in a backward bounding pass. CROWN-IBP is computationally efficient and consistently outperforms IBP baselines on training verifiably robust neural networks. We conduct large scale experiments on MNIST and CIFAR datasets, and outperform all previous linear relaxation and bound propagation based certified defenses in $\ell_\infty$ robustness. Notably, we achieve 7.02% verified test error on MNIST at $\epsilon=0.3$, and 66.94% on CIFAR-10 with $\epsilon=8/255$. Code is available at https://github.com/deepmind/interval-bound-propagation (TensorFlow) and https://github.com/huanzhang12/CROWN-IBP (PyTorch).

• The paper presents CROWN-IBP, a novel approach that blends IBP's speed with CROWN's tight bounds to ensure stable training and verifiable robustness against adversarial attacks.
• The method demonstrates significant improvements, achieving a 7.02% verified test error on MNIST and 66.94% on CIFAR-10, thereby outperforming baseline techniques.
• By balancing computational efficiency and accuracy, CROWN-IBP provides a promising framework for enhancing reliability in safety-critical deep neural networks.

Towards Stable and Efficient Training of Verifiably Robust Neural Networks

The paper presents an innovative approach, CROWN-IBP, for training neural networks with verifiable robustness, specifically targeting computational efficiency and training stability. The goal is to ensure that neural networks remain robust in environments where input perturbations, such as adversarial attacks, are a concern, which is particularly important for safety-critical applications.

Context and Motivation

Deep neural networks (DNNs) are highly susceptible to adversarial attacks, where slight perturbations to the input data can lead to significant misclassifications. Traditional defensive strategies lack the ability to provide certifiable robustness—that is, mathematical guarantees that ensure the model's integrity against such attacks. Existing methods to provide these guarantees, predominantly based on linear relaxation techniques, face significant computational and training efficiency challenges. Meanwhile, Interval Bound Propagation (IBP) offers computational efficiency but struggles with loose bounds, leading to potentially unstable training. The confluence of these challenges underscores the need for the development of new methods that can ensure both efficiency and stability.

Methodology: The CROWN-IBP Approach

The proposed method, CROWN-IBP, fundamentally integrates the fast computation of IBP with the tightness of the CROWN (a linear relaxation-based verification method). This dual approach involves leveraging the quick IBP bounds during the forward pass and the precise CROWN bounds during the backward pass of training. Such combination enables the method to overcome the computational inefficiencies of purely linear relaxation-based methods and the instability of IBP.

The forward pass is responsible for determining preliminary bounds using IBP, while the backward pass refines these bounds with the CROWN mechanism. This effectively balances computational overhead and bounding accuracy, making CROWN-IBP well-suited for large networks, which are often constrained by computational resources.

Experimental Results

Extensive experimentation was conducted on popular benchmark datasets such as MNIST and CIFAR-10. The results highlight several key points:

• Efficiency and Accuracy: CROWN-IBP consistently delivered superior verified test performance compared to baseline IBP methods, achieving a 7.02% verified test error on MNIST with ε=0.3, and 66.94% on CIFAR-10 with ε=8/255. This underscores its efficacy over existing relaxation-based methods concerning both computation time and verifiably robust error rates.
• Stability Across Hyperparameters: CROWN-IBP demonstrates increased model robustness and stability across different hyperparameter settings, offering a more reliable framework for researchers and practitioners seeking guaranteed model robustness.

Implications and Future Work

The development of CROWN-IBP presents significant advancements in the domain of adversarial robustness, delivering a method that promises efficiency without compromising on robustness guarantees. As computational demands continue to grow with more complex networks and datasets, methods like CROWN-IBP become crucial.

Future directions could delve into further refining the backward pass using more sophisticated linear relaxation techniques or might explore scaling CROWN-IBP to networks with a broader class of activation functions beyond ReLU. Another avenue could focus on enhancing the flexibility and scalability of CROWN-IBP across other architectures and larger-scale datasets like ImageNet, employing techniques such as random sampling to manage computational load.

In conclusion, CROWN-IBP represents a significant step forward in training verifiably robust networks, marrying the strengths of established methods while addressing their respective limitations. This methodological blend provides a robust path forward for future research in secure and dependable artificial intelligence systems.