1-Lipschitz Layers Compared: Memory, Speed, and Certifiable Robustness

Abstract: The robustness of neural networks against input perturbations with bounded magnitude represents a serious concern in the deployment of deep learning models in safety-critical systems. Recently, the scientific community has focused on enhancing certifiable robustness guarantees by crafting 1-Lipschitz neural networks that leverage Lipschitz bounded dense and convolutional layers. Although different methods have been proposed in the literature to achieve this goal, understanding the performance of such methods is not straightforward, since different metrics can be relevant (e.g., training time, memory usage, accuracy, certifiable robustness) for different applications. For this reason, this work provides a thorough theoretical and empirical comparison between methods by evaluating them in terms of memory usage, speed, and certifiable robust accuracy. The paper also provides some guidelines and recommendations to support the user in selecting the methods that work best depending on the available resources. We provide code at https://github.com/berndprach/1LipschitzLayersCompared.

• The paper presents a comprehensive comparison of 1-Lipschitz methods, assessing memory usage, speed efficiency, and certifiable robustness in both training and inference.
• Empirical evaluations on CIFAR-10, CIFAR-100, and Tiny ImageNet show that methods like CPL optimize efficiency while approaches such as SOC perform best when computational resources are abundant.
• The theoretical analysis highlights trade-offs in computational complexity and robust performance, providing guidance for selecting the optimal method based on resource constraints.

Theoretical and Empirical Analysis of 1-Lipschitz Layers

Overview

In the pursuit of enhancing neural networks' robustness to perturbed inputs—especially critical for safety applications—research has centered around certifiable robustness and the use of 1-Lipschitz neural networks. The term "1-Lipschitz" refers to a property where a layer's output does not change more than the input, thus preserving distances and aiding robustness. To implement such networks, various strategies have been developed for dense and convolutional layers, each with distinct computational and memory implications.

Comparing 1-Lipschitz Methods

The study presents a comprehensive comparison of methods for creating 1-Lipschitz layers. The evaluation is centered on memory usage, execution speed, and the robust accuracy that can be certified. Researchers aim to offer guidance for selecting the appropriate method based on resource constraints by examining layers like BCOP, Cayley, SOC, AOL, LOT, CPL, and SLL. Included is a GitHub repository to encourage further research.

Theoretical Insights

The paper undertakes a theoretical comparison of various 1-Lipschitz methods under different conditions. It specifically addresses the complexity and memory usage during both training and inference. This comparison reveals how different layers scale in terms of computational resources required as model sizes increase, and how they affect the overall training and inference times.

Empirical Results

Experiments using different time-constrained training scenarios give a practical evaluation of these layers. The methods were tested on CIFAR-10, CIFAR-100, and Tiny ImageNet datasets to compare memory requirements and certified robust accuracy. Results indicate that while some methods, like SOC, offer slightly better performance when resources are plentiful, others like CPL are more efficient, maintaining high performance with lower computational demands. Another finding is that some methods, such as AOL and BCOP, do not introduce additional inference time overhead compared to standard convolutions, which could be especially beneficial in applications where inference speed is crucial.