Smooth Adversarial Training (2006.14536v2)

Abstract: It is commonly believed that networks cannot be both accurate and robust, that gaining robustness means losing accuracy. It is also generally believed that, unless making networks larger, network architectural elements would otherwise matter little in improving adversarial robustness. Here we present evidence to challenge these common beliefs by a careful study about adversarial training. Our key observation is that the widely-used ReLU activation function significantly weakens adversarial training due to its non-smooth nature. Hence we propose smooth adversarial training (SAT), in which we replace ReLU with its smooth approximations to strengthen adversarial training. The purpose of smooth activation functions in SAT is to allow it to find harder adversarial examples and compute better gradient updates during adversarial training. Compared to standard adversarial training, SAT improves adversarial robustness for "free", i.e., no drop in accuracy and no increase in computational cost. For example, without introducing additional computations, SAT significantly enhances ResNet-50's robustness from 33.0% to 42.3%, while also improving accuracy by 0.9% on ImageNet. SAT also works well with larger networks: it helps EfficientNet-L1 to achieve 82.2% accuracy and 58.6% robustness on ImageNet, outperforming the previous state-of-the-art defense by 9.5% for accuracy and 11.6% for robustness. Models are available at https://github.com/cihangxie/SmoothAdversarialTraining.

• The paper demonstrates that replacing ReLU with smooth alternatives enhances gradient quality for generating more effective adversarial examples.
• The paper proposes Smooth Adversarial Training (SAT), which increased ResNet-50's robustness from 33.0% to 42.3% on ImageNet while slightly improving accuracy.
• The paper shows that SAT scales to larger architectures like EfficientNet-L1, achieving 82.2% accuracy and 58.6% robustness without added computational overhead.

Smooth Adversarial Training: Enhancing Robustness without Compromising Accuracy

The paper "Smooth Adversarial Training" presents a significant investigation into the relationship between activation function smoothness and adversarial robustness in neural networks. The authors address the prevalent assumption that robustness against adversarial attacks typically results in reduced accuracy and increased computational costs. They challenge this notion by examining how the widely utilized ReLU activation function might compromise adversarial training due to its inherent non-smooth nature.

Key Findings and Contributions

1. Role of Activation Functions: The research highlights that the non-smooth nature of the ReLU activation function adversely affects the gradient quality during adversarial training. This observation is critical as adversarial training requires precise gradient computations for both generating adversarial examples and updating network parameters.
2. Proposal of Smooth Adversarial Training (SAT): The authors propose Smooth Adversarial Training wherein ReLU is replaced with its smooth approximations, such as Parametric Softplus, SILU, and GELU. These alternatives are smooth by definition, with continuous derivatives that enhance gradient computation, which in turn facilitates the identification of harder adversarial examples and optimal parameter updates.
3. Empirical Validation: The paper validates SAT by demonstrating its ability to bolster adversarial robustness without incurring accuracy penalties or additional computational costs. For instance, SAT increased ResNet-50's robustness from 33.0% to 42.3% while also achieving a 0.9% increase in accuracy on ImageNet.
4. Testing Larger Architectures: The researchers extend SAT to larger network architectures such as EfficientNet. They find that SAT is highly effective across different scales of network size, with EfficientNet-L1 achieving 82.2% accuracy and 58.6% robustness on ImageNet, surpassing previous state-of-the-art methods significantly.
5. Gradient Quality Improvement: The paper also shows that improving gradient quality in either the adversarial attacker or the network optimizer leads to better robustness. The importance of both forward and backward pass smoothness is underscored by these findings.

Practical Implications

The implications of this research are substantial for both theoretical and practical applications in AI. The deployment of SAT does not necessitate additional computational resources, making it a practical choice for enhancing adversarial robustness in existing and new neural network models. By focusing on the activation function smoothness, practitioners can potentially improve model robustness without sacrificing accuracy—a valuable proposition for real-world applications like autonomous driving, where robustness is crucial.

Future Directions

The research suggests several potential directions for future work:

• Architectural Advancements: Further exploration into neural architectures that inherently integrate smooth activation functions could lead to even more robust models.
• Cross-Dataset Evaluations: While initial results on datasets like ImageNet and CIFAR-10 are promising, extending evaluations to other datasets will help generalize the applicability of SAT.
• Adaptive Activation Functions: The development of adaptive or hybrid activation functions that dynamically adjust smoothness based on the specific requirements during training could be explored.

In conclusion, the authors propose a novel approach to adversarial training that emphasizes the fundamental role of activation function smoothness. By systematically addressing the limitations of ReLU, this research provides a pathway toward achieving both high robustness and accuracy in neural networks, thereby contributing to the ongoing endeavor of making AI systems more reliable and secure against adversarial threats.