Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks (2110.03825v5)

Abstract: Deep neural networks (DNNs) are known to be vulnerable to adversarial attacks. A range of defense methods have been proposed to train adversarially robust DNNs, among which adversarial training has demonstrated promising results. However, despite preliminary understandings developed for adversarial training, it is still not clear, from the architectural perspective, what configurations can lead to more robust DNNs. In this paper, we address this gap via a comprehensive investigation on the impact of network width and depth on the robustness of adversarially trained DNNs. Specifically, we make the following key observations: 1) more parameters (higher model capacity) does not necessarily help adversarial robustness; 2) reducing capacity at the last stage (the last group of blocks) of the network can actually improve adversarial robustness; and 3) under the same parameter budget, there exists an optimal architectural configuration for adversarial robustness. We also provide a theoretical analysis explaning why such network configuration can help robustness. These architectural insights can help design adversarially robust DNNs. Code is available at \url{https://github.com/HanxunH/RobustWRN}.

Insights into Architectural Ingredients of Adversarially Robust Deep Neural Networks

The vulnerability of Deep Neural Networks (DNNs) to adversarial attacks has driven significant research into various defense methods, among which adversarial training has emerged as a prominent approach. Despite its promise, the architectural aspects that contribute to the robustness of adversarially trained DNNs remain inadequately understood. This paper addresses this gap through a comprehensive investigation of network width and depth configurations that enhance adversarial robustness in DNNs.

Key Observations and Findings

The paper's exploration is rooted in adversarial training and focuses on WideResNet-34-10 (WRN-34-10) architecture. The authors conduct a finely-controlled grid search on width and depth configurations, leading to several key observations:

1. Model Capacity and Robustness: Increasing the number of parameters, i.e., upscaling width or depth, does not necessarily enhance adversarial robustness. This finding contradicts prevailing assumptions that higher model capacity uniformly aids robustness.
2. Capacity Reduction in Deeper Layers: An intriguing discovery is that reducing capacity (either depth or width) at the last stage of the network can enhance adversarial robustness. This reduction at deeper layers, particularly in WRNs, can achieve a beneficial trade-off between capacity and Lipschitzness, fostering more robust models.
3. Optimal Architectural Configuration: The research identifies that under the same parameter constraints, an optimal architectural configuration exists that maximizes adversarial robustness. This configuration rule is also applicable for improving the robustness of other architectures like VGGs, DenseNets, and models found through NAS (Neural Architecture Search).

Theoretical Insights

The authors provide a theoretical framework to understand the relationship between architectural configurations and adversarial robustness. They establish that wider and deeper models have increased Lipschitz constants, which correlates with decreased robustness due to greater changes in output for small input perturbations. This insight underpins their empirical findings, especially the benefit of reducing capacity at deeper layers to manage this trade-off effectively.

Implications and Future Directions

This research offers valuable insights that can inform the design of more adversarially robust DNN architectures. It challenges the simplistic notion that increased model capacity is inherently beneficial for robustness and instead highlights the nuanced role of architectural configurations. By demonstrating that robustness can be enhanced through strategic capacity reductions, particularly at the deeper network stages, the paper opens avenues for optimizing network design without the extensive computational costs associated with NAS.

Considering the broader context of deep learning, these findings emphasize the importance of not only innovating on training dynamics but also critically evaluating and optimizing neural architecture components. Future developments could expand these insights to other DNN architectures or explore dynamic adjustment strategies that adapt architectural components based on evolving robustness criteria.

In summary, this paper furnishes the academic community with evidence-backed strategies for architecting robust DNNs and provides a groundwork for future research into the complex interplay between model architectures and adversarial robustness.