Fixing Data Augmentation to Improve Adversarial Robustness (2103.01946v2)

Abstract: Adversarial training suffers from robust overfitting, a phenomenon where the robust test accuracy starts to decrease during training. In this paper, we focus on both heuristics-driven and data-driven augmentations as a means to reduce robust overfitting. First, we demonstrate that, contrary to previous findings, when combined with model weight averaging, data augmentation can significantly boost robust accuracy. Second, we explore how state-of-the-art generative models can be leveraged to artificially increase the size of the training set and further improve adversarial robustness. Finally, we evaluate our approach on CIFAR-10 against $\ell_\infty$ and $\ell_2$ norm-bounded perturbations of size $\epsilon = 8/255$ and $\epsilon = 128/255$, respectively. We show large absolute improvements of +7.06% and +5.88% in robust accuracy compared to previous state-of-the-art methods. In particular, against $\ell_\infty$ norm-bounded perturbations of size $\epsilon = 8/255$, our model reaches 64.20% robust accuracy without using any external data, beating most prior works that use external data.

Improving Adversarial Robustness with Optimized Data Augmentation Techniques

The research presented in the paper "Fixing Data Augmentation to Improve Adversarial Robustness" explores addressing the pervasive phenomenon of robust overfitting in adversarial training of neural networks. Robust overfitting is characterized by a decrement in robust test accuracy over time, despite a continuous improvement in training accuracy. The authors focus on leveraging both heuristics-driven and data-driven augmentations to mitigate this issue and improve adversarial robustness.

Methodological Framework

The authors revisit prior claims that data augmentation does not significantly enhance adversarial robustness. They introduce the concept of combining data augmentation strategies with model weight averaging (WA) to substantially elevate robust accuracy. They further investigate leveraging state-of-the-art generative models to synthetically increase the training dataset size, thereby improving adversarial robustness.

Three pivotal contributions are as follows:

1. Heuristics-Driven Augmentations: Techniques such as Cutout, CutMix, and MixUp, when employed alongside model weight averaging, were shown to meaningfully increase robustness. Notably, CutMix demonstrated a 60.07% robust accuracy on the CIFAR-10 dataset against l_infinity perturbations of size € = 8/255, an improvement of +2.93% over existing state-of-the-art techniques.
2. Data-Driven Augmentations: Exploiting generative models such as the Denoising Diffusion Probabilistic Model (DDPM), BigGAN, and VDVAE allowed the researchers to further enhance adversarial robustness. Among these, DDPM-generated images proved the most efficient, facilitating a robust accuracy of 63.58% under the specified perturbation conditions, thus manifesting a +6.44% improvement over contemporary methods.
3. Combination of Approaches: By synergistically integrating both heuristics-driven and data-driven augmentations, the authors achieved robust accuracies of 64.20% and 80.38% against l_ and l2 norm-bounded perturbations respectively, displaying impressive improvements over prior work.

Evaluation and Results

The experiments conducted on CIFAR-10 with norm-bounded perturbations demonstrated substantial robustness improvements. The models were subjected to rigorous adversarial tests using AUTOATTACK, a composite of advanced parameter-free attacks to ensure the reliability of the robustness claims. The research showcased how strategic data augmentation coupled with advanced model techniques can considerably mitigate robust overfitting, presenting a viable approach to enhance adversarially-trained networks without reliance on external data.

Implications and Future Work

This research illuminates the potential of optimization in data augmentation techniques for adversarial robustness, setting a benchmark for future explorations in neural network defenses. The implications extend to applications requiring heightened security and reliability, such as autonomous systems and financial forecasting models.

Future developments could explore the extension of these findings to more complex datasets and architectures, assessing scalability and efficiency. Additionally, incorporating other forms of data synthesis and augmentation strategies could reveal further enhancements in adversarial robustness, making neural networks more resilient to adversarial threats.

Overall, the paper provides significant insights into reducing robust overfitting and augmenting the adversarial robustness of neural networks, establishing a foundation for further research in robust machine learning methodologies.