Feature Purification: How Adversarial Training Performs Robust Deep Learning (2005.10190v4)

Abstract: Despite the empirical success of using Adversarial Training to defend deep learning models against adversarial perturbations, so far, it still remains rather unclear what the principles are behind the existence of adversarial perturbations, and what adversarial training does to the neural network to remove them. In this paper, we present a principle that we call Feature Purification, where we show one of the causes of the existence of adversarial examples is the accumulation of certain small dense mixtures in the hidden weights during the training process of a neural network; and more importantly, one of the goals of adversarial training is to remove such mixtures to purify hidden weights. We present both experiments on the CIFAR-10 dataset to illustrate this principle, and a theoretical result proving that for certain natural classification tasks, training a two-layer neural network with ReLU activation using randomly initialized gradient descent indeed satisfies this principle. Technically, we give, to the best of our knowledge, the first result proving that the following two can hold simultaneously for training a neural network with ReLU activation. (1) Training over the original data is indeed non-robust to small adversarial perturbations of some radius. (2) Adversarial training, even with an empirical perturbation algorithm such as FGM, can in fact be provably robust against ANY perturbations of the same radius. Finally, we also prove a complexity lower bound, showing that low complexity models such as linear classifiers, low-degree polynomials, or even the neural tangent kernel for this network, CANNOT defend against perturbations of this same radius, no matter what algorithms are used to train them.

Insights into Feature Purification through Adversarial Training

The paper presents a detailed investigation into the phenomenon of adversarial examples in neural networks, addressing both why they arise and how adversarial training can effectively mitigate their impact. Through rigorous theoretical analysis and empirical evidence, the authors introduce and substantiate the concept of "feature purification." This principle highlights how adversarial training refines the features learned during standard training, specifically targeting and removing non-robust components, or "dense mixtures," that make neural networks susceptible to adversarial attacks.

Theoretical Contributions

1. Understanding Adversarial Vulnerability: The authors first elucidate why adversarial examples exist in neural networks trained on clean data. They identify that during standard (or "clean") training, neural networks inadvertently capture dense mixtures in their feature space. These dense mixtures, while not significantly affecting clean data accuracy, are highly sensitive to minor perturbations, thus providing avenues for adversarial exploits.
2. Feature Purification: A central theoretical contribution is the concept of feature purification. The paper posits that adversarial training does not necessitate the discovery of entirely new features; instead, it emphasizes purifying existing ones by eliminating the dense mixtures accumulated during clean training. Such purification enhances the robustness of these features against adversarial perturbations.
3. Empirical and Theoretical Robustness: The paper demonstrates that robust training using algorithms like the Fast Gradient Method (FGM) not only achieves high empirical accuracy against specific adversarial attacks but also ensures theoretical robustness against worst-case norm-bounded adversarial scenarios.
4. Lower Bounds and Model Complexity: The paper explores the limitations of low-complexity models, such as linear classifiers and low-degree polynomials, under adversarial conditions. It shows that even under linearly separable conditions, these models cannot achieve meaningful robustness, underscoring the necessity for high-complexity models like ReLU-based neural networks for adversarial defense.

Practical Implications

• Adversarial Training Regimen: The findings suggest a training strategy where adversarial training proceeds from a model already refined via clean training. This sequential approach is shown to be effective in empirical trials, aligning with practitioners' observations that pre-trained models can benefit from adversarial fine-tuning.
• Low-Rank Updates: A novel insight is that adversarial robustness can often be achieved by low-rank updates to clean-trained models, which can significantly reduce the computational overhead associated with full retraining while still achieving substantial robustness gains.

Empirical Validation

The paper is complemented by comprehensive empirical analyses that corroborate theoretical predictions and provide tangible evidence of feature purification in practice. Visualization techniques applied to models like AlexNet and ResNet reveal that adversarially trained models exhibit more semantically meaningful feature representations, which align closely with real-world image structures, compared to those derived from clean training.

Future Directions

The concept of feature purification presents a new frontier in understanding and improving the robustness of deep learning models. Future research could aim to extend these findings to multi-layer networks and more complex data structures, potentially leading to hierarchical feature purification strategies. The interplay between model architecture and feature purification efficacy also warrants further exploration to optimize the robustness of increasingly sophisticated neural networks.

In summary, this work not only advances our theoretical understanding of adversarial attacks but also offers practical pathways to enhance neural network robustness through a refined understanding of feature dynamics. This dual focus on theory and application makes it a valuable resource for ongoing efforts to make AI systems more secure and reliable.