Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness? (2104.09425v3)

Abstract: While additional training data improves the robustness of deep neural networks against adversarial examples, it presents the challenge of curating a large number of specific real-world samples. We circumvent this challenge by using additional data from proxy distributions learned by advanced generative models. We first seek to formally understand the transfer of robustness from classifiers trained on proxy distributions to the real data distribution. We prove that the difference between the robustness of a classifier on the two distributions is upper bounded by the conditional Wasserstein distance between them. Next we use proxy distributions to significantly improve the performance of adversarial training on five different datasets. For example, we improve robust accuracy by up to 7.5% and 6.7% in $\ell_{\infty}$ and $\ell_2$ threat model over baselines that are not using proxy distributions on the CIFAR-10 dataset. We also improve certified robust accuracy by 7.6% on the CIFAR-10 dataset. We further demonstrate that different generative models bring a disparate improvement in the performance in robust training. We propose a robust discrimination approach to characterize the impact of individual generative models and further provide a deeper understanding of why current state-of-the-art in diffusion-based generative models are a better choice for proxy distribution than generative adversarial networks.

Overview of "Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?"

The paper addresses a significant challenge in training robust deep neural networks against adversarial examples, specifically the difficulty of acquiring a sufficiently large and diverse dataset. The authors propose a novel approach that leverages proxy distributions generated by advanced generative models to enhance adversarial robustness.

The core contributions of the paper are threefold:

1. Theoretical Insights on Robustness Transfer
   • The authors establish a formal framework to analyze the transfer of robustness from classifiers trained on proxy distributions to those evaluated on real data. They derive a theoretical upper bound on the difference in robustness between these distributions, which is defined by the conditional Wasserstein distance. This theoretical result provides a quantitative measure of how closely a proxy distribution can approximate the real data distribution in terms of robustness.
2. Empirical Validation and Improvement in Robust Training
   • An extensive series of experiments demonstrate the utility of using proxy distributions in robust training across various datasets. Remarkably, the use of synthetic data from proxy distributions leads to robust accuracy improvements of up to 7.5% and 6.7% for the $\ell_{\infty}$ and $\ell_2$ threat models, respectively, on the CIFAR-10 dataset when compared to existing baselines. Additionally, the incorporation of these proxy distributions also enhances certified robust accuracy, achieving a boost of 7.6%.
3. Robust Discrimination and Proxy Distribution Characterization
   • The work introduces a robust discrimination approach to empirically measure the effectiveness of different generative models as proxy distributions. By evaluating the rate at which a discriminator's success diminishes when distinguishing adversarially perturbed samples from synthetic and real data, the authors define a metric called ARC (Adversarial Robustness Consistency). This metric serves as a surrogate for conditional Wasserstein distance and can accurately predict the transfer of robustness. It is also used to identify effective individual synthetic samples that contribute maximally to robustness.

Implications and Future Directions

The implications of this research are profound both practically and theoretically. On the practical side, leveraging proxy distributions significantly reduces the cost and complexity of data curation while simultaneously improving adversarial robustness. This approach can be especially beneficial in domains where acquiring extensive labeled datasets is prohibitively expensive or impractical.

The theoretical foundation laid by the authors provides valuable insights into the structure and behavior of proxy distributions, prompting further research into optimizing generative models for realistic and robust data generation.

Looking forward, this concept could lead to advancements in generating high-quality proxy data for other AI applications, beyond adversarial robustness. It also opens up new research avenues in exploring the limits of generative models to create data that are not only similar in distribution but also semantically rich and diverse. Future work could integrate these methodologies with self-supervised or semi-supervised learning paradigms to further reduce the dependency on labeled data.

In conclusion, this paper offers a comprehensive paper merging generative modeling and robust machine learning, presenting a promising direction for overcoming one of the pivotal challenges in adversarial robustness.