Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Improving Adversarial Robustness by Contrastive Guided Diffusion Process (2210.09643v2)

Published 18 Oct 2022 in cs.LG and cs.CV

Abstract: Synthetic data generation has become an emerging tool to help improve the adversarial robustness in classification tasks since robust learning requires a significantly larger amount of training samples compared with standard classification tasks. Among various deep generative models, the diffusion model has been shown to produce high-quality synthetic images and has achieved good performance in improving the adversarial robustness. However, diffusion-type methods are typically slow in data generation as compared with other generative models. Although different acceleration techniques have been proposed recently, it is also of great importance to study how to improve the sample efficiency of generated data for the downstream task. In this paper, we first analyze the optimality condition of synthetic distribution for achieving non-trivial robust accuracy. We show that enhancing the distinguishability among the generated data is critical for improving adversarial robustness. Thus, we propose the Contrastive-Guided Diffusion Process (Contrastive-DP), which adopts the contrastive loss to guide the diffusion model in data generation. We verify our theoretical results using simulations and demonstrate the good performance of Contrastive-DP on image datasets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. In ICLR, 2022.
  2. A survey on generative diffusion model. arXiv preprint arXiv:2209.02646, 2022.
  3. Unlabeled data improves adversarial robustness. In NeurIPS, 2019.
  4. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020.
  5. Debiased contrastive learning. In NeurIPS, 2020.
  6. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. ArXiv, abs/2003.01690, 2020.
  7. Conditional synthetic data generation for robust machine learning applications with limited pandemic data. In AAAI, 2022.
  8. Improving adversarial robustness via unlabeled out-of-domain data. In AISTATS, 2021.
  9. Diffusion models beat gans on image synthesis. ArXiv, abs/2105.05233, 2021.
  10. When does contrastive learning preserve adversarial robustness from pretraining to finetuning? In Neural Information Processing Systems, 2021.
  11. Uncovering the limits of adversarial training against norm-bounded adversarial examples. ArXiv, abs/2010.03593, 2020.
  12. Improving robustness using generated data. In NeurIPS, 2021.
  13. Bootstrap your own latent: A new approach to self-supervised learning. ArXiv, abs/2006.07733, 2020.
  14. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  9726–9735, 2020.
  15. Gaussian error linear units (gelus). arXiv: Learning, 2016.
  16. Denoising diffusion probabilistic models. In NeurIPS, 2020.
  17. Detection of traffic signs in real-world images: The German Traffic Sign Detection Benchmark. In International Joint Conference on Neural Networks, number 1288, 2013.
  18. Averaging weights leads to wider optima and better generalization. ArXiv, abs/1803.05407, 2018.
  19. Jammalamadaka, S. R. Directional statistics, i. 2011.
  20. Adversarial self-supervised contrastive learning. ArXiv, abs/2006.07589, 2020a.
  21. Lada: Look-ahead data acquisition via augmentation for active learning. ArXiv, abs/2011.04194, 2020b.
  22. Krizhevsky, A. Learning multiple layers of features from tiny images. 2009.
  23. Gradient-based learning applied to document recognition. Proc. IEEE, 86:2278–2324, 1998.
  24. Liu, Q. Stein variational gradient descent as gradient flow. In NIPS, 2017.
  25. Stein variational gradient descent: A general purpose bayesian inference algorithm. In NIPS, 2016.
  26. Accelerating score-based generative models for high-resolution image synthesis. arXiv preprint arXiv:2206.04029, 2022.
  27. Towards deep learning models resistant to adversarial attacks. ArXiv, abs/1706.06083, 2017.
  28. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162–8171. PMLR, 2021.
  29. Diffusion models for adversarial purification. In International Conference on Machine Learning, 2022.
  30. Scalable end-to-end autonomous vehicle testing via rare-event simulation. In NeurIPS, 2018.
  31. Contrastive learning with hard negative samples. In ICLR, 2021.
  32. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10684–10695, 2022.
  33. Learning to simulate. ArXiv, abs/1810.02513, 2019.
  34. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
  35. Adversarially robust generalization requires more data. In NeurIPS, 2018.
  36. Robust learning meets generative models: Can proxy distributions improve adversarial robustness? In ICLR, 2022.
  37. A finite-particle convergence rate for stein variational gradient descent. ArXiv, abs/2211.09721, 2022.
  38. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256–2265. PMLR, 2015.
  39. Denoising diffusion implicit models. In ICLR, 2021a.
  40. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
  41. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438–12448, 2020.
  42. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pp.  574–584. PMLR, 2020.
  43. Score-based generative modeling through stochastic differential equations. ArXiv, abs/2011.13456, 2021b.
  44. Pointdp: Diffusion-driven purification against adversarial attacks on 3d point cloud recognition. ArXiv, abs/2208.09801, 2022.
  45. Bayesian generative active deep learning. ArXiv, abs/1904.11643, 2019.
  46. Robustness may be at odds with accuracy. arXiv: Machine Learning, 2018.
  47. Representation learning with contrastive predictive coding. ArXiv, abs/1807.03748, 2018.
  48. Sample-efficient neural architecture search by learning action space. ArXiv, abs/1906.06832, 2019.
  49. Learning fast samplers for diffusion models by differentiating through sample quality. In ICLR, 2022.
  50. Theoretical analysis of self-training with deep networks on unlabeled data. In International Conference on Learning Representations, 2020.
  51. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint arXiv:2209.00796, 2022.
  52. Wide residual networks. ArXiv, abs/1605.07146, 2016.
  53. Decoupled adversarial contrastive learning for self-supervised adversarial robustness. ArXiv, abs/2207.10899, 2022.
  54. Theoretically principled trade-off between robustness and accuracy. ArXiv, abs/1901.08573, 2019.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Yidong Ouyang (6 papers)
  2. Liyan Xie (34 papers)
  3. Guang Cheng (136 papers)
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now