Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

DD-RobustBench: An Adversarial Robustness Benchmark for Dataset Distillation (2403.13322v3)

Published 20 Mar 2024 in cs.CV

Abstract: Dataset distillation is an advanced technique aimed at compressing datasets into significantly smaller counterparts, while preserving formidable training performance. Significant efforts have been devoted to promote evaluation accuracy under limited compression ratio while overlooked the robustness of distilled dataset. In this work, we introduce a comprehensive benchmark that, to the best of our knowledge, is the most extensive to date for evaluating the adversarial robustness of distilled datasets in a unified way. Our benchmark significantly expands upon prior efforts by incorporating a wider range of dataset distillation methods, including the latest advancements such as TESLA and SRe2L, a diverse array of adversarial attack methods, and evaluations across a broader and more extensive collection of datasets such as ImageNet-1K. Moreover, we assessed the robustness of these distilled datasets against representative adversarial attack algorithms like PGD and AutoAttack, while exploring their resilience from a frequency perspective. We also discovered that incorporating distilled data into the training batches of the original dataset can yield to improvement of robustness.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., “Imagenet large scale visual recognition challenge,” IJCV, 2015.
  2. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in ECCV, 2014.
  3. M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes challenge: A retrospective,” IJCV, 2015.
  4. T. Wang, J. Zhu, A. Torralba, and A. A. Efros, “Dataset distillation,” CoRR, 2018.
  5. B. Zhao, K. R. Mopuri, and H. Bilen, “Dataset condensation with gradient matching,” in ICLR, 2021.
  6. B. Zhao and H. Bilen, “Dataset condensation with differentiable siamese augmentation,” in ICML, M. Meila and T. Zhang, Eds., 2021.
  7. ——, “Dataset condensation with distribution matching,” in WACV, 2023.
  8. G. Cazenavette, T. Wang, A. Torralba, A. A. Efros, and J.-Y. Zhu, “Dataset distillation by matching training trajectories,” in CVPR, 2022.
  9. F. P. Such, A. Rawal, J. Lehman, K. O. Stanley, and J. Clune, “Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data,” in ICML, 2020.
  10. M. Sangermano, A. Carta, A. Cossu, and D. Bacciu, “Sample condensation in online continual learning,” in IJCNN, 2022.
  11. Y. Xiong, R. Wang, M. Cheng, F. Yu, and C. Hsieh, “Feddm: Iterative distribution matching for communication-efficient federated learning,” in CVPR, 2023.
  12. J. Cui, R. Wang, S. Si, and C.-J. Hsieh, “Scaling up dataset distillation to imagenet-1k with constant memory,” in ICML, 2023.
  13. I. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in ICLR, 2015.
  14. A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in ICLR, 2018.
  15. N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in IEEE symposium on security and privacy, 2017.
  16. F. Croce and M. Hein, “Minimally distorted adversarial examples with a fast adaptive boundary attack,” in ICML, 2020.
  17. Z. Chen, J. Geng, D. Zhu, H. Woisetschlaeger, Q. Li, S. Schimmler, R. Mayer, and C. Rong, “A Comprehensive Study on Dataset Distillation: Performance, Privacy, Robustness and Fairness,” arXiv e-prints, 2023.
  18. Z. Yin, E. Xing, and Z. Shen, “Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective,” in NeurIPS, 2023.
  19. A. Krizhevsky, G. Hinton et al., “Learning multiple layers of features from tiny images,” 2009.
  20. A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry, “Adversarial examples are not bugs, they are features,” in NeurIPS, 2019.
  21. H. Wang, X. Wu, Z. Huang, and E. P. Xing, “High-frequency component helps explain the generalization of convolutional neural networks,” in CVPR, 2020.
  22. D. Medvedev and A. D’yakonov, “Learning to generate synthetic training data using gradient matching and implicit differentiation,” in Recent Trends in Analysis of Images, Social Networks and Texts - 10th International Conference, AIST 2021, Tbilisi, Georgia, December 16-18, 2021, Revised Supplementary Proceedings, E. Burnaev, D. I. Ignatov, S. Ivanov, M. Y. Khachay, O. Koltsova, A. Kutuzov, S. O. Kuznetsov, N. V. Loukachevitch, A. Napoli, A. Panchenko, P. M. Pardalos, J. Saramäki, A. V. Savchenko, E. Tsymbalov, and E. Tutubalina, Eds., 2021.
  23. A. Rosasco, A. Carta, A. Cossu, V. Lomonaco, and D. Bacciu, “Distilled replay: Overcoming forgetting through synthetic samples,” in International Workshop on Continual Semi-Supervised Learning, 2021.
  24. P. Liu, X. Yu, and J. T. Zhou, “Meta knowledge condensation for federated learning,” in ICLR, 2023.
  25. T. Nguyen, Z. Chen, and J. Lee, “Dataset meta-learning from kernel ridge-regression,” in ICLR, 2021.
  26. K. Wang, B. Zhao, X. Peng, Z. Zhu, S. Yang, S. Wang, G. Huang, H. Bilen, X. Wang, and Y. You, “CAFE: learning to condense dataset by aligning features,” in CVPR, 2022.
  27. Z. Guo, K. Wang, G. Cazenavette, H. Li, K. Zhang, and Y. You, “Towards lossless dataset distillation via difficulty-aligned trajectory matching,” CoRR, 2023.
  28. O. Bohdal, Y. Yang, and T. M. Hospedales, “Flexible dataset distillation: Learn labels instead of images,” in 4th Workshop on Meta-Learning at NeurIPS 2020, 2020.
  29. I. Sucholutsky and M. Schonlau, “Soft-label dataset distillation and text dataset distillation,” in IJCNN, 2021.
  30. Y. Liu, J. Gu, K. Wang, Z. Zhu, W. Jiang, and Y. You, “DREAM: efficient dataset distillation by representative matching,” in ICCV, 2023.
  31. Y. Liu, J. Gu, K. Wang, Z. Zhu, K. Zhang, W. Jiang, and Y. You, “DREAM+: efficient dataset distillation by bidirectional representative matching,” CoRR, 2023.
  32. Y. Xu, Y.-L. Li, K. Cui, Z. Wang, C. Lu, Y.-W. Tai, and C.-K. Tang, “Distill gold from massive ores: Efficient dataset distillation via critical samples selection,” arXiv preprint arXiv:2305.18381, 2023.
  33. Z. Yin and Z. Shen, “Dataset distillation in large data era,” arXiv preprint arXiv:2311.18838, 2023.
  34. J. Cui, R. Wang, S. Si, and C. Hsieh, “DC-BENCH: dataset condensation benchmark,” in NeurIPS, 2022.
  35. Y. Hu, F. Wu, H. Zhang, and H. Zhao, “Understanding the impact of adversarial robustness on accuracy disparity,” in ICML, 2023.
  36. S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool: a simple and accurate method to fool deep neural networks,” in CVPR, 2016.
  37. F. Croce and M. Hein, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,” in ICML, 2020.
  38. C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in ICLR, 2014.
  39. A. Kurakin, I. Goodfellow, and S. Bengio, “Adversarial examples in the physical world,” ICLR Workshop, 2017.
  40. Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li, “Boosting adversarial attacks with momentum,” in CVPR, 2018.
  41. M. Andriushchenko, F. Croce, N. Flammarion, and M. Hein, “Square attack: a query-efficient black-box adversarial attack via random search,” in ECCV, 2020.
  42. H. Kannan, A. Kurakin, and I. J. Goodfellow, “Adversarial logit pairing,” in NeurIPS, 2018.
  43. A. Shafahi, M. Najibi, M. A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein, “Adversarial training for free!” in NeurIPS, 2019.
  44. M. Goldblum, L. Fowl, S. Feizi, and T. Goldstein, “Adversarially robust distillation,” in AAAI, 2020.
  45. H. Wang, Y. Deng, S. Yoo, H. Ling, and Y. Lin, “Agkd-bml: Defense against adversarial attack by attention guided knowledge distillation and bi-directional metric learning,” in ICCV, 2021.
  46. K. Yang, T. Zhou, Y. Zhang, X. Tian, and D. Tao, “Class-disentanglement and applications in adversarial detection and defense,” in NeurIPS, 2021.
  47. Z. Wang, T. Jian, A. Masoomi, S. Ioannidis, and J. Dy, “Revisiting hilbert-schmidt information bottleneck for adversarial robustness,” in NeurIPS, 2021.
  48. P. Bashivan, R. Bayat, A. Ibrahim, K. Ahuja, M. Faramarzi, T. Laleh, B. Richards, and I. Rish, “Adversarial feature desensitization,” in NeurIPS, 2021.
  49. D. Zhou, T. Liu, B. Han, N. Wang, C. Peng, and X. Gao, “Towards defending against adversarial examples via attack-invariant features,” in ICML, 2021.
  50. C. Xie, Y. Wu, L. v. d. Maaten, A. L. Yuille, and K. He, “Feature denoising for improving adversarial robustness,” in CVPR, 2019.
  51. Y. Wang, D. Zou, J. Yi, J. Bailey, X. Ma, and Q. Gu, “Improving adversarial robustness requires revisiting misclassified examples,” in ICLR, 2019.
  52. H. Shen, S. Chen, R. Wang, and X. Wang, “Adversarial learning with cost-sensitive classes,” IEEE Transactions on Cybernetics, 2022.
  53. Y. Le and X. Yang, “Tiny imagenet visual recognition challenge,” CS 231N, 2015.
  54. Y. Liu, Y. Cheng, L. Gao, X. Liu, Q. Zhang, and J. Song, “Practical evaluation of adversarial robustness via adaptive auto attack,” in CVPR, 2022.
  55. D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry, “Robustness may be at odds with accuracy,” in ICLR, 2019.
  56. A. Kessy, A. Lewin, and K. Strimmer, “Optimal whitening and decorrelation,” The American Statistician, 2018.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets