Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Towards Robust Vision Transformer via Masked Adaptive Ensemble (2407.15385v1)

Published 22 Jul 2024 in cs.CV and cs.AI

Abstract: Adversarial training (AT) can help improve the robustness of Vision Transformers (ViT) against adversarial attacks by intentionally injecting adversarial examples into the training data. However, this way of adversarial injection inevitably incurs standard accuracy degradation to some extent, thereby calling for a trade-off between standard accuracy and robustness. Besides, the prominent AT solutions are still vulnerable to adaptive attacks. To tackle such shortcomings, this paper proposes a novel ViT architecture, including a detector and a classifier bridged by our newly developed adaptive ensemble. Specifically, we empirically discover that detecting adversarial examples can benefit from the Guided Backpropagation technique. Driven by this discovery, a novel Multi-head Self-Attention (MSA) mechanism is introduced to enhance our detector to sniff adversarial examples. Then, a classifier with two encoders is employed for extracting visual representations respectively from clean images and adversarial examples, with our adaptive ensemble to adaptively adjust the proportion of visual representations from the two encoders for accurate classification. This design enables our ViT architecture to achieve a better trade-off between standard accuracy and robustness. Besides, our adaptive ensemble technique allows us to mask off a random subset of image patches within input data, boosting our ViT's robustness against adaptive attacks, while maintaining high standard accuracy. Experimental results exhibit that our ViT architecture, on CIFAR-10, achieves the best standard accuracy and adversarial robustness of 90.3% and 49.8%, respectively.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (94)
  1. Maksym Andriushchenko and Nicolas Flammarion. 2020. Understanding and Improving Fast Adversarial Training. In NeurIPS.
  2. ViViT: A Video Vision Transformer. In ICCV.
  3. BEiT: BERT Pre-Training of Image Transformers. In ICLR.
  4. UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training. In ICML.
  5. Nicholas Carlini and David A. Wagner. 2017. Towards Evaluating the Robustness of Neural Networks. In Symposium on Security and Privacy (SP).
  6. Emerging Properties in Self-Supervised Vision Transformers. In International Conference on Computer Vision (ICCV).
  7. More data can expand the generalization gap between adversarially robust and standard models. In ICML.
  8. A Simple Framework for Contrastive Learning of Visual Representations. In ICML.
  9. Francesco Croce and Matthias Hein. 2020. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML.
  10. Imagenet: A large-scale hierarchical image database. In CVPR.
  11. GreedyFool: Distortion-Aware Sparse Adversarial Attack. In NeurIPS.
  12. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR.
  13. Multiscale Vision Transformers. In International Conference on Computer Vision (ICCV).
  14. Analyzing and Improving Representations with the Soft Nearest Neighbor Loss. In ICML.
  15. Differential analysis of triggers and benign features for black-box DNN backdoor detection. IEEE Transactions on Information Forensics and Security (2023).
  16. Mitigating Backdoor Attacks on Deep Neural Networks. In Embedded Machine Learning for Cyber-Physical, IoT, and Edge Computing: Use Cases and Emerging Challenges. 395–431.
  17. A feature-based on-line detector to remove adversarial-backdoors by iterative demarcation. IEEE Access (2022), 5545–5558.
  18. Explaining and Harnessing Adversarial Examples. In ICLR.
  19. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 (2017).
  20. Masked autoencoders are scalable vision learners. In CVPR.
  21. Deep Residual Learning for Image Recognition. In CVPR.
  22. Robust electric vehicle balancing of autonomous mobility-on-demand system: A multi-agent reinforcement learning approach. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 5471–5478.
  23. Robust multi-agent reinforcement learning with state uncertainty. arXiv preprint arXiv:2307.16212 (2023).
  24. A Robust and Constrained Multi-Agent Reinforcement Learning Electric Vehicle Rebalancing Method in AMoD Systems. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 5637–5644.
  25. Interpretable Minority Synthesis for Imbalanced Classification. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI). 2542–2548.
  26. Natural Adversarial Examples. In CVPR.
  27. Rethinking Spatial Dimensions of Vision Transformers. In International Conference on Computer Vision (ICCV).
  28. Artificial intelligence aspect of transportation analysis using large scale systems. In Artificial Intelligence and Cloud Computing Conference. 54–59.
  29. LAS-AT: Adversarial Training with Learnable Attack Strategy. In CVPR.
  30. Understanding Catastrophic Overfitting in Single-step Adversarial Training. In AAAI.
  31. Learning multiple layers of features from tiny images. (2009).
  32. ImageNet Classification with Deep Convolutional Neural Networks. In Neural Information Processing Systems (NeurIPS).
  33. Adversarial examples in the physical world. In ICLR.
  34. A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. In NeurIPS.
  35. Subspace Adversarial Training. In CVPR.
  36. MViTv2: Improved Multiscale Vision Transformers for Classification and Detection. In Computer Vision and Pattern Recognition (CVPR).
  37. Yichen Li and Chicheng Zhang. 2023. Ensemble-based Interactive Imitation Learning. arXiv preprint arXiv:2312.16860 (2023).
  38. MMST-ViT: Climate Change-aware Crop Yield Prediction via Multi-Modal Spatial-Temporal Vision Transformer. In IEEE/CVF International Conference on Computer Vision (ICCV). 5774–5784.
  39. An Open and Large-Scale Dataset for Multi-Modal Climate Change-aware Crop Yield Predictions. arXiv preprint arXiv:2406.06081 (2024).
  40. Cascade Variational Auto-Encoder for Hierarchical Disentanglement. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (CIKM). 1248–1257.
  41. Comprehensive Transformer-Based Model Architecture for Real-World Storm Prediction. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML-PKDD). 54–71.
  42. Slowlidar: Increasing the latency of lidar-based detection using adversarial examples. In CVPR. 5146–5155.
  43. RIATIG: Reliable and Imperceptible Adversarial Text-to-Image Generation with Natural Prompts. In CVPR. 20585–20594.
  44. When evil calls: Targeted adversarial voice over ip network. In ACM SIGSAC Conference on Computer and Communications Security (CCS). 2009–2023.
  45. Practical Evaluation of Adversarial Robustness via Adaptive Auto Attack. In CVPR.
  46. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In ICCV.
  47. Ilya Loshchilov and Frank Hutter. 2017. SGDR: Stochastic Gradient Descent with Warm Restarts. In ICLR.
  48. Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In ICLR.
  49. A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction. In American Medical Informatics Association Annual Symposium (AMIA).
  50. A Study of the Attention Abnormality in Trojaned BERTs. In NAACL. 4727–4741.
  51. Shiqing Ma and Yingqi Liu. 2019. Nic: Detecting adversarial samples with neural network invariant checking. In Network and Distributed System Security Symposium (NDSS).
  52. Towards Deep Learning Models Resistant to Adversarial Attacks. In ICLR.
  53. The curious case of adversarially robust models: More data can help, double descend, or hurt generalization. In Uncertainty in Artificial Intelligence.
  54. Deepfool: a simple and accurate method to fool deep neural networks. In CVPR.
  55. The Limitations of Deep Learning in Adversarial Settings. In European Symposium on Security and Privacy (EuroS&P).
  56. Fast Minimum-norm Adversarial Attacks through Adaptive Norm Constraints. In NeurIPS.
  57. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research (2020).
  58. A General Framework For Detecting Anomalous Inputs to DNN Classifiers. In ICML.
  59. High-Resolution Image Synthesis with Latent Diffusion Models. In CVPR.
  60. The Odds are Odd: A Statistical Test for Detecting Adversarial Examples. In ICML.
  61. Kangrui Ruan and Xuan Di. 2022. Learning human driving behaviors with sequential causal imitation learning. In AAAI. 4583–4592.
  62. Kangrui Ruan and Xuan Di. 2024. InfoSTGCAN: An Information-Maximizing Spatial-Temporal Graph Convolutional Attention Network for Heterogeneous Human Trajectory Prediction. Computers 13, 6 (2024), 151.
  63. Causal imitation learning via inverse reinforcement learning. In International Conference on Learning Representations (ICLR).
  64. Ruslan Salakhutdinov and Geoffrey E. Hinton. 2007. Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure. In AISTATS.
  65. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In ICCV.
  66. Adversarial training for free!. In NeurIPS.
  67. Provably robust classification of adversarial examples with detection. In ICLR.
  68. Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In ICLR, Yoshua Bengio and Yann LeCun (Eds.).
  69. BundledSLAM: An Accurate Visual SLAM System Using Multiple Cameras. In 2024 IEEE 7th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), Vol. 7. 106–111.
  70. ETA-INIT: Enhancing the Translation Accuracy for Stereo Visual-Inertial SLAM Initialization. arXiv preprint arXiv:2405.15082 (2024).
  71. Striving for Simplicity: The All Convolutional Net. In ICLR Workshop.
  72. VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. In NeurIPS.
  73. Training data-efficient image transformers & distillation through attention. In ICML.
  74. Florian Tramèr. 2022. Detecting Adversarial Examples Is (Nearly) As Hard As Classifying Them. In ICML.
  75. On Adaptive Attacks to Adversarial Example Defenses. In NeurIPS.
  76. Ensemble Adversarial Training: Attacks and Defenses. In ICLR.
  77. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing Data using t-SNE. Journal of Machine Learning Research (JMLR) (2008).
  78. Attention is All you Need. In NeurIPS.
  79. Removing Batch Normalization Boosts Adversarial Training. In ICML.
  80. Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions. In ICCV.
  81. PVT v2: Improved baselines with Pyramid Vision Transformer. Comput. Vis. Media (2022).
  82. Fast is better than free: Revisiting adversarial training. In ICLR.
  83. Bottrinet: A unified and efficient embedding for social bots detection via metric learning. In 2023 11th International Symposium on Digital Forensics and Security (ISDFS). 1–6.
  84. Botshape: A Novel Social Bots Detection Approach Via Behavioral Patterns. In International Conference on Data Mining & Knowledge Management Process.
  85. ML-LOO: Detecting Adversarial Examples with Feature Attribution. In AAAI.
  86. Automated Discovery of Adaptive Attacks on Adversarial Defenses. In NeurIPS.
  87. Automated Discovery of Adaptive Attacks on Adversarial Defenses. In NeurIPS, Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.).
  88. GAT: Generative Adversarial Training for Adversarial Example Detection and Robust Classification. In ICLR.
  89. Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet. In International Conference on Computer Vision (ICCV).
  90. Theoretically Principled Trade-off between Robustness and Accuracy. In ICML.
  91. Sparse and Imperceptible Adversarial Attack via a Homotopy Algorithm. In ICML.
  92. Wenbo Zhu and Tiechuan Hu. 2021. Twitter Sentiment analysis of covid vaccines. In International Conference on Artificial Intelligence and Virtual Reality (AIVR). 118–122.
  93. Jun Zhuang and Mohammad Al Hasan. 2022a. Defending graph convolutional networks against dynamic graph perturbations via bayesian self-supervision. In AAAI. 4405–4413.
  94. Jun Zhuang and Mohammad Al Hasan. 2022b. Robust node classification on graphs: Jointly from bayesian label transition and topology-based label propagation. In International Conference on Information & Knowledge Management. 2795–2805.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Fudong Lin (6 papers)
  2. Jiadong Lou (5 papers)
  3. Xu Yuan (37 papers)
  4. Nian-Feng Tzeng (12 papers)
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now