Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Enhancing the "Immunity" of Mixture-of-Experts Networks for Adversarial Defense (2402.18787v1)

Published 29 Feb 2024 in cs.LG and cs.CR

Abstract: Recent studies have revealed the vulnerability of Deep Neural Networks (DNNs) to adversarial examples, which can easily fool DNNs into making incorrect predictions. To mitigate this deficiency, we propose a novel adversarial defense method called "Immunity" (Innovative MoE with MUtual information & positioN stabilITY) based on a modified Mixture-of-Experts (MoE) architecture in this work. The key enhancements to the standard MoE are two-fold: 1) integrating of Random Switch Gates (RSGs) to obtain diverse network structures via random permutation of RSG parameters at evaluation time, despite of RSGs being determined after one-time training; 2) devising innovative Mutual Information (MI)-based and Position Stability-based loss functions by capitalizing on Grad-CAM's explanatory power to increase the diversity and the causality of expert networks. Notably, our MI-based loss operates directly on the heatmaps, thereby inducing subtler negative impacts on the classification performance when compared to other losses of the same type, theoretically. Extensive evaluation validates the efficacy of the proposed approach in improving adversarial robustness against a wide range of attacks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Adversarial example detection using latent neighborhood graph. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7687–7696, 2021.
  2. Deep learning approach for suspicious activity detection from surveillance video. In 2020 2nd International Conference on Innovative Mechanisms for Industry Applications (ICIMIA), pages 335–339. IEEE, 2020.
  3. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
  4. Mutual information neural estimation. In International conference on machine learning, pages 531–540. PMLR, 2018.
  5. Yoshua Bengio. The consciousness prior. arXiv preprint arXiv:1709.08568, 2017.
  6. Learning independent features with adversarial nets for non-linear ica. arXiv preprint arXiv:1710.05050, 2017.
  7. Audio adversarial examples: Targeted attacks on speech-to-text. 2018 IEEE Security and Privacy Workshops (SPW), pages 1–7, 2018.
  8. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1294–1303, 2019.
  9. Deep learning algorithms for detection of critical findings in head ct scans: a retrospective study. The Lancet, 392(10162):2388–2396, 2018.
  10. Provably robust adversarial examples. arXiv preprint arXiv:2007.12133, 2020.
  11. Boosting adversarial attacks with momentum. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9185–9193, 2017.
  12. Explaining and harnessing adversarial examples. CoRR, abs/1412.6572, 2014.
  13. Lau-rens van der maaten,“countering adversarial images using input transformations,”. In International Conference on Learning Representations, 2018.
  14. When nas meets robustness: In search of robust architectures against adversarial attacks. arXiv preprint arXiv:1911.10695, 2019.
  15. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. Advances in Neural Information Processing Systems, 34:29335–29347, 2021.
  16. Comdefend: An efficient image compression model to defend adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6084–6092, 2019.
  17. Adversarial logit pairing. arXiv preprint arXiv:1803.06373, 2018.
  18. Designing deep learning studies in cancer diagnostics. Nature Reviews Cancer, 21(3):199–211, 2021.
  19. Adversarial machine learning at scale. arXiv preprint arXiv:1611.01236, 2016.
  20. Adversarial examples in the physical world. ArXiv, abs/1607.02533, 2016.
  21. Towards robust neural networks via random self-ensemble. In Proceedings of the European Conference on Computer Vision (ECCV), pages 369–385, 2018.
  22. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
  23. Towards deep learning models resistant to adversarial attacks. ArXiv, abs/1706.06083, 2017.
  24. Improving interpretability in medical imaging diagnosis using adversarial training. arXiv preprint arXiv:2012.01166, 2020.
  25. Mixture of experts: a literature survey. Artificial Intelligence Review, 42:275–293, 2014.
  26. Advrush: Searching for adversarially robust neural architectures. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 12302–12312, 2021.
  27. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pages 506–519, 2017.
  28. Deflecting adversarial attacks with pixel deflection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8571–8580, 2018.
  29. Learning disentangled representations via mutual information estimation. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16, pages 205–221. Springer, 2020.
  30. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  31. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017.
  32. Thirdeye: Attention maps for safe autonomous driving systems. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1–12, 2022.
  33. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXiv:1705.07204, 2017.
  34. Skipnet: Learning dynamic routing in convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 409–424, 2018.
  35. Protecting neural networks with hierarchical random switching: Towards better robustness-accuracy trade-off for stochastic defenses. arXiv preprint arXiv:1908.07116, 2019.
  36. Ss-cam: Smoothed score-cam for sharper visual feature localization. arXiv preprint arXiv:2006.14255, 2020.
  37. Score-cam: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 24–25, 2020.
  38. Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 819–828, 2020.
  39. Auxblocks: defense adversarial examples via auxiliary blocks. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2019.
  40. Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems, 30(9):2805–2824, 2019.
  41. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pages 7472–7482. PMLR, 2019.
  42. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921–2929, 2016.
  43. Improving adversarial robustness via mutual information estimation. In International Conference on Machine Learning, pages 27338–27352. PMLR, 2022.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now