Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Neural Architecture Design and Robustness: A Dataset (2306.06712v1)

Published 11 Jun 2023 in cs.LG and cs.CV

Abstract: Deep learning models have proven to be successful in a wide range of machine learning tasks. Yet, they are often highly sensitive to perturbations on the input data which can lead to incorrect decisions with high confidence, hampering their deployment for practical use-cases. Thus, finding architectures that are (more) robust against perturbations has received much attention in recent years. Just like the search for well-performing architectures in terms of clean accuracy, this usually involves a tedious trial-and-error process with one additional challenge: the evaluation of a network's robustness is significantly more expensive than its evaluation for clean accuracy. Thus, the aim of this paper is to facilitate better streamlined research on architectural design choices with respect to their impact on robustness as well as, for example, the evaluation of surrogate measures for robustness. We therefore borrow one of the most commonly considered search spaces for neural architecture search for image classification, NAS-Bench-201, which contains a manageable size of 6466 non-isomorphic network designs. We evaluate all these networks on a range of common adversarial attacks and corruption types and introduce a database on neural architecture design and robustness evaluations. We further present three exemplary use cases of this dataset, in which we (i) benchmark robustness measurements based on Jacobian and Hessian matrices for their robustness predictability, (ii) perform neural architecture search on robust accuracies, and (iii) provide an initial analysis of how architectural design choices affect robustness. We find that carefully crafting the topology of a network can have substantial impact on its robustness, where networks with the same parameter count range in mean adversarial robust accuracy from 20%-41%. Code and data is available at http://robustness.vision/.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Square attack: a query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision, pp.  484–501. Springer, 2020.
  2. Understanding and simplifying one-shot architecture search. In ICML, 2018.
  3. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(10):281–305, 2012.
  4. Proxylessnas: Direct neural architecture search on target task and hardware. In ICLR, 2019.
  5. On evaluating adversarial robustness. CoRR, abs/1902.06705, 2019. URL http://arxiv.org/abs/1902.06705.
  6. Gradvis: Visualization and second order analysis of optimization surfaces during the training of deep neural networks. In Workshop on Machine Learning in High Performance Computing Environments, MLHPC@SC, 2019.
  7. A downsampled variant of imagenet as an alternative to the CIFAR datasets. CoRR, abs/1707.08819, 2017.
  8. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206–2216. PMLR, 2020.
  9. Robustbench: a standardized adversarial robustness benchmark. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), NeurIPS Datasets and Benchmarks, 2021.
  10. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
  11. On adversarial robustness: A neural architecture search perspective. In ICCVW, 2021.
  12. A study and comparison of human and deep learning recognition performance under visual distortions. CoRR, abs/1705.02498, 2017. URL http://arxiv.org/abs/1705.02498.
  13. Adversarially robust neural architectures. CoRR, abs/2009.00902, 2020a.
  14. Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four GPU hours. In CVPR, 2019.
  15. Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In ICLR, 2020.
  16. Benchmarking adversarial robustness on image classification. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  318–328, 2020b. doi: 10.1109/CVPR42600.2020.00040.
  17. Transnas-bench-101: Improving transferability and generalizability of cross-task neural architecture search. In CVPR, 2021.
  18. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6572.
  19. When NAS meets robustness: In search of robust architectures against adversarial attacks. In CVPR, 2020.
  20. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), pp.  1026–1034, 2015. doi: 10.1109/ICCV.2015.123.
  21. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
  22. Robust learning with jacobian regularization. CoRR, abs/1908.02729, 2019.
  23. DSRNA: differentiable search of robust neural architectures. In CVPR, 2021.
  24. Neural architecture search with bayesian optimisation and optimal transport. In NIPS, 2018.
  25. Alex Krizhevsky. Learning multiple layers of features from tiny images. In Technical report, 2009.
  26. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
  27. Adversarial machine learning at scale, 2017.
  28. Random search and reproducibility for neural architecture search. In UAI, 2019.
  29. Learning deep generative models of graphs. CoRR, abs/1803.03324, 2018.
  30. Deepsec: A uniform platform for security analysis of deep learning model. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 673–690, 2019. doi: 10.1109/SP.2019.00023.
  31. DARTS: differentiable architecture search. In ICLR, 2019.
  32. Smooth variational graph embeddings for efficient neural architecture search. In International Joint Conference on Neural Networks, IJCNN 2021, Shenzhen, China, July 18-22, 2021, pp.  1–8. IEEE, 2021.
  33. Learning where to look - generative NAS is surprisingly efficient. In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIII, volume 13683 of Lecture Notes in Computer Science, pp.  257–273. Springer, 2022.
  34. Advrush: Searching for adversarially robust neural architectures. In ICCV, 2021.
  35. Bag of tricks for adversarial training. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=Xb8xvrtB8Ce.
  36. Efficient neural architecture search via parameter sharing. In ICML, 2018.
  37. Foolbox: A python toolbox to benchmark the robustness of machine learning models. In Reliable Machine Learning in the Wild Workshop, 34th International Conference on Machine Learning, 2017. URL http://arxiv.org/abs/1707.04131.
  38. Large-scale evolution of image classifiers. In ICML, 2017.
  39. Regularized evolution for image classifier architecture search. In AAAI, 2019.
  40. Generative adversarial neural architecture search. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pp.  2227–2234. ijcai.org, 2021.
  41. Interpretable neural architecture search via bayesian optimisation with weisfeiler-lehman kernels. In ICLR, 2021.
  42. Is robustness the cost of accuracy? - a comprehensive study on the robustness of 18 deep image classification models. In ECCV, 2018.
  43. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014. URL http://arxiv.org/abs/1312.6199.
  44. Robustart: Benchmarking robustness on architecture design and training techniques. ArXiv, abs/2109.05211, 2021.
  45. Bananas: Bayesian optimization with neural architectures for neural architecture search. In AAAI, 2021a.
  46. Exploring the loss landscape in neural architecture search. In UAI, 2021b.
  47. Feature denoising for improving adversarial robustness. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  501–509, 2019a.
  48. Smooth adversarial training. ArXiv, abs/2006.14536, 2020.
  49. SNAS: stochastic neural architecture search. In ICLR, 2019b.
  50. PC-DARTS: partial channel connections for memory-efficient architecture search. In ICLR, 2020.
  51. Nas-bench-101: Towards reproducible neural architecture search. In ICML, 2019.
  52. Understanding and robustifying differentiable architecture search. In ICLR, 2020.
  53. Bridging mode connectivity in loss landscapes and adversarial robustness. In ICLR, 2020.
  54. Neural architecture search with reinforcement learning. In ICLR, 2017.
Citations (15)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now