The devil is in discretization discrepancy. Robustifying Differentiable NAS with Single-Stage Searching Protocol (2405.16610v1)
Abstract: Neural Architecture Search (NAS) has been widely adopted to design neural networks for various computer vision tasks. One of its most promising subdomains is differentiable NAS (DNAS), where the optimal architecture is found in a differentiable manner. However, gradient-based methods suffer from the discretization error, which can severely damage the process of obtaining the final architecture. In our work, we first study the risk of discretization error and show how it affects an unregularized supernet. Then, we present that penalizing high entropy, a common technique of architecture regularization, can hinder the supernet's performance. Therefore, to robustify the DNAS framework, we introduce a novel single-stage searching protocol, which is not reliant on decoding a continuous architecture. Our results demonstrate that this approach outperforms other DNAS methods by achieving 75.3% in the searching stage on the Cityscapes validation dataset and attains performance 1.1% higher than the optimal network of DCNAS on the non-dense search space comprising short connections. The entire training process takes only 5.5 GPU days due to the weight reuse, and yields a computationally efficient architecture. Additionally, we propose a new dataset split procedure, which substantially improves results and prevents architecture degeneration in DARTS.
- Neural optimizer search with reinforcement learning. In ICML, 2017.
- GOLD-NAS: gradual, one-level, differentiable. CoRR, abs/2007.03331, 2020.
- Proxylessnas: Direct neural architecture search on target task and hardware. In ICLR, 2019.
- Searching for efficient multi-scale architectures for dense image prediction. In NIPS, 2018a.
- Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018b.
- Rspnet: Relative speed perception for unsupervised video representation learning. In AAAI, 2021.
- Stabilizing differentiable architecture search via perturbation-based regularization. In ICML, 2020.
- Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In ICCV, 2019.
- Fair DARTS: eliminating unfair advantages in differentiable architecture search. In ECCV, 2020.
- DARTS-: robustly stepping out of performance collapse without indicators. In ICLR, 2021.
- The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
- Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
- Residual conv-deconv grid network for semantic segmentation. In BMVC, 2017.
- DOTS: decoupling operation and topology in differentiable architecture search. In CVPR, 2021.
- Single path one-shot neural architecture search with uniform sampling. In ECCV, 2020.
- Milenas: Efficient neural architecture search via mixed-level reformulation. In CVPR, 2020.
- Searching for mobilenetv3. In ICCV, 2019.
- Adam: A method for stochastic optimization. In ICLR, 2015.
- DARTS+: improved differentiable architecture search with early stopping. CoRR, abs/1909.06035, 2019.
- Graph-guided architecture search for real-time semantic segmentation. In CVPR, 2020.
- Progressive neural architecture search. In ECCV, 2018.
- Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In CVPR, 2019a.
- DARTS: differentiable architecture search. In ICLR, 2019b.
- Megdet: A large mini-batch object detector. In CVPR, 2018.
- Efficient neural architecture search via parameter sharing. In ICML, 2018.
- Full-resolution residual networks for semantic segmentation in street scenes. In CVPR, 2017.
- Large-scale evolution of image classifiers. In ICML, 2017.
- Regularized evolution for image classifier architecture search. In AAAI, 2019.
- Mnasnet: Platform-aware neural architecture search for mobile. In CVPR, 2019.
- Discretization-aware architecture search. Pattern Recognit., 2021.
- Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In CVPR, 2020.
- Rethinking architecture selection in differentiable NAS. In ICLR, 2021.
- Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In CVPR, 2019a.
- Sparsemask: Differentiable connectivity learning for dense image prediction. In ICCV, 2019b.
- SNAS: stochastic neural architecture search. In ICLR, 2019.
- PC-DARTS: partial channel connections for memory-efficient architecture search. In ICLR, 2020.
- CARS: continuous evolution for efficient neural architecture search. In CVPR, 2020.
- β𝛽\betaitalic_β-darts: Beta-decay regularization for differentiable architecture search. In CVPR, 2022.
- Object-contextual representations for semantic segmentation. In ECCV, 2020.
- Understanding and robustifying differentiable architecture search. In ICLR, 2020.
- Interpreting operation selection in differentiable architecture search: A perspective from influence-directed explanations. In NIPS, 2022.
- DCNAS: densely connected neural architecture search for semantic image segmentation. In CVPR, 2021.
- Customizable architecture search for semantic segmentation. In CVPR, 2019.
- Pyramid scene parsing network. In CVPR, 2017.
- Practical block-wise neural network architecture generation. In CVPR, 2018.
- Neural architecture search with reinforcement learning. In ICLR, 2017.
- Learning transferable architectures for scalable image recognition. In CVPR, 2018.