Densely Connected Search Space for More Flexible Neural Architecture Search (1906.09607v3)

Abstract: Neural architecture search (NAS) has dramatically advanced the development of neural network design. We revisit the search space design in most previous NAS methods and find the number and widths of blocks are set manually. However, block counts and block widths determine the network scale (depth and width) and make a great influence on both the accuracy and the model cost (FLOPs/latency). In this paper, we propose to search block counts and block widths by designing a densely connected search space, i.e., DenseNAS. The new search space is represented as a dense super network, which is built upon our designed routing blocks. In the super network, routing blocks are densely connected and we search for the best path between them to derive the final architecture. We further propose a chained cost estimation algorithm to approximate the model cost during the search. Both the accuracy and model cost are optimized in DenseNAS. For experiments on the MobileNetV2-based search space, DenseNAS achieves 75.3% top-1 accuracy on ImageNet with only 361MB FLOPs and 17.9ms latency on a single TITAN-XP. The larger model searched by DenseNAS achieves 76.1% accuracy with only 479M FLOPs. DenseNAS further promotes the ImageNet classification accuracies of ResNet-18, -34 and -50-B by 1.5%, 0.5% and 0.3% with 200M, 600M and 680M FLOPs reduction respectively. The related code is available at https://github.com/JaminFong/DenseNAS.

An Evaluation of DenseNAS: Densely Connected Search Space for More Flexible Neural Architecture Search

The paper by Jiemin Fang et al. introduces DenseNAS, a novel approach to Neural Architecture Search (NAS) which aims to enhance flexibility in discovering neural network designs by introducing a densely connected search space. This work underscores the importance of automatic block count and width selection as crucial factors affecting network architecture effectiveness, striving for a balance between high accuracy and optimized computational costs, notably FLOPs and latency. The authors propose a distinctive search space represented as a dense super network that employs routing blocks and implements a chained cost estimation algorithm to effectively evaluate the model cost during the search.

Key Contributions

DenseNAS is built on several pivotal contributions which distinguish it from preceding NAS methodologies:

1. Densely Connected Search Space: The primary innovation is in its search space realized as a dense super network. This space facilitates exploring numerous architectural paths by automatically determining block counts and block widths. Unlike prior methods that rely on manually fixed block configurations, DenseNAS enables an adaptive approach based on the best-performing paths across the network.
2. Cost Optimization via Chained Estimation: The introduction of a chained cost estimation algorithm addresses both adaptation for accuracy and computational expense minimization. This approach calculates model cost iteratively by accounting for interconnected routing blocks, refining cost predictions across potential architectures during the search.
3. Extended Search Flexibility: DenseNAS extends exploration capabilities not only regarding layer depth within blocks but also across the number of blocks per stage. It simultaneously permits dynamic decisions about spatial down-sampling, which is a crucial aspect of convolutional neural network scaling.

Empirical Validation and Results

Empirical evaluations on prominent datasets such as ImageNet demonstrate DenseNAS's capacity to deliver competitive top-1 accuracies of up to 76.1% while significantly reducing FLOPs and latency when benchmarked against alternative architectures like MobileNetV2, ProxylessNAS, and FBNet. Notable results include a DenseNAS-derived model achieving 75.3% accuracy with merely 361M FLOPs and 17.9ms latency on a single TITAN-XP. The advantages extend to other models, with significant improvements over traditional ResNet architectures in terms of both accuracy and resource demands, such as a 1.5% increase over ResNet-18 paired with a 200M reduction in FLOPs. These benchmark results, complemented by object detection tasks on COCO, illustrate the model's generalization capabilities beyond classification.

Theoretical and Practical Implications

From a theoretical perspective, DenseNAS provides a framework for NAS that substantially decreases reliance on expert knowledge for determining block parameters, thereby potentially accelerating the development of more generalized NAS methodologies. Practically, it could enable the design of more cost-efficient deep learning models suitable for deployment on edge devices, where computational resources are constrained.

Future Directions and Contributions

The concepts introduced in DenseNAS pave the way for further explorations into flexible search spaces and cost-efficient networks. Future research could examine extending its framework to diverse neural network architectures beyond the dispensaries tested, like MobileNetV3 or VarGNet, and its applicability in various tasks such as semantic segmentation. Additionally, enhancing the search efficiency under varied hardware scenarios remains a pertinent area for further investigation.

In conclusion, DenseNAS advances the contemporary understanding of NAS, illustrating the potent role of search space design in neural network optimization. It effectively integrates accuracy and efficiency considerations into the architecture design process through innovative methodologies, and solidifies the importance of dynamic and densely connected architectures in deep learning.