An Evaluation of DenseNAS: Densely Connected Search Space for More Flexible Neural Architecture Search
The paper by Jiemin Fang et al. introduces DenseNAS, a novel approach to Neural Architecture Search (NAS) which aims to enhance flexibility in discovering neural network designs by introducing a densely connected search space. This work underscores the importance of automatic block count and width selection as crucial factors affecting network architecture effectiveness, striving for a balance between high accuracy and optimized computational costs, notably FLOPs and latency. The authors propose a distinctive search space represented as a dense super network that employs routing blocks and implements a chained cost estimation algorithm to effectively evaluate the model cost during the search.
Key Contributions
DenseNAS is built on several pivotal contributions which distinguish it from preceding NAS methodologies:
- Densely Connected Search Space: The primary innovation is in its search space realized as a dense super network. This space facilitates exploring numerous architectural paths by automatically determining block counts and block widths. Unlike prior methods that rely on manually fixed block configurations, DenseNAS enables an adaptive approach based on the best-performing paths across the network.
- Cost Optimization via Chained Estimation: The introduction of a chained cost estimation algorithm addresses both adaptation for accuracy and computational expense minimization. This approach calculates model cost iteratively by accounting for interconnected routing blocks, refining cost predictions across potential architectures during the search.
- Extended Search Flexibility: DenseNAS extends exploration capabilities not only regarding layer depth within blocks but also across the number of blocks per stage. It simultaneously permits dynamic decisions about spatial down-sampling, which is a crucial aspect of convolutional neural network scaling.
Empirical Validation and Results
Empirical evaluations on prominent datasets such as ImageNet demonstrate DenseNAS's capacity to deliver competitive top-1 accuracies of up to 76.1% while significantly reducing FLOPs and latency when benchmarked against alternative architectures like MobileNetV2, ProxylessNAS, and FBNet. Notable results include a DenseNAS-derived model achieving 75.3% accuracy with merely 361M FLOPs and 17.9ms latency on a single TITAN-XP. The advantages extend to other models, with significant improvements over traditional ResNet architectures in terms of both accuracy and resource demands, such as a 1.5% increase over ResNet-18 paired with a 200M reduction in FLOPs. These benchmark results, complemented by object detection tasks on COCO, illustrate the model's generalization capabilities beyond classification.
Theoretical and Practical Implications
From a theoretical perspective, DenseNAS provides a framework for NAS that substantially decreases reliance on expert knowledge for determining block parameters, thereby potentially accelerating the development of more generalized NAS methodologies. Practically, it could enable the design of more cost-efficient deep learning models suitable for deployment on edge devices, where computational resources are constrained.
Future Directions and Contributions
The concepts introduced in DenseNAS pave the way for further explorations into flexible search spaces and cost-efficient networks. Future research could examine extending its framework to diverse neural network architectures beyond the dispensaries tested, like MobileNetV3 or VarGNet, and its applicability in various tasks such as semantic segmentation. Additionally, enhancing the search efficiency under varied hardware scenarios remains a pertinent area for further investigation.
In conclusion, DenseNAS advances the contemporary understanding of NAS, illustrating the potent role of search space design in neural network optimization. It effectively integrates accuracy and efficiency considerations into the architecture design process through innovative methodologies, and solidifies the importance of dynamic and densely connected architectures in deep learning.