Rethinking Bottleneck Structure for Efficient Mobile Network Design (2007.02269v4)

Abstract: The inverted residual block is dominating architecture design for mobile networks recently. It changes the classic residual bottleneck by introducing two design rules: learning inverted residuals and using linear bottlenecks. In this paper, we rethink the necessity of such design changes and find it may bring risks of information loss and gradient confusion. We thus propose to flip the structure and present a novel bottleneck design, called the sandglass block, that performs identity mapping and spatial transformation at higher dimensions and thus alleviates information loss and gradient confusion effectively. Extensive experiments demonstrate that, different from the common belief, such bottleneck structure is more beneficial than the inverted ones for mobile networks. In ImageNet classification, by simply replacing the inverted residual block with our sandglass block without increasing parameters and computation, the classification accuracy can be improved by more than 1.7% over MobileNetV2. On Pascal VOC 2007 test set, we observe that there is also 0.9% mAP improvement in object detection. We further verify the effectiveness of the sandglass block by adding it into the search space of neural architecture search method DARTS. With 25% parameter reduction, the classification accuracy is improved by 0.13% over previous DARTS models. Code can be found at: https://github.com/zhoudaquan/rethinking_bottleneck_design.

• The paper introduces MobileNeXt, a modified bottleneck design addressing the limitations of inverted residual blocks by enhancing gradient propagation.
• The architecture improves performance, achieving a 1.7% accuracy gain on ImageNet and a 0.9% boost on the Pascal VOC 2007 test set.
• Integration into neural architecture search reduced parameters by 25% while yielding further accuracy improvements in mobile networks.

Overview of "Rethinking Bottleneck Structure for Efficient Mobile Network Design"

The paper presents an analysis and improvement on the architecture of residual blocks in mobile networks, challenging the dominance of inverted residual blocks. Authored by researchers affiliated with the National University of Singapore and Yitu Technology, the work introduces a novel modification to residual structures that promises improved performance for mobile networks.

Key Contributions and Methodology

1. Analysis of Inverted Residual Blocks: The authors critically examine the inverted residual block, questioning its efficacy and pointing out potential issues such as information loss and gradient confusion. They argue that these shortcomings arise from connecting identity mapping through low-dimensional bottlenecks.
2. Proposed Bottleneck Design - MobileNeXt: The authors propose a new structure termed MobileNeXt. This architecture upends the traditional inverted residual by positioning identity mappings between high-dimensional features. The revised structure improves information retention and gradient propagation, addressing the bottlenecks' issues identified in existing mobile network architectures.
3. Implementation and Results: MobileNeXt integrates into various network evaluation scenarios, showing improvements without increasing the model's complexity. On ImageNet classification, replacing inverted residual blocks with the new design in MobileNetV2 achieved a 1.7% accuracy improvement. Additionally, it enhanced object detection performance by 0.9% on the Pascal VOC 2007 test set.
4. Neural Architecture Search Enhancement: The novel bottleneck is incorporated into the DARTS neural architecture search space, yielding an architecture with a 0.13% accuracy improvement and a 25% parameter reduction. This indicates the flexibility and additional efficiency of the proposed design in automated network optimization.

Implications and Future Directions

The research serves as a provocation to reevaluate widely accepted design principles in mobile network architectures. The proposed MobileNeXt architecture suggests that reverting to a more traditional bottleneck with strategic adjustments can provide significant performance advantages. Practically, this finding may inspire the adoption of MobileNeXt in various application domains where efficiency and computational resource constraints are critical.

Theoretically, the results invite further investigation into identity mapping and the role of depth-wise convolutions in bottleneck architectures. As neural network design continues to evolve, this work demonstrates the necessity of reassessing even the most established methodologies.

Looking forward, it would be interesting to see this approach employed in real-world scenarios, potentially leading to standardized benchmarks where these bottlenecks could be compared across various types of tasks and datasets. Moreover, the integration with neural architecture search highlights a promising avenue for future exploration, potentially leading to more robust automated network design frameworks.

In conclusion, by advancing the understanding and application of bottleneck structures, this research not only questions prevailing norms in mobile network design but also points towards a path for more efficient and effective deep learning architectures in the mobile context.