EMOv2: Pushing 5M Vision Model Frontier

Abstract: This work focuses on developing parameter-efficient and lightweight models for dense predictions while trading off parameters, FLOPs, and performance. Our goal is to set up the new frontier of the 5M magnitude lightweight model on various downstream tasks. Inverted Residual Block (IRB) serves as the infrastructure for lightweight CNNs, but no counterparts have been recognized by attention-based design. Our work rethinks the lightweight infrastructure of efficient IRB and practical components in Transformer from a unified perspective, extending CNN-based IRB to attention-based models and abstracting a one-residual Meta Mobile Block (MMBlock) for lightweight model design. Following neat but effective design criterion, we deduce a modern Improved Inverted Residual Mobile Block (i2RMB) and improve a hierarchical Efficient MOdel (EMOv2) with no elaborate complex structures. Considering the imperceptible latency for mobile users when downloading models under 4G/5G bandwidth and ensuring model performance, we investigate the performance upper limit of lightweight models with a magnitude of 5M. Extensive experiments on various vision recognition, dense prediction, and image generation tasks demonstrate the superiority of our EMOv2 over state-of-the-art methods, e.g., EMOv2-1M/2M/5M achieve 72.3, 75.8, and 79.4 Top-1 that surpass equal-order CNN-/Attention-based models significantly. At the same time, EMOv2-5M equipped RetinaNet achieves 41.5 mAP for object detection tasks that surpasses the previous EMO-5M by +2.6. When employing the more robust training recipe, our EMOv2-5M eventually achieves 82.9 Top-1 accuracy, which elevates the performance of 5M magnitude models to a new level. Code is available at https://github.com/zhangzjn/EMOv2.

• The paper introduces EMOv2 with a novel Meta Mobile Block that integrates IRB and MHSA to boost performance in dense prediction tasks.
• The paper presents an improved inverted residual block (iRMB) with an Expanded Window MHSA that reduces computational complexity while enhancing accuracy.
• The paper demonstrates state-of-the-art results with up to 82.9% Top-1 accuracy and 41.5 mAP on RetinaNet, highlighting its efficiency across various tasks.

Overview of EMOv2: Pushing 5M Vision Model Frontier

The presented paper titled "EMOv2: Pushing 5M Vision Model Frontier" represents a focused effort in developing parameter-efficient models that specifically target dense prediction tasks. The authors have introduced an innovative framework called EMOv2, which sets the foundation for the 5M parameter frontier in a way that significantly challenges the paradigms established by baseline CNNs and Transformers.

EMOv2 leverages a newly hypothesized one-residual Meta Mobile Block (MMBlock), encapsulating efficient design principles from both Inverted Residual Blocks (IRB) used in MobileNet and attention-based designs. This new abstraction not only extends the lightweight computational advantages of IRBs to attention-based models but also innovatively merges them into what the authors call the Improved Inverted Residual Mobile Block (iRMB).

Key Contributions

1. Meta Mobile Block (MMBlock): The MMBlock is a versatile lightweight module derived through the integration of core aspects of IRB and Multi-Head Self-Attention (MHSA) mechanisms. It effectively combines convolutional and attention-based strategies to increase the performance ceiling for models with capped parameter counts.
2. Improved Inverted Residual Mobile Block (iRMB): This is a refined version of the IRB, incorporating an efficient attention mechanism suitable for low-computation environments. The attention mechanism includes an Expanded Window MHSA (EW-MHSA), designed to reduce quadratic complexities by focusing on channel-efficient, expanded dimension operations.
3. Spanning Attention Design: The spanning attention extends the receptive field and simultaneously enhances the model's accuracy and broadened applicability across different layers without a subsequent increase in parameter counts. Such enhancements are critical in high-resolution object detection and image segmentation tasks.
4. Efficiency and Performance: The EMOv2 model achieves state-of-the-art performance with notable efficiency. Compared to contemporary models operating within the similar parameter magnitude of 5M, EMOv2 applications in classification tasks achieve up to 82.9% Top-1 accuracy. Furthermore, the model demonstrates substantial improvements across object detection benchmarks (41.5 mAP on RetinaNet) and image generation tasks compared to its predecessors like EMOv1 and even advanced ViTs.
5. Wide Applicability: The model's architecture can be extended to various tasks, including modified architectures like UNet for segmentation or temporal dimensions like V-EMO for video classification, which showcases its adaptability without losing the lightweight attribute.

Implications and Speculations

The development and implementation of EMOv2 can significantly impact both theoretical directions and practical deployments of lightweight vision models. By demonstrating such high performance curves on existing and potential applications, the paper posits these hybrid approaches as indispensable for resource-constrained environments prevalent in mobile and edge computing. The lightweight aspect addresses latency, power, and storage constraints—a longstanding challenge in deploying deep learning architectures at scale.

Future work may focus on expanding this framework to both larger model sizes and newer domains, including real-time analytics and other application areas demanding balanced operations between performance and resource usage. Furthermore, the inclusion of more robust training recipes and exploration of the model under diverse data conditions can enhance its generalization potential across unseen environments.

In conclusion, EMOv2 establishes a comprehensive evaluation framework for designing efficient, scalable architectures crucial for next-gen computing platforms, setting a strong precedent for future research in this domain.