Dynamic Perceiver for Efficient Visual Recognition (2306.11248v2)

Abstract: Early exiting has become a promising approach to improving the inference efficiency of deep networks. By structuring models with multiple classifiers (exits), predictions for ``easy'' samples can be generated at earlier exits, negating the need for executing deeper layers. Current multi-exit networks typically implement linear classifiers at intermediate layers, compelling low-level features to encapsulate high-level semantics. This sub-optimal design invariably undermines the performance of later exits. In this paper, we propose Dynamic Perceiver (Dyn-Perceiver) to decouple the feature extraction procedure and the early classification task with a novel dual-branch architecture. A feature branch serves to extract image features, while a classification branch processes a latent code assigned for classification tasks. Bi-directional cross-attention layers are established to progressively fuse the information of both branches. Early exits are placed exclusively within the classification branch, thus eliminating the need for linear separability in low-level features. Dyn-Perceiver constitutes a versatile and adaptable framework that can be built upon various architectures. Experiments on image classification, action recognition, and object detection demonstrate that our method significantly improves the inference efficiency of different backbones, outperforming numerous competitive approaches across a broad range of computational budgets. Evaluation on both CPU and GPU platforms substantiate the superior practical efficiency of Dyn-Perceiver. Code is available at https://www.github.com/LeapLabTHU/Dynamic_Perceiver.

• The paper introduces the Dyn-Perceiver, a dual-branch model that uses symmetric cross-attention to decouple feature extraction from classification for dynamic early exiting.
• Experimental results show 1.9 to 4.8x computational savings on ImageNet with comparable or improved accuracy across various visual tasks.
• The study provides comprehensive ablation analyses that validate the design components and offer guidelines for future efficient network architectures.

Dynamic Perceiver for Efficient Visual Recognition

In the research paper titled "Dynamic Perceiver for Efficient Visual Recognition," the authors address the challenge of inference efficiency in deep neural networks, particularly in visual recognition tasks. They focus on dynamic early exiting as a means to achieve this efficiency, leveraging a novel architecture called the Dynamic Perceiver (Dyn-Perceiver). The paper is structured to introduce and validate the effectiveness of this architecture across multiple visual tasks and computational budgets.

The Dyn-Perceiver framework is positioned as a refinement over existing early-exiting strategies, which traditionally incorporate linear classifiers at various intermediate layers of a deep network. In previous methodologies, these linear classifiers necessitate low-level features to exhibit high-level semantics prematurely, often impairing the performance of subsequent deeper network exits. The Dyn-Perceiver architecture circumvents this limitation through a dual-branch strategy that explicitly decouples feature extraction from early classification tasks. This is achieved by a novel design where a feature extraction branch and a classification branch operate in tandem through symmetric cross-attention mechanisms. The feature branch progressively abstracts image data, while the classification branch processes a latent code devised for prediction tasks. This design choice allows early exits to reside within the classification branch, thereby exempting the low-level features from needing linear separability, which preserves and even enhances the accuracy of deeper layers.

Experimental evaluations are presented across several domains—image classification, action recognition, and object detection—demonstrating the dyn-perceiver's significant improvements in computational efficiency compared to several state-of-the-art models. Notably, it achieves a remarkable reduction in computation by 1.9 to 4.8 times with comparable or improved accuracy levels on ImageNet classification using various backbones, including ResNet and MobileNet variants. These results indicate that the Dyn-Perceiver effectively balances accuracy and efficiency, a crucial outcome in resource-constrained environments.

A key strength of this model lies not only in its theoretical computational savings but also in tangible improvements in real-world latency, substantiated by testing on multiple hardware platforms including CPUs and GPUs. This is of significant practical importance, as actual computation time often diverges from theoretical FLOP counts due to various hardware-level optimizations and overheads. Furthermore, the model's design affords flexibility, enabling it to be seamlessly integrated into a broader array of vision tasks beyond classification, as demonstrated by its deployment in COCO object detection tasks.

Additionally, the paper provides a comprehensive ablation paper, confirming the impact of individual architectural components such as the dual-branch framework, the role of cross-attention layers, and the efficacy of self-distillation techniques in training. The authors also validate the contributions of these components to the robustness and efficiency of the Dyn-Perceiver, thereby offering insightful guidelines for future architecture design.

In conclusion, the Dyn-Perceiver represents a versatile and practical advancement in the ongoing development of more efficient deep networks. As the demand for real-time and computationally economical AI solutions continues to grow, this architecture holds promise for deployment in a wide array of applications, providing a robust baseline for future innovations in dynamic neural network designs. Future research directions might explore extending the cross-attention mechanism to more diverse data modalities and further optimizing hardware-specific implementations.