BlockDrop: Dynamic Inference Paths in Residual Networks (1711.08393v4)

Abstract: Very deep convolutional neural networks offer excellent recognition results, yet their computational expense limits their impact for many real-world applications. We introduce BlockDrop, an approach that learns to dynamically choose which layers of a deep network to execute during inference so as to best reduce total computation without degrading prediction accuracy. Exploiting the robustness of Residual Networks (ResNets) to layer dropping, our framework selects on-the-fly which residual blocks to evaluate for a given novel image. In particular, given a pretrained ResNet, we train a policy network in an associative reinforcement learning setting for the dual reward of utilizing a minimal number of blocks while preserving recognition accuracy. We conduct extensive experiments on CIFAR and ImageNet. The results provide strong quantitative and qualitative evidence that these learned policies not only accelerate inference but also encode meaningful visual information. Built upon a ResNet-101 model, our method achieves a speedup of 20\% on average, going as high as 36\% for some images, while maintaining the same 76.4\% top-1 accuracy on ImageNet.

• The paper introduces BlockDrop, a method that dynamically selects ResNet layers using a policy network trained with reinforcement learning to reduce inference cost.
• It employs a dual-reward system and curriculum learning to navigate layer selection, achieving significant speedups, with up to a 36% improvement on ImageNet for certain images.
• This dynamic inference strategy not only maintains accuracy but also offers practical benefits for real-time applications, inspiring new adaptive network design approaches.

BlockDrop: Dynamic Inference Paths in Residual Networks

The paper "BlockDrop: Dynamic Inference Paths in Residual Networks" introduces a novel approach to enhance the computational efficiency of Residual Networks (ResNets) during inference without adversely affecting prediction accuracy. Given the widespread deployment of deep convolutional neural networks (CNNs) in computer vision applications, their computational demand has often posed a limitation, particularly in scenarios requiring real-time processing or operation on resource-constrained devices.

The authors propose BlockDrop, a technique that dynamically selects which layers of a deep ResNet to execute during inference. This decision-making process is dependent on the input image, thereby allowing the network to tailor its computation based on the complexity of the input. The architecture exploits the inherent robustness of ResNets to dropping layers, leveraging their unique structure where many direct paths (or skip connections) exist between layers, allowing for effective feature reuse even when some layers are not computed.

An essential component of BlockDrop is the policy network, trained using an associative reinforcement learning framework. This network learns to predict a configuration of layers to invoke upon receiving an input image. The training regime incorporates a dual reward system that balances the reduction in computational cost with the preservation of the network's prediction accuracy. The deployment of curriculum learning aids in efficiently navigating the complex search space for layer configurations, while joint finetuning of the ResNet with the policy network ensures the learning of compatible feature transformations despite the dynamic dropping of layers.

Experiments conducted on benchmark datasets such as CIFAR-10, CIFAR-100, and ImageNet demonstrate the efficacy of BlockDrop. When applied to a ResNet-101 model, BlockDrop achieves an average speedup of 20% on ImageNet, while maintaining the original model's top-1 accuracy of 76.4%. In particular, a speedup as high as 36% is recorded for certain images, underscoring the system's ability to adaptively reduce computational cost depending on the input's complexity. These improvements reveal that the learned policies can not only accelerate inference but often encode meaningful visual distinctions based on the image content.

In contrast to existing methods focusing on static model compression—where all images are processed through a fixed compressed network—BlockDrop offers a more granular, instance-specific approach. This method reflects the adaptability of human visual recognition, which allocates varying levels of attention and resources depending on the complexity of the visual stimuli. By allowing the network to decide the computational paths on-the-fly, BlockDrop attains notable efficiency gains over methods like early-exit strategies and other forms of conditional computation that may necessitate intricate, often prohibitive, sequential decision-making frameworks.

Practically, such dynamic inference strategies imply notable reductions in computational costs for numerous real-time applications, especially in domains like autonomous vehicles, where environmental complexities necessitate rapid adaptations. From a theoretical standpoint, BlockDrop provides insights into the optimization of neural network computations through the modular activation of sub-network pathways, which could pave the way for further explorations into adaptive network models.

Looking forward, the BlockDrop framework could be generalized and refined to accommodate other architectures beyond ResNets, such as ResNeXt or Multi-Residual Networks, and extended to tasks beyond image classification, including object detection and video analysis. This broad applicability signifies a compelling area for future research in adaptive neural networks, with the potential to reshape how computational resources are utilized in deep learning models.