Improved Residual Networks for Image and Video Recognition (2004.04989v1)

Abstract: Residual networks (ResNets) represent a powerful type of convolutional neural network (CNN) architecture, widely adopted and used in various tasks. In this work we propose an improved version of ResNets. Our proposed improvements address all three main components of a ResNet: the flow of information through the network layers, the residual building block, and the projection shortcut. We are able to show consistent improvements in accuracy and learning convergence over the baseline. For instance, on ImageNet dataset, using the ResNet with 50 layers, for top-1 accuracy we can report a 1.19% improvement over the baseline in one setting and around 2% boost in another. Importantly, these improvements are obtained without increasing the model complexity. Our proposed approach allows us to train extremely deep networks, while the baseline shows severe optimization issues. We report results on three tasks over six datasets: image classification (ImageNet, CIFAR-10 and CIFAR-100), object detection (COCO) and video action recognition (Kinetics-400 and Something-Something-v2). In the deep learning era, we establish a new milestone for the depth of a CNN. We successfully train a 404-layer deep CNN on the ImageNet dataset and a 3002-layer network on CIFAR-10 and CIFAR-100, while the baseline is not able to converge at such extreme depths. Code is available at: https://github.com/iduta/iresnet

Improved Residual Networks for Image and Video Recognition

The paper under review proposes advancements in the architecture of Residual Networks (ResNets), a prominent convolutional neural network (CNN) structure known for its ability to train very deep networks. The innovations addressed comprise enhancements in information flow through network layers, improvements in the residual building block, and refinements in the projection shortcut. These modifications collectively lead to improved accuracy and learning convergence, with convincing results demonstrated across multiple datasets: ImageNet, CIFAR-10/100, COCO for object detection, and Kinetics-400/Something-Something-v2 for video action recognition.

Deep Network Training and Optimization

ResNets have historically addressed the issue of degradation in deep networks through residual learning. The degradation problem arises as the network depth increases, leading to optimization difficulties. This issue is especially pronounced when the training error unexpectedly increases with additional layers. The paper introduces strategies to tackle these challenges, enabling the training of networks as deep as 404 layers on ImageNet and 3002 layers on CIFAR-10, underlining the capability to extend ResNet architectures beyond typical depths without facing severe optimization issues.

Performance Improvements

The paper notes consistent accuracy improvements, illustrating that their approach yields benefits across a variety of image and video recognition tasks without increasing model complexity. Specifically, the paper documents a top-1 accuracy improvement of 1.19% on 50-layer ResNet when applied to ImageNet, with similar gains demonstrated for deeper architectures. Such performance enhancements are attributed to the improved flow of information through a stage-based architecture, sophisticated projection shortcuts, and a focus on spatial convolutions within the building block.

Architectural Enhancements

1. Stage-Based Approach: The network is divided into stages, each with a specified sequence of start, middle, and end residual building blocks. This configuration facilitates more efficient data propagation across the network's layers, reducing signal degradation typically seen in other architectures.
2. Improved Projection Shortcut: The authors propose a projection shortcut that uses max pooling to manage spatial dimensions, decreasing information loss without adding complexity to the model. The design choice to decouple spatial and channel projections helps maintain information integrity across layers.
3. Enhanced Building Block: Introducing a new building block with increased spatial channel focus allows the middle 3x3 convolutions to operate on a larger number of channels. This design offers improved performance compared to traditional bottleneck structures used in previous ResNet iterations.

Empirical Validation and Implications

Comprehensive comparisons on datasets like ImageNet reveal that these architectural improvements enhance network training speed and convergence stability. The incremental accuracy improvements, achieved without magnifying computational requirements, signify the practical impact of these enhancements for real-world tasks. Notably, object detection and video recognition tasks reflected similar trends in accuracy gains.

Future Directions

The findings suggest a promising avenue for developing even deeper CNNs that can exploit substantial network depth without encountering significant learning obstacles. Future research could explore the coalescence of these techniques with other network architectures or delve into dynamic network structures, where parts of the network can be activated or deactivated based on input complexity and task specificity. The pursuit of highly efficient, deeper networks might propel advancements toward achieving human-like image and video understanding capabilities in AI systems.

In summary, this paper makes a valuable contribution to the ResNet architecture by demonstrating enhancements that facilitate efficient training of exceptionally deep networks. These improvements are likely to impact various domains where deep learning methods are employed, providing a platform for further research into optimizing network depth without compromising learning efficiency.