Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers (2203.08120v1)

Abstract: Training very deep neural networks is still an extremely challenging task. The common solution is to use shortcut connections and normalization layers, which are both crucial ingredients in the popular ResNet architecture. However, there is strong evidence to suggest that ResNets behave more like ensembles of shallower networks than truly deep ones. Recently, it was shown that deep vanilla networks (i.e. networks without normalization layers or shortcut connections) can be trained as fast as ResNets by applying certain transformations to their activation functions. However, this method (called Deep Kernel Shaping) isn't fully compatible with ReLUs, and produces networks that overfit significantly more than ResNets on ImageNet. In this work, we rectify this situation by developing a new type of transformation that is fully compatible with a variant of ReLUs -- Leaky ReLUs. We show in experiments that our method, which introduces negligible extra computational cost, achieves validation accuracies with deep vanilla networks that are competitive with ResNets (of the same width/depth), and significantly higher than those obtained with the Edge of Chaos (EOC) method. And unlike with EOC, the validation accuracies we obtain do not get worse with depth.

• The paper presents Tailored Activation Transformation (TAT) that adapts Leaky ReLUs to enable effective training of deep vanilla networks without shortcuts.
• It leverages deep kernel shaping techniques to adjust Q/C map conditions and overcome ReLU limitations and the Edge of Chaos method.
• The approach simplifies network design by eliminating shortcuts and normalization layers while matching ResNet accuracy on ImageNet.

Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers

The paper under review presents a method for training deep neural networks, with a focus on those without shortcut connections or normalization layers, collectively termed as "vanilla" networks. These networks have historically been difficult to train efficiently at large depths, due to architectural challenges associated with signal propagation and gradient decay. The authors propose a novel approach named Tailored Activation Transformation (TAT) that modifies activation functions, specifically Leaky ReLUs, to ensure trainability and generalization capabilities comparable to ResNet architectures.

Summary

The authors critique the effectiveness of widespread architectures such as ResNets, which incorporate shortcut connections and normalization layers to facilitate depth scalability. Extensive evidence suggests that ResNets, while effective, behave like ensembles of shallower networks rather than truly deep networks. The paper builds upon the current understanding of deep learning dynamics, particularly focusing on Deep Kernel Shaping (DKS), a precedent methodology that analyzes initialization-time kernel properties. Unfortunately, DKS has limitations, especially its incompatibility with ReLU and issues with overfitting on datasets like ImageNet.

In addressing these shortcomings, the authors introduce TAT, which involves the use of Leaky ReLUs instead of ReLUs. This approach allows for manipulation of the activation functions to achieve desirable Q/C map conditions—mathematical constructs that describe signal propagation through the network layers. The authors demonstrate that they can nearly match the validation accuracy metrics of ResNets for ImageNet classification with deep vanilla networks by employing TAT, significantly outperforming the Edge of Chaos (EOC) method and observing no performance degradation with increased depth.

Numerical Results

Key results illustrate that 50-layer vanilla TAT networks achieve nearly the same accuracy on ImageNet as their ResNet counterparts, showcasing the potential of this method to replace widely prevalent architectures without sacrificing performance. Specifically, the TAT method with Leaky ReLUs showed increased robustness and alleviated the maladaptive kernel representation resulting from deep network configurations.

Implications and Future Work

The findings of this paper hold both theoretical and practical implications. Theoretically, TAT contributes to understanding how activation functions can be dynamically adapted to control kernel behavior in deep networks, offering a pathway towards new architectural paradigms. The approach hints at the potential for more streamlined models devoid of traditional shortcuts or normalization architectures, fundamentally impacting memory efficiency and computational demands during inference.

Practically, adopting TAT could simplify design strategies for neural networks, especially in scenarios where traditional architectural complexities are undesirable or infeasible.

Looking ahead, this research may pave the way for exploring alternative architectures informed by kernel and activation dynamics, with further empirical validation across diverse tasks necessary. The research prompts consideration of how these learnings could influence neural architecture search and automated optimization frameworks. Additionally, it raises intriguing questions about how such kernel shaping techniques can be integrated with advances in other domains, such as reinforcement learning and unsupervised learning models.

In conclusion, the paper presents a rigorous exploration into kernel shaping through tailored activation transformations, offering a promising direction in the pursuit of efficiently training deeper networks. By dispensing with traditional architectural crutches such as shortcuts and normalization, it challenges and extends the limits of current neural network design methodologies. The work sets the stage for further investigation into how tailored activations could be generalized beyond current architectures and applications.