All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation (1703.01827v3)

Abstract: Deep neural network is difficult to train and this predicament becomes worse as the depth increases. The essence of this problem exists in the magnitude of backpropagated errors that will result in gradient vanishing or exploding phenomenon. We show that a variant of regularizer which utilizes orthonormality among different filter banks can alleviate this problem. Moreover, we design a backward error modulation mechanism based on the quasi-isometry assumption between two consecutive parametric layers. Equipped with these two ingredients, we propose several novel optimization solutions that can be utilized for training a specific-structured (repetitively triple modules of Conv-BNReLU) extremely deep convolutional neural network (CNN) WITHOUT any shortcuts/ identity mappings from scratch. Experiments show that our proposed solutions can achieve distinct improvements for a 44-layer and a 110-layer plain networks on both the CIFAR-10 and ImageNet datasets. Moreover, we can successfully train plain CNNs to match the performance of the residual counterparts. Besides, we propose new principles for designing network structure from the insights evoked by orthonormality. Combined with residual structure, we achieve comparative performance on the ImageNet dataset.

• The paper introduces orthonormal regularization and backward error modulation to stabilize deep CNN training without the need for shortcut connections.
• It demonstrates that maintaining quasi-isometry during backpropagation enables effective learning in networks exceeding 100 layers.
• Empirical results on CIFAR-10 and ImageNet reveal performance gains of 3-4% and successful training of 44-layer to 110-layer networks.

Overview of Training Deep Convolutional Neural Networks

The paper, titled "All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation," presents an advanced methodology for alleviating common issues encountered during the training of significantly deep convolutional neural networks (CNNs). The work is primarily driven by the challenges of vanishing and exploding gradients, phenomena exacerbated as network depth increases.

Key Contributions

The authors propose a novel approach that combines orthonormal regularization and backward error modulation to enhance the trainability of deep CNNs without relying on shortcut connections, typically employed in architectures such as residual networks. This methodology specifically targets networks structured with repetitive modules composed of Convolution, Batch Normalization (BN), and Rectified Linear Units (ReLU).

Technical Insights

• Orthonormal Regularization: The authors introduce orthonormality constraints across filter banks within each convolutional layer, replacing traditional weight decay regularization. This approach mitigates gradient-related issues by promoting the orthonormality of filters as a sufficient condition for stable error propagation backward through the network. Unlike techniques that only focus on initial conditions, this regularization is persistent throughout the training process and helps maintain orthonormality despite non-linear operations like BN and ReLU.
• Backward Error Modulation: Based on quasi-isometry assumptions, the paper describes a dynamic modulation mechanism that adjusts the global scale of error signal magnitudes during backpropagation. Analysis shows the modulation leverages the near-isometric relations between consecutive layers presumed under BN processing. This strategy helps counteract cumulative non-orthogonal impacts and maintains quasi-isometry even beyond 100 layers in depth, enabling smooth and stable learning dynamics.

Experimental Results

Empirical evaluations on CIFAR-10 and ImageNet datasets demonstrate substantial improvements when training 44-layer and 110-layer networks using the authors' methods. The orthonormal and modulation techniques allow plain CNNs to match the performance of their residual counterparts effectively and, in some cases, achieve superior results.

For instance:

• A 44-layer plain network utilizing orthonormal regularization showed a distinct gains of 3% to 4% on CIFAR-10.
• The application of modulation successfully enabled training on a 110-layer network, showcasing the method's efficacy in overcoming significant depth-related obstacles.

Implications and Future Directions

The outlined approach implies meaningful advancements in designing network architectures for very deep learning models. The introduced modulation mechanism and regularization principles offer routes to optimize CNNs that traditionally rely on shortcuts, potentially reducing computational overhead and unlocking higher layers' expressive power.

Future explorations may delve into refining modulation strategies, potentially integrating orthonormality with more complex adaptive methods tailored for specific architectures. Additionally, examining the impacts on other learning paradigms, such as reinforcement learning or generative models, where depth and expressivity are crucial, could yield insightful results.

The paper extends its contribution by suggesting new principles rooted in orthonormality for network design. These principles, when combined with residual structures, promise comparative performance foundations on large-scale datasets like ImageNet, advocating a shift towards genuinely deep network architectures without dependency on residual shortcuts.

The carefully implemented methodologies detailed in this paper not only address longstanding challenges in deep network training but also lay groundwork for continued theoretical and practical advancements—fueling deeper explorations into AI capabilities and network design optimizations.