Can We Gain More from Orthogonality Regularizations in Training Deep CNNs? (1810.09102v1)

Abstract: This paper seeks to answer the question: as the (near-) orthogonality of weights is found to be a favorable property for training deep convolutional neural networks, how can we enforce it in more effective and easy-to-use ways? We develop novel orthogonality regularizations on training deep CNNs, utilizing various advanced analytical tools such as mutual coherence and restricted isometry property. These plug-and-play regularizations can be conveniently incorporated into training almost any CNN without extra hassle. We then benchmark their effects on state-of-the-art models: ResNet, WideResNet, and ResNeXt, on several most popular computer vision datasets: CIFAR-10, CIFAR-100, SVHN and ImageNet. We observe consistent performance gains after applying those proposed regularizations, in terms of both the final accuracies achieved, and faster and more stable convergences. We have made our codes and pre-trained models publicly available: https://github.com/nbansal90/Can-we-Gain-More-from-Orthogonality.

• The paper proposes novel regularization techniques, such as SRIP and DSO, to enhance deep CNN training dynamics.
• Experimental results on architectures like WideResNet and ResNeXt show notable improvements, including a 2.31% reduction in top-1 error on CIFAR-100.
• Incorporating these orthogonality constraints leads to faster convergence, improved optimization efficiency, and greater model stability.

Overview of Orthogonality Regularizations in CNN Training

The paper "Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?" explores enhancing deep convolutional neural network (CNN) training through orthogonality regularizations. Orthogonality of weight matrices is recognized as a favorable property for CNN training; however, achieving and maintaining this property throughout training presents challenges. The authors develop advanced orthogonality regularization techniques, leveraging analytical tools such as mutual coherence (MC) and restricted isometry property (RIP), and evaluate their effectiveness on state-of-the-art CNN architectures like ResNet, WideResNet, and ResNeXt across popular datasets, including CIFAR-10, CIFAR-100, SVHN, and ImageNet.

The introduction acknowledges the intrinsic difficulties inherent in CNN training, such as vanishing and exploding gradients, feature statistic shifts, and saddle points proliferation. The pursuit of orthogonality is motivated by its energy-preserving properties, which, as recognized in signal processing, can improve the stability of activation distributions across network layers and enhance optimization efficiency.

Proposed Orthogonality Regularizations

The authors propose novel regularizations: soft orthogonality (SO), double soft orthogonality (DSO), mutual coherence (MC) regularization, and spectral restricted isometry property (SRIP). These regularizations can be seamlessly integrated into CNN training and require minimal modifications to existing architectures.

1. Baseline Soft Orthogonality (SO): This involves minimizing the Frobenius norm of the difference between the Gram matrix of weights and the identity matrix, which encourages orthogonality. However, it suffers from inherent limitations with overcomplete matrices, where orthogonality cannot be strictly enforced.
2. Double Soft Orthogonality (DSO): Attempts to address limitations in SO by additionally considering the difference between the product of the weight matrix and its transpose with the identity matrix.
3. Mutual Coherence (MC): This regularization method minimizes the infinity norm of the differences between the Gram matrix and identity matrix, aiming to reduce maximum column correlation. However, its effect relies heavily on implicit column normalization.
4. Spectral RIP (SRIP): Derived from the spectral norm properties and RIP, SRIP minimizes the spectral norm of the difference between the Gram matrix and identity matrix, providing a robust formulation for maintaining near-orthogonality.

Experimental Evaluation and Results

The paper conducts extensive experiments across multiple network architectures and datasets. The SRIP regularization consistently outperforms others, demonstrating significant reductions in top-1 error rates across various models and datasets. For instance, when applied to WideResNet on CIFAR-100, a notable 2.31% error reduction is achieved. The paper finds that the incorporation of orthogonality regularizations improves training dynamics, leading to faster, smoother convergence and enhanced model stability.

Additionally, the paper compares SRIP with other existing regularization methods, including spectral regularization, hard Stiefel manifold constraints, and Jacobian norm-based methods. SRIP's superior performance highlights its potential as a valuable tool for boosting training efficiency and accuracy in deep CNNs.

Implications and Future Directions

The findings promote the use of orthogonality regularizations as standard practices when training deep CNNs. The SRIP regularization, in particular, offers a promising approach that bridges theoretical rigor with practical application efficiency. Future research directions could explore extending SRIP to other neural network architectures like RNNs or GANs, where dynamics and weight regularization criteria differ, potentially benefiting from the orthogonality paradigm explored in this paper.

Overall, the paper contributes valuable insights and methodologies for advancing CNN training paradigms, endorsing orthogonality as a beneficial property with quantifiable impacts on model performance and training efficacy.