Decorrelated Batch Normalization (1804.08450v1)

Abstract: Batch Normalization (BN) is capable of accelerating the training of deep models by centering and scaling activations within mini-batches. In this work, we propose Decorrelated Batch Normalization (DBN), which not just centers and scales activations but whitens them. We explore multiple whitening techniques, and find that PCA whitening causes a problem we call stochastic axis swapping, which is detrimental to learning. We show that ZCA whitening does not suffer from this problem, permitting successful learning. DBN retains the desirable qualities of BN and further improves BN's optimization efficiency and generalization ability. We design comprehensive experiments to show that DBN can improve the performance of BN on multilayer perceptrons and convolutional neural networks. Furthermore, we consistently improve the accuracy of residual networks on CIFAR-10, CIFAR-100, and ImageNet.

• The paper introduces DBN, applying ZCA whitening to mitigate PCA-induced stochastic axis swapping and stabilize training.
• It develops an effective backpropagation method through the inverse square root of covariance matrices, improving training speed and generalization.
• Experimental results on CNNs and ResNets across CIFAR and ImageNet show DBN's enhanced performance in optimization and gradient flow preservation.

Analysis of Decorrelated Batch Normalization

The paper "Decorrelated Batch Normalization" introduces an advancement over the traditional Batch Normalization (BN) technique by proposing Decorrelated Batch Normalization (DBN). This method not only centers and scales activations, akin to BN, but also whitens the activations within mini-batches. The objective here is to improve both the optimization efficiency and generalization ability of neural networks by including the decorrelation of activations, resolving the limitations identified in BN.

Key Contributions and Findings

DBN diverges from BN by performing ZCA whitening, which avoids the stochastic axis swapping issue encountered with PCA whitening. The research identifies that PCA whitening can cause stochastic axis swapping due to random permutations of neurons across mini-batches, which can destabilize learning processes. By applying ZCA whitening instead, DBN achieves consistent covariance decorrelation without introducing these adverse effects.

Another significant contribution is the implementation of an effective back-propagation method through the inverse square root of covariance matrices. The authors leverage matrix differential calculus to overcome the challenges associated with differentiating through the whitening process, which previously deterred attempts at integrating whitening in BN.

Experimental Validation

1. Improved Performance in Neural Networks: The authors conduct comprehensive experiments demonstrating that DBN improves both the training speed and generalization performance over BN in multilayer perceptrons and convolutional neural networks. Notably, DBN shows consistent accuracy enhancements in residual networks across datasets like CIFAR-10, CIFAR-100, and ImageNet.
2. Group Whitening: They address computational concerns and estimation instability with small mini-batches through grouping strategies. By dividing activations into smaller groups for whitening, the DBN method maintains robustness in its covariance estimates and reduces computational overhead, thus making it suitable for practical applications in deep neural networks.

Theoretical Implications

Theoretically, DBN supports approximate dynamical isometry more effectively than BN, due to decorrelating activations rather than merely standardizing them. This property is theorized to preserve gradient magnitudes better during back-propagation, mitigating the vanishing/exploding gradient problems, thus facilitating deeper network architectures.

Moreover, decorrelated activations induce a block-diagonal structure in the Fisher Information Matrix, potentially improving its conditioning and thereby enhancing the convergence rates of optimization processes.

Potential for Future Developments

The integration of DBN into neural network architectures opens new pathways for robust and efficient training methodologies, particularly in deep architectures where gradient flow issues are pronounced. An optimized implementation of DBN reducing computational overhead and enhancing parallelization, especially for architectures requiring heavy computational loads, would further elevate its utility.

Future research might explore the synergy between DBN and other normalization or regularization techniques in neural networks to fully exploit its covariate shift reduction and regularization capacities, potentially crafting generality and applicability in more diverse machine learning paradigms.

Overall, Decorrelated Batch Normalization represents a valuable development in the evolution of training methodologies for neural networks, bridging theoretical insights and practical implementations to address existing limitations in Batch Normalization.