Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs

Abstract: A wide variety of deep learning techniques from style transfer to multitask learning rely on training affine transformations of features. Most prominent among these is the popular feature normalization technique BatchNorm, which normalizes activations and then subsequently applies a learned affine transform. In this paper, we aim to understand the role and expressive power of affine parameters used to transform features in this way. To isolate the contribution of these parameters from that of the learned features they transform, we investigate the performance achieved when training only these parameters in BatchNorm and freezing all weights at their random initializations. Doing so leads to surprisingly high performance considering the significant limitations that this style of training imposes. For example, sufficiently deep ResNets reach 82% (CIFAR-10) and 32% (ImageNet, top-5) accuracy in this configuration, far higher than when training an equivalent number of randomly chosen parameters elsewhere in the network. BatchNorm achieves this performance in part by naturally learning to disable around a third of the random features. Not only do these results highlight the expressive power of affine parameters in deep learning, but - in a broader sense - they characterize the expressive power of neural networks constructed simply by shifting and rescaling random features.

• The paper demonstrates that training only BatchNorm’s affine parameters, which comprise just 0.6% of total weights, can yield high performance in CNNs.
• It isolates the impact of γ and β by freezing all other weights, revealing their capacity to disable noisy features and induce sparsity.
• The study shows that deeper networks exploit these affine transformations effectively, achieving notable results like 82% accuracy on CIFAR-10.

Insights into the Expressive Power of Affine Parameters in CNNs

The paper "Training Batch and Only Batch: On the Expressive Power of Random Features in CNNs" by Jonathan Frankle, David J. Schwab, and Ari S. Morcos explores the underexplored area of the expressive power of affine transformations in Convolutional Neural Networks (CNNs). The research centers on Batch Normalization (BatchNorm), a technique ubiquitously employed in modern neural networks. BatchNorm normalizes activations followed by an affine transformation controlled by learnable parameters $\gamma$ (scaling) and $\beta$ (shifting). This study offers a rigorous analysis of the roles these parameters play when disentangled from the learning of network weights.

Key Findings and Methodology

The authors isolate the contribution of affine transformations in BatchNorm by freezing all other network weights at their random initializations and exclusively training the affine parameters. This approach leads to unexpectedly high performance, especially given that only up to 0.6% of parameters are trainable. For instance, they reported that deep ResNets achieved notable accuracy: 82% on CIFAR-10 and 32% top-5 accuracy on ImageNet using this restricted training regimen.

Interestingly, the performance of training only affine parameters was significantly superior to training an equivalent number of randomly chosen parameters elsewhere in the network. This observation suggests that $\gamma$ and $\beta$ possess formidable expressive power, arguably greater than other randomly selected subsets of parameters.

Detailed Analysis and Observations

The study reveals several intriguing behaviors of $\gamma$ and $\beta$ when trained in isolation:

1. Feature Disabling and Sparsity: When only these parameters are trained, the network naturally learns to disable a substantial portion of random features, with $\gamma$ converging to near-zero values for many features. This induces a form of sparsity, highlighting the role of affine transformations in selectively emphasizing certain features while suppressing others.
2. Impact of Network Depth and Width: The work demonstrates that increasing the depth or width of networks provides more foundational random features for $\gamma$ and $\beta$ to manipulate, often resulting in improved performance. However, performance tends to increase more efficiently with depth than with width for a given parameter budget.
3. Comparison with Small Networks: When compared with dense networks of the same total parameter count, networks trained only on affine parameters still lag in performance. However, the results underscore the unique contribution of these parameters when compared to random parameters scattered throughout the network.

Implications and Future Directions

This study accentuates the broad applicability and potential of affine transformations within deep networks, particularly in the context of BatchNorm. By demonstrating that such a small subset of parameters can yield comparatively high accuracy, this work prompts consideration of alternative neural architectures that exploit the expressivity of affine parameters more fully.

The results also invite further research into initialization schemes that could enhance the expressivity of networks with fixed weights, as well as initialization designs that might optimize performance when only $\gamma$ and $\beta$ are trainable. Additionally, exploring the utility of affine parameters in other normalization-free architectures, like those using WeightNorm or FixUp initialization, could broaden the applicability of these findings.

Overall, this paper advances the understanding of BatchNorm's affine transformations in CNNs by elucidating their pivotal role and capacity to encapsulate meaningful representations, even in the absence of tunable network weights. This work lays a foundation for future explorations into the architectural and theoretical dimensions of deep learning models.