Three Mechanisms of Weight Decay Regularization (1810.12281v1)

Abstract: Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of $L_2$ regularization. Literal weight decay has been shown to outperform $L_2$ regularization for optimizers for which they differ. We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures. We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the input-output Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization. Our results provide insight into how to improve the regularization of neural networks.

An Analytical Overview of "Three Mechanisms of Weight Decay Regularization"

The paper "Three Mechanisms of Weight Decay Regularization" by Guodong Zhang et al. provides a rigorous examination of weight decay within the context of neural network optimization. The authors aim to demystify the regularization effects of weight decay, separating it from its often-associated $L_2$ norm regularization, and uncovering distinct mechanisms that drive its efficacy in improving generalization across different optimization algorithms and network architectures.

The investigation begins by acknowledging how weight decay has traditionally been intertwined with $L_2$ regularization. However, the paper emphasizes recent observations that challenge this viewpoint, particularly highlighting its superior performance over $L_2$ regularization in scenarios where these methods diverge, such as when using optimization algorithms like Adam.

The paper identifies three core mechanisms through which weight decay provides its regularization benefits:

1. Increased Effective Learning Rate in First-Order Optimization Methods: The paper identifies that in networks employing Batch Normalization (BN), weight decay effectively serves to increase the learning rate by limiting the weights' magnitude. This process enhances the noise in gradients, which plays a pivotal role as a stochastic regularizer. The correlation between learning rate, weight scale, and generalization is methodically illustrated with empirical data showing constant effective learning rates for weight decay, as opposed to decaying rates without it.
2. Regularization of the Input-Output Jacobian Norm in K-FAC Optimization: For K-FAC (Kronecker-Factored Approximate Curvature) optimized networks without BN, the paper proposes that weight decay regularizes the squared Frobenius norm of the input-output Jacobian. This is significant because it aligns with findings that associate reduced Jacobian norms with enhanced generalization. The underlying theory suggests that weight decay indirectly pushes networks toward configurations with less extreme output predictions, empirically validated by strong correlations between reduced Jacobian norms and performance enhancements.
3. Maintenance of the Second-Order Properties via Reduced Effective Damping in Networks with BN: In BN networks optimized with K-FAC, the paper uncovers that weight decay limits weight growth, thereby keeping the damping factor of the curvature matrix small. This maintenance of K-FAC's second-order properties prevents it from degrading into a first-order optimizer and contributes significantly to generalization. Notably, this phenomenon is observed less prominently in Fisher matrix computations due to changes in its norm across training.

The paper's extensive experimental setups, which include results from CIFAR-10 and CIFAR-100 datasets using widely recognized architectures such as VGG16 and ResNet32, emphasize the nuanced differences in performance enhancements associated with weight decay across different settings. Through meticulous testing and hypothesis verification, the paper effectively bridges observations and mathematical insights to deliver a comprehensive understanding of weight decay's impact on neural network training.

In terms of implications, the findings hold substantial promise for optimizing neural network architectures and regularization strategies, particularly in aligning optimization hyperparameters with network design. The paper encourages further exploration into dynamic adaptation of these parameters to better harness the complex interplay between training dynamics and model generalization. By dissecting these mechanisms, the paper provides an actionable pathway toward refining weight decay use in both academia and industry applications, thereby refining design strategies for more robust machine learning models.