Hessian-based Analysis of Large Batch Training and Robustness to Adversaries

Abstract: Large batch size training of Neural Networks has been shown to incur accuracy loss when trained with the current methods. The exact underlying reasons for this are still not completely understood. Here, we study large batch size training through the lens of the Hessian operator and robust optimization. In particular, we perform a Hessian based study to analyze exactly how the landscape of the loss function changes when training with large batch size. We compute the true Hessian spectrum, without approximation, by back-propagating the second derivative. Extensive experiments on multiple networks show that saddle-points are not the cause for generalization gap of large batch size training, and the results consistently show that large batch converges to points with noticeably higher Hessian spectrum. Furthermore, we show that robust training allows one to favor flat areas, as points with large Hessian spectrum show poor robustness to adversarial perturbation. We further study this relationship, and provide empirical and theoretical proof that the inner loop for robust training is a saddle-free optimization problem \textit{almost everywhere}. We present detailed experiments with five different network architectures, including a residual network, tested on MNIST, CIFAR-10, and CIFAR-100 datasets. We have open sourced our method which can be accessed at [1].

• The paper shows that larger batch training results in sharper minima using Hessian eigenvalue analysis.
• Empirical results indicate that high Hessian eigenvalues predict increased generalization error and susceptibility to adversarial attacks.
• The study suggests that adjusting batch sizes and regularization techniques can enhance model robustness without compromising training efficiency.

Hessian-based Analysis of Large Batch Training and Robustness to Adversaries

The paper "Hessian-based Analysis of Large Batch Training and Robustness to Adversaries" (1802.08241) addresses the intricate dynamics of optimizing deep neural networks with large batch sizes. It presents a novel approach to analyzing the sensitivity and robustness of large batch training against adversarial attacks using Hessian-based metrics. This research offers significant insights into the stability and generalization capabilities of models trained under these conditions.

Methodology

In this study, the authors employ the Hessian matrix to evaluate the curvature of the loss surface in large batch training regimes. The central thesis is that the curvature, characterized by the eigenvalues of the Hessian, provides crucial information about the model's susceptibility to adversarial perturbations. Large batch training often leads to sharp minima due to its optimization dynamics, which can potentially affect generalization negatively.

The paper leverages empirical measurements of the Hessian's spectrum to dissect the behavior of models trained with varying batch sizes. By focusing on the largest eigenvalue of the Hessian, the researchers discern patterns that correlate with how models respond to adversarial inputs. Their findings suggest that these eigenvalues can be predictive of both the generalization error and the adversarial robustness.

Numerical Results

Through extensive computational experiments, the research demonstrates that models trained with large batches exhibit sharper minima as evidenced by larger Hessian eigenvalues. This sharper curvature often correlates with reduced robustness to adversarial attacks, a conclusion substantiated with numerical simulations on standard benchmark datasets.

Notably, the paper presents a comparative analysis between small and large batch training. The analysis shows a quantifiable difference in the spectrum of Hessian eigenvalues, with large batch sizes consistently impacting models' resilience against adversaries. Furthermore, the study explores how regularization techniques, such as weight decay, affect the curvature and thus the robustness of the models.

Theoretical and Practical Implications

From a theoretical perspective, the findings underscore the importance of considering the Hessian spectrum in understanding the behavior and limitations of optimization methods under large batch regimes. The research challenges the notion that lower generalization errors are solely attributable to sharper minima, suggesting instead that curvature informs adversarial robustness – an insight with potential implications for designing new training protocols and architectures.

Practically, the insights derived from the Hessian analysis could inform more robust training practices. For instance, adjusting batch sizes or incorporating regularization techniques to modulate the Hessian spectrum can enhance the model's adversarial resilience without sacrificing performance gains from large batch efficiency. This could be critical in safety-sensitive applications where robustness is paramount.

Speculations on Future Developments

The study opens avenues for further exploration into fine-grained control over the optimization landscape through curvature tuning. Future developments could integrate Hessian-based metrics directly into the loss function optimization or design novel regularizers targeting specific eigenvalues of the Hessian. This approach could lead to enhanced robustness, providing a balance between computational efficiency and model stability.

Moreover, the application of these findings in conjunction with emerging training paradigms such as federated learning and adaptive batch sizing could provide a more comprehensive framework for robust model deployments in distributed or constrained environments.

Conclusion

The paper presents a compelling investigation into the consequences of large batch training using a Hessian-based approach, highlighting important aspects of model robustness to adversarial strategies. By linking the curvature of the loss surface with practical outcomes, it lays the groundwork for further research into optimizing neural networks for both performance and resilience. These contributions invite continued exploration into adaptive strategies that exploit the diverse effects of training dynamics on model generalization and robustness.