Improved Performance of Stochastic Gradients with Gaussian Smoothing

Abstract: This paper formalizes and analyzes Gaussian smoothing applied to two prominent optimization methods: Stochastic Gradient Descent (GSmoothSGD) and Adam (GSmoothAdam) in deep learning. By attenuating small fluctuations, Gaussian smoothing lowers the risk of gradient-based algorithms converging to poor local minima. These methods simplify the loss landscape while boosting robustness to noise and improving generalization, helping base algorithms converge more effectively to global minima. Existing approaches often rely on zero-order approximations, which increase training time due to inefficiencies in automatic differentiation. To address this, we derive Gaussian-smoothed loss functions for feedforward and convolutional networks, improving computational efficiency. Numerical experiments demonstrate the enhanced performance of our smoothing algorithms over unsmoothed counterparts, confirming the theoretical benefits.