Gradient Smoothing: Methods & Applications

• Gradient smoothing is a set of techniques that replace raw gradients with averaged or convolved surrogates to suppress noise while preserving significant structural information.
• In optimization, methods such as Laplacian and Gaussian smoothing reduce variance and improve convergence rates by stabilizing descent directions and update trajectories.
• In image processing and model explanation, gradient smoothing preserves essential edges and clear attribution maps by mitigating high-frequency fluctuations.

Searching arXiv for the cited papers on gradient smoothing and related methods. Gradient smoothing denotes a family of techniques that replace a raw gradient, a gradient-derived update, or a gradient-controlled penalty with a smoother surrogate. The term is not monosemous. In optimization, it may mean preconditioning or averaging descent directions, convolving the loss with a Gaussian or data-driven noise kernel, or introducing smooth approximations of nonsmooth objectives; in explainability, it denotes averaging attribution gradients over perturbations; in image processing, it denotes reducing pixel-wise gradients while preserving salient edges and contours (Osher et al., 2018, Smilkov et al., 2017, Tran et al., 19 Jan 2026). Across these settings, the recurrent aim is to suppress high-frequency noise, small oscillations, or unstable local variation without discarding the large-scale structure that carries semantic, geometric, or task-specific significance.

1. Terminological scope and recurring mathematical pattern

A central source of ambiguity is that different communities smooth different objects. In inverse problems and machine learning optimization, the smoothed quantity is often the descent direction itself. In Gaussian-smoothing methods, the smoothed quantity is the objective or its gradient field. In composite optimization, smoothing is applied to a nonsmooth term so that first-order methods become applicable. In gradient-based explanation, the smoothed object is the saliency field. In edge-preserving image processing, the phrase can refer to suppressing image gradients associated with texture or noise while retaining strong discontinuities (Nayak, 2019, Starnes et al., 2023, Zhou et al., 2024, Wang et al., 2023).

This terminological diversity reflects a common mathematical pattern: a raw local signal is treated as too brittle, too noisy, or too nonsmooth, and is replaced by an averaged, regularized, or structurally coupled surrogate. The mechanism may be an inverse Laplacian, a discrete Laplacian, a Gaussian convolution, a Moreau envelope, Monte Carlo mollification, local depth-wise averaging, or an auxiliary-variable splitting scheme (Nayak, 2019, Osher et al., 2018, Starnes et al., 2024, Argyriou et al., 2014).

A common misconception is that gradient smoothing is synonymous with adding noise. The literature shows a broader picture. SmoothGrad averages gradients over noisy inputs, but Laplacian Smoothing Gradient Descent multiplies the gradient by the inverse of a positive definite matrix derived from a discrete Laplacian, and Depth-wise Gradient Augmentation smooths optimizer updates across layers rather than across perturbations of the input or objective (Smilkov et al., 2017, Osher et al., 2018, Meng et al., 29 Jun 2026). This suggests that “smoothing” is best understood as a structural operation on a gradient-bearing object, not as a single stochastic recipe.

2. Smoothing the descent direction and update trajectory

In iterative regularization for inverse problems, gradient smoothing is implemented by replacing the standard adjoint-residual descent direction with a smoother one. The usual update is

$$\psi_{m+1}^\delta = \psi_m^\delta + \tau_m \xi_m^\delta,$$

with the standard gradient direction

$\xi_m^\delta = -T^*(T\psi_m^\delta - g_\delta).$

The smoothed variant instead uses

$$\xi_m^\delta = T^*(I-\Delta^{-1})(g_\delta - T\psi_m^\delta),$$

where $-\Delta^{-1}$ is defined through a Poisson-type boundary value problem. The variational interpretation introduces a functional

$$G(\psi)=\|T\psi-g_\delta\|^2+\|u_\psi' - u_\delta'\|^2,$$

so the method regularizes not only data misfit but also mismatch in derivatives of smoothed potentials. The reported effect is improved robustness to noise, lower relative error, and reduced semi-convergence in problems such as deblurring and tomography (Nayak, 2019).

Laplacian Smoothing Gradient Descent applies a related principle directly to optimization updates. Standard SGD is replaced by

$$x^{k+1}=x^k-\eta_k\,A_\sigma^{-1}\nabla f_{i_k}(x^k),$$

with $A_\sigma = I-\sigma\Delta_1$, where $\Delta_1$ is a periodic discrete Laplacian. The smoothed gradient $g_\sigma=A_\sigma^{-1}g$ preserves the mean, decreases the largest component, increases the smallest component, and can be computed efficiently by FFT. The paper states that these surrogates reduce variance, allow larger step sizes, and improve generalization accuracy, while the associated Hamilton–Jacobi interpretation characterizes the implicit version as descent on a “more convex” surrogate with the same global minima (Osher et al., 2018).

A more recent reformulation smooths optimizer updates across network depth rather than across parameter coordinates. In “Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization,” the base optimizer produces block-wise updates $u_1,\dots,u_L$, and a depth-wise operator $\xi_m^\delta = -T^*(T\psi_m^\delta - g_\delta).$0 transforms them into $\xi_m^\delta = -T^*(T\psi_m^\delta - g_\delta).$1. The paper’s Window Smoothing operator replaces each interior layer update by

$\xi_m^\delta = -T^*(T\psi_m^\delta - g_\delta).$2

with analogous boundary rules for the first and last layers. The method is presented as a structured depth-wise preconditioner compatible with SGD, Adam, and Muon, and the paper reports gains in language-model pretraining, RL post-training, diffusion modeling, and Vision Transformer classification, together with reduced microbatch gradient variance and more aligned depth-wise representations (Meng et al., 29 Jun 2026).

In LLM unlearning, an application-specific variant appears as Smoothed Gradient Ascent. Plain Gradient Ascent uses only the forget gradient and is described as unstable. SGA mixes that direction with gradients from multiple constructed normal samples using a smoothing rate $\xi_m^\delta = -T^*(T\psi_m^\delta - g_\delta).$3, yielding an update proportional to

$\xi_m^\delta = -T^*(T\psi_m^\delta - g_\delta).$4

The paper derives an “optimal” smoothing rate

$\xi_m^\delta = -T^*(T\psi_m^\delta - g_\delta).$5

and reports that SGA consistently outperforms original Gradient Ascent across all metrics on TOFU, Harry Potter, and MUSE-NEWS, while reducing catastrophic collapse (Pang et al., 25 Oct 2025).

3. Smoothing the loss landscape and constructing nonlocal gradients

Another major line of work smooths the objective rather than the update. Gradient Noise Convolution interprets SGD as gradient descent on a noise-convolved loss and extends that perspective to large-batch distributed training. In DP-SGD with $\xi_m^\delta = -T^*(T\psi_m^\delta - g_\delta).$6 workers, GNC evaluates local gradients at worker-specific perturbed weights and averages them, effectively optimizing a smoothed objective

$\xi_m^\delta = -T^*(T\psi_m^\delta - g_\delta).$7

The paper emphasizes that the perturbations are not isotropic random noise but estimates of actual gradient noise, which are anisotropic and aligned with sharp directions of the landscape. It reports improved loss stability, gradient predictiveness, $\xi_m^\delta = -T^*(T\psi_m^\delta - g_\delta).$8-smoothness, and a best ImageNet top-1 accuracy of $\xi_m^\delta = -T^*(T\psi_m^\delta - g_\delta).$9 at batch size $$\xi_m^\delta = T^*(I-\Delta^{-1})(g_\delta - T\psi_m^\delta),$$0 under its benchmark settings (Haruki et al., 2019).

Gaussian smoothing methods define a smoothed objective explicitly. In GSmoothGD,

$$\xi_m^\delta = T^*(I-\Delta^{-1})(g_\delta - T\psi_m^\delta),$$1

and the descent direction is the nonlocal gradient $$\xi_m^\delta = T^*(I-\Delta^{-1})(g_\delta - T\psi_m^\delta),$$2, represented by an expectation over perturbed function values. The paper’s rationale is that Gaussian averaging suppresses high-frequency fluctuations and shallow local minima while retaining broad landscape structure. It gives convex and nonconvex convergence estimates, a Monte Carlo approximation of the nonlocal gradient, and continuation strategies for updating $$\xi_m^\delta = T^*(I-\Delta^{-1})(g_\delta - T\psi_m^\delta),$$3 so that optimization can move from heavily smoothed to less smoothed regimes (Starnes et al., 2023).

Anisotropic Gaussian Smoothing generalizes the scalar smoothing radius to a full symmetric invertible matrix $$\xi_m^\delta = T^*(I-\Delta^{-1})(g_\delta - T\psi_m^\delta),$$4. The smoothed objective is

$$\xi_m^\delta = T^*(I-\Delta^{-1})(g_\delta - T\psi_m^\delta),$$5

with a corresponding nonlocal gradient that averages function values directionally. The paper argues that this directional control allows stronger smoothing where the loss is rough and weaker smoothing elsewhere, and extends the framework to AGS-GD, AGS-SGD, and AGS-Adam. In stochastic settings, the theory states convergence to a noisy ball whose size is determined by the smoothing parameters (Starnes et al., 2024).

Directional Gaussian smoothing in black-box optimization makes the nonlocality even more explicit. AdaDGS computes one-dimensional Gaussian-smoothed directional derivatives along an orthonormal basis, assembles them into a DGS gradient, then performs line search along that direction. The learned step size updates the smoothing radius via

$$\xi_m^\delta = T^*(I-\Delta^{-1})(g_\delta - T\psi_m^\delta),$$6

so smoothing scale and search scale co-evolve. The paper presents this as a way to avoid fine tuning of both learning rate and smoothing radius in highly multimodal landscapes (Tran et al., 2020).

4. Smoothing as a remedy for nonsmooth and composite objectives

In convex and stochastic optimization, gradient smoothing often means replacing a nonsmooth term by a differentiable approximation whose smoothing parameter decreases to zero. A generic formulation writes the original problem as

$$\xi_m^\delta = T^*(I-\Delta^{-1})(g_\delta - T\psi_m^\delta),$$7

with $$\xi_m^\delta = T^*(I-\Delta^{-1})(g_\delta - T\psi_m^\delta),$$8 convex but possibly nondifferentiable, and introduces a smooth approximation $$\xi_m^\delta = T^*(I-\Delta^{-1})(g_\delta - T\psi_m^\delta),$$9 satisfying

$-\Delta^{-1}$0

The resulting smoothing gradient flow and its forward-Euler discretization use the time-varying smooth objective $-\Delta^{-1}$1, with stepsize

$-\Delta^{-1}$2

The convergence analysis in continuous and discrete time gives $-\Delta^{-1}$3 in the strongly convex case, $-\Delta^{-1}$4 in the convex case, and, for $-\Delta^{-1}$5, a best discrete rate of $-\Delta^{-1}$6 at $-\Delta^{-1}$7 (Toyoda et al., 2023).

A related approach appears in stochastic compositional optimization with a nonsmooth outer function. The smoothing stochastic gradient method replaces the outer function $-\Delta^{-1}$8 by a continuously differentiable smoothing function $-\Delta^{-1}$9, updates

$$G(\psi)=\|T\psi-g_\delta\|^2+\|u_\psi' - u_\delta'\|^2,$$0

and gradually decreases $$G(\psi)=\|T\psi-g_\delta\|^2+\|u_\psi' - u_\delta'\|^2,$$1. Under unbiasedness, bounded second moments of order $$G(\psi)=\|T\psi-g_\delta\|^2+\|u_\psi' - u_\delta'\|^2,$$2, and approximation error $$G(\psi)=\|T\psi-g_\delta\|^2+\|u_\psi' - u_\delta'\|^2,$$3, the paper proves a convex rate of $$G(\psi)=\|T\psi-g_\delta\|^2+\|u_\psi' - u_\delta'\|^2,$$4 in the number of stochastic gradient evaluations, with strongly convex rates approaching $$G(\psi)=\|T\psi-g_\delta\|^2+\|u_\psi' - u_\delta'\|^2,$$5 (Giovannelli et al., 15 Jul 2025).

Nesterov-style smoothing of structured penalties provides another influential template. In “Smoothing Proximal Gradient Method for General Structured Sparse Learning,” a structured penalty is written as

$$G(\psi)=\|T\psi-g_\delta\|^2+\|u_\psi' - u_\delta'\|^2,$$6

and smoothed as

$$G(\psi)=\|T\psi-g_\delta\|^2+\|u_\psi' - u_\delta'\|^2,$$7

The resulting smooth term has an explicit gradient $$G(\psi)=\|T\psi-g_\delta\|^2+\|u_\psi' - u_\delta'\|^2,$$8, while the $$G(\psi)=\|T\psi-g_\delta\|^2+\|u_\psi' - u_\delta'\|^2,$$9 term remains unsmoothed and is handled by soft-thresholding. The method is positioned as a scalable solver for overlapping group lasso and graph-guided fusion, with faster convergence than subgradient methods and substantially better scalability than interior-point methods (Chen et al., 2012).

Hybrid Conditional Gradient-Smoothing Algorithms combine Moreau-envelope smoothing of a Lipschitz term $$x^{k+1}=x^k-\eta_k\,A_\sigma^{-1}\nabla f_{i_k}(x^k),$$0 with a projection-free conditional gradient step for $$x^{k+1}=x^k-\eta_k\,A_\sigma^{-1}\nabla f_{i_k}(x^k),$$1. The $$x^{k+1}=x^k-\eta_k\,A_\sigma^{-1}\nabla f_{i_k}(x^k),$$2-smoothed term

$$x^{k+1}=x^k-\eta_k\,A_\sigma^{-1}\nabla f_{i_k}(x^k),$$3

yields a smooth composite gradient while preserving the large-scale advantages of conditional gradient methods on bounded domains. This tradeoff is slower in worst-case rate than accelerated proximal methods, but the paper emphasizes lower per-iteration cost in matrix problems where a leading singular vector is much cheaper than a full SVD (Argyriou et al., 2014).

A stochastic counterpart appears in “A Smoothing Stochastic Gradient Method for Composite Optimization,” where the nonsmooth term $$x^{k+1}=x^k-\eta_k\,A_\sigma^{-1}\nabla f_{i_k}(x^k),$$4 is replaced by

$$x^{k+1}=x^k-\eta_k\,A_\sigma^{-1}\nabla f_{i_k}(x^k),$$5

The smoothed algorithm preserves the $$x^{k+1}=x^k-\eta_k\,A_\sigma^{-1}\nabla f_{i_k}(x^k),$$6 stochastic rate while eliminating difficult proximal maps for structured regularizers (Lin et al., 2010).

5. Gradient smoothing in explanation and attribution

In neural-network explanation, the basic object is the input gradient

$$x^{k+1}=x^k-\eta_k\,A_\sigma^{-1}\nabla f_{i_k}(x^k),$$7

or an analogous gradient-based saliency map. SmoothGrad replaces this single-point gradient by an average over noisy copies of the input,

$$x^{k+1}=x^k-\eta_k\,A_\sigma^{-1}\nabla f_{i_k}(x^k),$$8

The method is motivated by the observation that raw sensitivity maps are often noisy because local derivatives fluctuate sharply in piecewise linear networks. The paper reports that visual improvement shows diminishing returns after about $$x^{k+1}=x^k-\eta_k\,A_\sigma^{-1}\nabla f_{i_k}(x^k),$$9 samples, often uses $A_\sigma = I-\sigma\Delta_1$0 samples, and notes that $A_\sigma = I-\sigma\Delta_1$1 noise relative to the image value range often worked well for ImageNet/Inception v3 (Smilkov et al., 2017).

Later work reinterprets SmoothGrad as convolution. “Rethinking the Principle of Gradient Smooth Methods in Model Explanation” shows that, as the number of samples grows,

$A_\sigma = I-\sigma\Delta_1$2

where $A_\sigma = I-\sigma\Delta_1$3 is the Gaussian density induced by the perturbation variance. On this basis, the paper argues that residual noise remains because Gaussian perturbations are sampled over $A_\sigma = I-\sigma\Delta_1$4 while inputs live in a bounded domain $A_\sigma = I-\sigma\Delta_1$5. Using $A_\sigma = I-\sigma\Delta_1$6 ImageNet images with $A_\sigma = I-\sigma\Delta_1$7 and $A_\sigma = I-\sigma\Delta_1$8, it reports $A_\sigma = I-\sigma\Delta_1$9 out-of-bounds probability at the image level and $\Delta_1$0 at the pixel level for SmoothGrad perturbations. AdaptGrad addresses this by assigning a per-dimension variance

$\Delta_1$1

with the paper recommending $\Delta_1$2 as a balance between smoothing and in-bounds sampling (Zhou et al., 2024).

An axiomatic account casts these methods as Monte Carlo mollification. The smoothed gradient is written as a convolution $\Delta_1$3, and the practical estimator is

$\Delta_1$4

Within this framework, SmoothGrad corresponds to Gaussian mollification of the input, NoiseGrad to mollification in parameter space, and FusionGrad to a double mollification over both. The paper further derives Gaussian, Poisson, Hyperbolic, Sigmoid, and Rectangular kernels as alternative Dirac sequences and reports that Gaussian, Poisson, and Rectangular kernels are especially convincing overall across Consistency, Invariance, Localization, and Sparseness metrics (Zhou et al., 2024).

6. Edge-preserving smoothing, image gradients, and direct gradient-profile estimation

In image processing, “gradient smoothing” often refers to suppressing image gradients associated with texture or artifacts while preserving strong edges. “Deep Image Prior with L0 Gradient Regularizer for Image Smoothing” formulates this as

$\Delta_1$5

The method combines Deep Image Prior with an $\Delta_1$6 gradient regularizer, uses ADMM with an auxiliary variable $\Delta_1$7, and solves the difficult $\Delta_1$8-subproblem with the off-the-shelf Region Fusion $\Delta_1$9 gradient minimization solver. The paper reports best average PSNR on the NKS edge-preserving smoothing dataset and best average PSNR with second-best average SSIM for JPEG artifact removal on clip-art images (Tran et al., 19 Jan 2026).

A related critique appears in improved real-time image smoothing based on iterative least squares. That paper argues that raw gradient magnitude is an unreliable discriminator because weak structures and high-contrast details can occupy overlapping gradient ranges. It replaces the gradient in the ILS penalty with a structure-aware weighted gradient $g_\sigma=A_\sigma^{-1}g$0, where $g_\sigma=A_\sigma^{-1}g$1 is derived from an interval gradient and logistic mapping. The reported practical effect is better preservation of weak structures, better removal of high-contrast details, and fewer iterations than original ILS, roughly $g_\sigma=A_\sigma^{-1}g$2–$g_\sigma=A_\sigma^{-1}g$3 instead of around $g_\sigma=A_\sigma^{-1}g$4 (Wang et al., 2023).

Earlier work on bilateral and guided filters uses “gradient smoothing” in the edge-preserving denoising sense. “Conjugate Gradient Acceleration of Non-Linear Smoothing Filters” treats repeated application of bilateral and guided filters as graph-based diffusion and accelerates them by truncated preconditioned conjugate gradient with freezing and restarting for the nonlinear self-guided case. The paper reports about $g_\sigma=A_\sigma^{-1}g$5 reduction in iterations for the self-guided bilateral filter and about $g_\sigma=A_\sigma^{-1}g$6 reduction in iterations for the self-guided guided filter, with outputs described as almost identical to the original iterative filters when properly tuned (Knyazev et al., 2015).

The same underlying principle appears in direct gradient-profile estimation from noisy measurements. Rather than fit the signal and differentiate afterward, “Gradient Profile Estimation Using Exponential Cubic Spline Smoothing in a Bayesian Framework” models the gradient profile itself in gradient space $g_\sigma=A_\sigma^{-1}g$7, integrates it to reconstruct the observed signal in data space $g_\sigma=A_\sigma^{-1}g$8, and performs Bayesian inference with an exponential cubic spline prior structure. On synthetic noisy position data, the paper reports markedly smaller gradient error norms for the direct gradient-space fit than for the fit-then-differentiate alternative, for example $g_\sigma=A_\sigma^{-1}g$9 versus $u_1,\dots,u_L$0 at noise level $u_1,\dots,u_L$1, and $u_1,\dots,u_L$2 versus $u_1,\dots,u_L$3 at noise level $u_1,\dots,u_L$4 (Silva et al., 2019).

Taken together, these image- and signal-oriented formulations show that gradient smoothing can target the gradient field of a signal itself, not merely the gradient used by an optimizer. A plausible implication is that the phrase spans both algorithmic regularization and structural decomposition: in one case smoothing stabilizes updates, in the other it enforces piecewise-constant or edge-aware representations of the underlying data (Tran et al., 19 Jan 2026, Silva et al., 2019).