AdaGrad–Diagonal Optimization

• AdaGrad–Diagonal is an adaptive optimization method that leverages a diagonal preconditioner to adjust stepsizes based on historical squared gradients.
• It scales learning rates per coordinate to effectively address anisotropic curvature and gradient noise in large-scale machine learning tasks.
• The method offers convergence guarantees in both convex and nonconvex settings, demonstrating improved stability and efficiency compared to standard SGD.

AdaGrad–Diagonal is an adaptive first-order stochastic optimization method characterized by its coordinate-wise stepsize adaptation via a diagonal preconditioning scheme. It operates by adjusting each parameter’s learning rate inversely to the square root of the sum of historical squared gradients, thereby naturally decaying stepsizes and providing robustness to the geometry of the objective landscape. This variant is widely used in large-scale machine learning, especially where problem curvature or gradient noise exhibits significant anisotropy, and has been the subject of recent advances in convergence theory under both convex and nonconvex regimes (Jin et al., 2022, Liu et al., 2024, Gratton et al., 19 Apr 2026, Wang et al., 2023).

1. Algorithmic Structure and Update Rule

AdaGrad–Diagonal maintains a separate accumulator for each coordinate, forming a diagonal preconditioner that scales the step taken at each iteration. Given a stochastic gradient $g_t = \nabla f(x_t, \xi_t) \in \mathbb{R}^d$, the historical gradient accumulator is updated as

$h_t = h_{t-1} + g_t \odot g_t,\quad h_0 = 0,$

where $\odot$ denotes the elementwise (Hadamard) product. The algorithm applies the following update rule:

$$x_{t+1} = x_t - \eta\, (h_t + \delta)^{-1/2} \odot g_t,$$

or equivalently, using matrix notation,

$$G_t = \operatorname{diag}(h_t + \delta\,\mathbf{1}), \quad x_{t+1} = x_t - \eta\, G_t^{-1/2} g_t,$$

where $\eta > 0$ is the global stepsize and $\delta > 0$ is a small smoothing term to prevent division by zero (Jin et al., 2022). Recent works follow this form, optionally initializing $h_0$ or $D_{-1}$ with a small positive regularizer rather than zero (Gratton et al., 19 Apr 2026).

A typical pseudocode outline is: $$G_t = \operatorname{diag}(h_t + \delta\,\mathbf{1}), \quad x_{t+1} = x_t - \eta\, G_t^{-1/2} g_t,$$8 This update shrinks the effective stepsize fastest in directions with persistently large gradients, aligning optimization progress with the local landscape.

2. Coordinate-wise Preconditioning and Anisotropy

Unlike global stepsize algorithms (e.g., standard SGD), AdaGrad–Diagonal utilizes a coordinate-wise preconditioner, enabling it to adapt to diverse smoothness and noise profiles across variables. This is particularly valuable under anisotropic smoothness, formalized by a coordinate-wise Lipschitz bound,

$$f(w') \le f(w) + \nabla f(w)^\top (w'-w) + \frac{1}{2} \|w'-w\|_L^2, \qquad L = (L_1, ..., L_d),$$

where $h_t = h_{t-1} + g_t \odot g_t,\quad h_0 = 0,$0 (Liu et al., 2024). When objective or noise has strong coordinate dependence (i.e., some $h_t = h_{t-1} + g_t \odot g_t,\quad h_0 = 0,$1 or gradient variances $h_t = h_{t-1} + g_t \odot g_t,\quad h_0 = 0,$2 significantly larger than others), AdaGrad–Diagonal can exploit this structure to improve convergence guarantees, yielding better dimension dependence than uniform-stepsize methods.

Empirical validation confirms that on high-dimensional sparse problems, AdaGrad–Diagonal’s coordinate adaptation provides substantial stability and faster loss reduction, especially as batch size increases (Liu et al., 2024).

3. Convergence Theory

Nonconvex Setting

Recent advances establish almost sure convergence of AdaGrad–Diagonal iterates $h_t = h_{t-1} + g_t \odot g_t,\quad h_0 = 0,$3 to a connected component $h_t = h_{t-1} + g_t \odot g_t,\quad h_0 = 0,$4 of stationary points for $h_t = h_{t-1} + g_t \odot g_t,\quad h_0 = 0,$5 nonnegative $h_t = h_{t-1} + g_t \odot g_t,\quad h_0 = 0,$6-Lipschitz objectives, under standard stochastic assumptions:

• Unbiased noise: $h_t = h_{t-1} + g_t \odot g_t,\quad h_0 = 0,$7.
• Growth-controlled noise: $h_t = h_{t-1} + g_t \odot g_t,\quad h_0 = 0,$8 (Jin et al., 2022).

The main theorem asserts:

$h_t = h_{t-1} + g_t \odot g_t,\quad h_0 = 0,$9

where $\odot$0 is the terminal connected component (Jin et al., 2022). The proof employs a Lyapunov potential of the form $\odot$1, demonstrating monotonic decay of $\odot$2 and ultimately establishing subsequential then full convergence by controlling martingale noise.

For more general nonconvex settings and under affine variance conditions, diagonal AdaGrad attains a high-probability bound on the minimum gradient norm:

$\odot$3

(with explicit dependence on variance parameters and algorithmic constants), requiring only an $\odot$4 iteration count for $\odot$5 (Wang et al., 2023). Under certain over-parameterized regimes (zero bias), an $\odot$6 min-norm convergence can be obtained.

Convex and Anisotropic Cases

For convex objectives under anisotropic smoothness and coordinate-wise noise,

$\odot$7

where $\odot$8 is the $\odot$9-diameter of the domain, and $$x_{t+1} = x_t - \eta\, (h_t + \delta)^{-1/2} \odot g_t,$$0, $$x_{t+1} = x_t - \eta\, (h_t + \delta)^{-1/2} \odot g_t,$$1 (Liu et al., 2024). Compared to SGD’s $$x_{t+1} = x_t - \eta\, (h_t + \delta)^{-1/2} \odot g_t,$$2-based rate, this demonstrates potential $$x_{t+1} = x_t - \eta\, (h_t + \delta)^{-1/2} \odot g_t,$$3 or $$x_{t+1} = x_t - \eta\, (h_t + \delta)^{-1/2} \odot g_t,$$4-fold gains in dimensions with sparse or strongly anisotropic noise or curvature.

A unified convergence theory for nonconvex settings places AdaGrad–Diagonal in a broad class of adaptive preconditioned methods, showing for unbiased gradient oracles:

$$x_{t+1} = x_t - \eta\, (h_t + \delta)^{-1/2} \odot g_t,$$5

(Gratton et al., 19 Apr 2026). This rate matches SGD (and more expensive full-matrix AdaGrad) up to constant/logarithmic factors but holds under milder assumptions and with full coordinatewise adaptation.

4. Theoretical Ingredients and Proof Techniques

Fundamental proof strategies for AdaGrad–Diagonal convergence include:

• Lyapunov-Controlled Descent: Using potential functions such as $$x_{t+1} = x_t - \eta\, (h_t + \delta)^{-1/2} \odot g_t,$$6 or auxiliary quantities like $$x_{t+1} = x_t - \eta\, (h_t + \delta)^{-1/2} \odot g_t,$$7 to offset correlations between the numerator (gradient) and denominator (accumulator) arising in each step (Jin et al., 2022, Wang et al., 2023).
• Drift-Plus-Noise Decomposition: Split descent into high-noise and low-noise regimes to bound martingale errors, using telescoping series arguments to aggregate local improvements.
• Structural Trace Inequalities: For block-adaptive or coordinatewise schemes, leveraging operator-monotone functions and trace bounds on the evolution of the diagonal (or full) preconditioning matrix (Gratton et al., 19 Apr 2026).
• Potential Telescoping: Utilizing auxiliary functions that guarantee supermartingale-type descent over iterations, enabling convergence in expectation and sometimes high probability.

Under coordinated (L₀, L₁)-smoothness, convergence requires careful stepsize tuning, with explicit thresholds on $$x_{t+1} = x_t - \eta\, (h_t + \delta)^{-1/2} \odot g_t,$$8 determined by local gradient growth parameters (Wang et al., 2023). Failure to respect these thresholds leads to divergence, as demonstrated by explicit construction.

5. Comparison with Related Adaptive Methods

Algorithm	Preconditioning Structure	Complexity Order (min grad norm)	Noted Features
AdaGrad–Diagonal	Diagonal accumulation	$$x_{t+1} = x_t - \eta\, (h_t + \delta)^{-1/2} \odot g_t,$$9, logs	Coordinatewise adaptation, efficient, no bounded gradient needed
AdaNorm	Scalar accumulation	$$G_t = \operatorname{diag}(h_t + \delta\,\mathbf{1}), \quad x_{t+1} = x_t - \eta\, G_t^{-1/2} g_t,$$0, logs	Simpler, isotropic, no coordinate adaptation
Full AdaGrad	Full-matrix accumulation	$$G_t = \operatorname{diag}(h_t + \delta\,\mathbf{1}), \quad x_{t+1} = x_t - \eta\, G_t^{-1/2} g_t,$$1, logs	High memory, matches diagonal in global order
SGD	None (global stepsize)	$$G_t = \operatorname{diag}(h_t + \delta\,\mathbf{1}), \quad x_{t+1} = x_t - \eta\, G_t^{-1/2} g_t,$$2	No adaptation, less stable on anisotropic/noisy problems

Diagonal AdaGrad provides nearly the same theoretical guarantees as full-matrix AdaGrad and AdaNorm in terms of asymptotic complexity, with practical gains from its low computational cost and capacity to track problem anisotropy. Unlike SGD, AdaGrad–Diagonal automatically adapts stepsizes per-coordinate without the need for intricate learning rate schedules (Gratton et al., 19 Apr 2026, Jin et al., 2022).

6. Practical Considerations and Limitations

• Initialization: Robustness to initialization of the diagonal accumulator is provided by the stabilizing regularizer ($$G_t = \operatorname{diag}(h_t + \delta\,\mathbf{1}), \quad x_{t+1} = x_t - \eta\, G_t^{-1/2} g_t,$$3 or $$G_t = \operatorname{diag}(h_t + \delta\,\mathbf{1}), \quad x_{t+1} = x_t - \eta\, G_t^{-1/2} g_t,$$4), which also prevents division by zero.
• Stepsize Selection: Convergence analyses often assume a fixed stepsize $$G_t = \operatorname{diag}(h_t + \delta\,\mathbf{1}), \quad x_{t+1} = x_t - \eta\, G_t^{-1/2} g_t,$$5. In practice, $$G_t = \operatorname{diag}(h_t + \delta\,\mathbf{1}), \quad x_{t+1} = x_t - \eta\, G_t^{-1/2} g_t,$$6 should be chosen sufficiently small, especially under non-uniform smoothness, as explicit upper bounds on $$G_t = \operatorname{diag}(h_t + \delta\,\mathbf{1}), \quad x_{t+1} = x_t - \eta\, G_t^{-1/2} g_t,$$7 are necessary to avoid divergence (Wang et al., 2023).
• High-Dimensional and Sparse Regimes: Practical evidence indicates AdaGrad–Diagonal is especially effective for sparse models or settings with high curvature anisotropy, both in classic convex problems and deep neural architectures (Liu et al., 2024).
• Limitations: Most convergence guarantees are asymptotic or “order”-level (i.e., without tight explicit nonasymptotic constants). High-probability rates and guarantees under general matrix-valued (non-diagonal) preconditioning remain only partially understood (Jin et al., 2022, Wang et al., 2023). The method still requires bounded second moments of the stochastic gradient for theoretical guarantees.

A plausible implication is that AdaGrad–Diagonal balances per-coordinate adaptivity and computational efficiency, offering fundamental theoretical and practical advantages for large-scale and anisotropic optimization tasks, with ongoing research focused on sharpening finite-time, high-probability guarantees and generalizing to broader stochastic preconditioning frameworks.