Curvature-Weighted Gradient Diversity (CWGD)

• CWGD is defined as a curvature-aware noise measure that reweights per-sample gradient differences by the inverse square root of the Hessian.
• Theoretical results demonstrate that modulating a cosine schedule with CWGD achieves up to a 20% lower final optimization error on strongly convex quadratics.
• The CWGD-Cosine algorithm employs Hutchinson probes for diagonal Hessian estimation, balancing computational overhead with improved noise adaptation.

Curvature-Weighted Gradient Diversity (CWGD) is a geometry-aware noise measure for mini-batch stochastic gradient descent (SGD) that reweights per-sample gradient diversity by the inverse square root of the Hessian, with the aim of approximating the effective optimization noise rather than treating all parameter directions uniformly (Hamza et al., 29 Jun 2026). In the formulation introduced in "Curvature-Weighted Gradient Diversity: A Noise Measure for Geometry-Adaptive SGD Schedules" (Hamza et al., 29 Jun 2026), CWGD is used to modulate a cosine learning-rate schedule, yielding a theoretically smaller asymptotic optimization error floor on strongly convex quadratic objectives with diagonal Hessians and, in synthetic quadratic experiments, approximately 20% lower final optimization error than standard cosine annealing.

1. Formal definition

Let $f:\mathbb R^d\to\mathbb R$ be twice differentiable with Hessian $H(x)=\nabla^2 f(x)\succ0$. At iterate $x$, a mini-batch $\mathcal B=\{z_i\}_{i=1}^B$ yields per-sample gradients

$$g_i=\nabla f(x;z_i)\in\mathbb R^d,\qquad \bar g=\tfrac1B\sum_{i=1}^B g_i.$$

CWGD is defined by averaging pairwise gradient differences after reweighting by $H(x)^{-1/2}$: $$\mathrm{CWGD}(x,\mathcal B) = \frac{2}{B(B-1)}\sum_{1\le i<j\le B} \bigl\|H(x)^{-1/2}(g_i-g_j)\bigr\|^2.$$

This definition emphasizes directional geometry. Rather than aggregating gradient variability with a single isotropic variance term, it computes diversity after transforming coordinates by the local curvature. In the diagonal case with eigenvalues $\lambda_k$, the paper gives the equivalent expression

$$\mathrm{CWGD} = 2\sum_{k=1}^d \frac{\hat\sigma_k^2}{\lambda_k}, \qquad \hat\sigma_k^2 = \frac{1}{B-1}\sum_{i=1}^B (g_{ik}-\bar g_k)^2,$$

so the coordinate-wise sample variance $\hat\sigma_k^2$ is down-weighted by the local curvature $H(x)=\nabla^2 f(x)\succ0$0 (Hamza et al., 29 Jun 2026).

Under the isotropic noise model

$H(x)=\nabla^2 f(x)\succ0$1

the expectation is

$H(x)=\nabla^2 f(x)\succ0$2

This makes CWGD an explicitly curvature-sensitive proxy for stochasticity.

2. Geometric rationale

The motivation for the $H(x)=\nabla^2 f(x)\succ0$3 weighting is that noise in high-curvature directions has less effect on the iterate because learning rates are already constrained there. In the paper’s description, high-curvature directions correspond to large $H(x)=\nabla^2 f(x)\succ0$4, and gradient steps must therefore be small, with $H(x)=\nabla^2 f(x)\succ0$5. Scaling coordinate differences by $H(x)=\nabla^2 f(x)\succ0$6 causes CWGD to assign less weight to noise in precisely those directions where optimization is already forced to move cautiously (Hamza et al., 29 Jun 2026).

This yields what the paper calls an effective variance proxy,

$H(x)=\nabla^2 f(x)\succ0$7

which satisfies

$H(x)=\nabla^2 f(x)\succ0$8

with strict inequality whenever the spectrum is non-flat. The contrast is with the standard convergence analysis of mini-batch SGD, which uses a single variance term that treats all directions equally. CWGD instead encodes anisotropy directly through the Hessian.

A common misunderstanding would be to interpret CWGD as a generic replacement for all gradient-noise models. The paper does not make that claim. Its formal development is tied to strongly convex quadratic objectives with diagonal Hessians, and the broader relevance is presented as motivation for future extensions rather than an established universal theory.

3. Theoretical guarantees for geometry-adaptive scheduling

The theoretical analysis is carried out under a specific set of assumptions. The objective is a strongly convex quadratic,

$H(x)=\nabla^2 f(x)\succ0$9

with diagonal $x$0 and $x$1. Per-sample noise is isotropic Gaussian,

$x$2

and the batch size satisfies $x$3.

Within this setting, the main theorem states that SGD with a CWGD-modulated cosine schedule uses

$x$4

with the usual cosine envelope denoted by $x$5. For all $x$6,

$x$7

The residual floor is therefore smaller by the exact factor $x$8 relative to $x$9, and at the recommended $\mathcal B=\{z_i\}_{i=1}^B$0 the asymptotic error floor improves by a factor of $\mathcal B=\{z_i\}_{i=1}^B$1 (Hamza et al., 29 Jun 2026).

The paper also identifies an “aspirational gap” between this proven factor and an ideal curvature-adaptive reduction,

$\mathcal B=\{z_i\}_{i=1}^B$2

which grows roughly as $\mathcal B=\{z_i\}_{i=1}^B$3. For $\mathcal B=\{z_i\}_{i=1}^B$4, the paper reports $\mathcal B=\{z_i\}_{i=1}^B$5. This suggests that the theorem captures only part of the geometry-dependent improvement potentially available from a more refined schedule.

4. Algorithmic realization: CWGD-Cosine

The proposed implementation is CWGD-Cosine. Its central practical requirement is an estimate of the diagonal curvature, used to separate curvature from gradient diversity rather than conflating the two. The paper explicitly identifies a degenerate estimator pitfall: estimating $\mathcal B=\{z_i\}_{i=1}^B$6 by normalizing coordinate variances by themselves collapses to the constant $\mathcal B=\{z_i\}_{i=1}^B$7, destroying the signal.

To avoid this, the method uses a Hutchinson estimator for $\mathcal B=\{z_i\}_{i=1}^B$8. With $\mathcal B=\{z_i\}_{i=1}^B$9 Rademacher probes $$g_i=\nabla f(x;z_i)\in\mathbb R^d,\qquad \bar g=\tfrac1B\sum_{i=1}^B g_i.$$0, finite-difference Hessian–vector products are computed and combined as

$$g_i=\nabla f(x;z_i)\in\mathbb R^d,\qquad \bar g=\tfrac1B\sum_{i=1}^B g_i.$$1

For a true quadratic with diagonal $$g_i=\nabla f(x;z_i)\in\mathbb R^d,\qquad \bar g=\tfrac1B\sum_{i=1}^B g_i.$$2, Proposition 5.2 states that this estimator recovers $$g_i=\nabla f(x;z_i)\in\mathbb R^d,\qquad \bar g=\tfrac1B\sum_{i=1}^B g_i.$$3 exactly for any $$g_i=\nabla f(x;z_i)\in\mathbb R^d,\qquad \bar g=\tfrac1B\sum_{i=1}^B g_i.$$4 (Hamza et al., 29 Jun 2026).

The pseudocode given for CWGD-Cosine consists of four steps:

1. At initialization, compute $$g_i=\nabla f(x;z_i)\in\mathbb R^d,\qquad \bar g=\tfrac1B\sum_{i=1}^B g_i.$$5 via $$g_i=\nabla f(x;z_i)\in\mathbb R^d,\qquad \bar g=\tfrac1B\sum_{i=1}^B g_i.$$6 Hutchinson probes.
2. Sample the first mini-batch and compute $$g_i=\nabla f(x;z_i)\in\mathbb R^d,\qquad \bar g=\tfrac1B\sum_{i=1}^B g_i.$$7.
3. For $$g_i=\nabla f(x;z_i)\in\mathbb R^d,\qquad \bar g=\tfrac1B\sum_{i=1}^B g_i.$$8: a. sample mini-batch $$g_i=\nabla f(x;z_i)\in\mathbb R^d,\qquad \bar g=\tfrac1B\sum_{i=1}^B g_i.$$9 and compute per-sample gradients $H(x)^{-1/2}$0; b. compute

$H(x)^{-1/2}$1

c. set

$H(x)^{-1/2}$2

d. update

$H(x)^{-1/2}$3

The computational overhead is reported as negligible in the quadratic setting when the curvature estimate is computed only once. Each Hutchinson refresh costs $H(x)^{-1/2}$4 extra gradients; if performed only at initialization, the overhead is $H(x)^{-1/2}$5, and the example given is 20 extra evaluations over 4000 steps, or 0.5%. Periodic refreshes every $H(x)^{-1/2}$6 steps incur amortized cost $H(x)^{-1/2}$7 extra gradients per step; the example $H(x)^{-1/2}$8 gives approximately 40% overhead. By contrast, each CWGD computation is $H(x)^{-1/2}$9 and is described as negligible by comparison.

5. Empirical behavior on synthetic quadratics

All experiments in the paper are conducted on synthetic strongly-convex quadratics with $$\mathrm{CWGD}(x,\mathcal B) = \frac{2}{B(B-1)}\sum_{1\le i<j\le B} \bigl\|H(x)^{-1/2}(g_i-g_j)\bigr\|^2.$$0, eigenvalues log-spaced in $$\mathrm{CWGD}(x,\mathcal B) = \frac{2}{B(B-1)}\sum_{1\le i<j\le B} \bigl\|H(x)^{-1/2}(g_i-g_j)\bigr\|^2.$$1, and noise level $$\mathrm{CWGD}(x,\mathcal B) = \frac{2}{B(B-1)}\sum_{1\le i<j\le B} \bigl\|H(x)^{-1/2}(g_i-g_j)\bigr\|^2.$$2. Final suboptimality $$\mathrm{CWGD}(x,\mathcal B) = \frac{2}{B(B-1)}\sum_{1\le i<j\le B} \bigl\|H(x)^{-1/2}(g_i-g_j)\bigr\|^2.$$3 is averaged over 20 seeds (Hamza et al., 29 Jun 2026).

In the condition-number sweep, with $$\mathrm{CWGD}(x,\mathcal B) = \frac{2}{B(B-1)}\sum_{1\le i<j\le B} \bigl\|H(x)^{-1/2}(g_i-g_j)\bigr\|^2.$$4, $$\mathrm{CWGD}(x,\mathcal B) = \frac{2}{B(B-1)}\sum_{1\le i<j\le B} \bigl\|H(x)^{-1/2}(g_i-g_j)\bigr\|^2.$$5, $$\mathrm{CWGD}(x,\mathcal B) = \frac{2}{B(B-1)}\sum_{1\le i<j\le B} \bigl\|H(x)^{-1/2}(g_i-g_j)\bigr\|^2.$$6, and $$\mathrm{CWGD}(x,\mathcal B) = \frac{2}{B(B-1)}\sum_{1\le i<j\le B} \bigl\|H(x)^{-1/2}(g_i-g_j)\bigr\|^2.$$7, CWGD-Cosine outperforms plain Cosine:

$$\mathrm{CWGD}(x,\mathcal B) = \frac{2}{B(B-1)}\sum_{1\le i<j\le B} \bigl\|H(x)^{-1/2}(g_i-g_j)\bigr\|^2.$$8	CWGD-Cosine	Plain Cosine
5	$$\mathrm{CWGD}(x,\mathcal B) = \frac{2}{B(B-1)}\sum_{1\le i<j\le B} \bigl\|H(x)^{-1/2}(g_i-g_j)\bigr\|^2.$$9	$\lambda_k$0
20	$\lambda_k$1	$\lambda_k$2
50	$\lambda_k$3	$\lambda_k$4

The corresponding improvements are reported as $\lambda_k$5, $\lambda_k$6, and $\lambda_k$7, with all differences satisfying $\lambda_k$8. The paper summarizes the broader result as a consistent approximately 20% lower final optimization error than standard cosine annealing.

The batch-size ablation with $\lambda_k$9 at $$\mathrm{CWGD} = 2\sum_{k=1}^d \frac{\hat\sigma_k^2}{\lambda_k}, \qquad \hat\sigma_k^2 = \frac{1}{B-1}\sum_{i=1}^B (g_{ik}-\bar g_k)^2,$$0 shows a stable $$\mathrm{CWGD} = 2\sum_{k=1}^d \frac{\hat\sigma_k^2}{\lambda_k}, \qquad \hat\sigma_k^2 = \frac{1}{B-1}\sum_{i=1}^B (g_{ik}-\bar g_k)^2,$$1 gap in final loss, matching the theory that the improvement factor is independent of $$\mathrm{CWGD} = 2\sum_{k=1}^d \frac{\hat\sigma_k^2}{\lambda_k}, \qquad \hat\sigma_k^2 = \frac{1}{B-1}\sum_{i=1}^B (g_{ik}-\bar g_k)^2,$$2. In the noise-structure ablation, aligned noise proportional to $$\mathrm{CWGD} = 2\sum_{k=1}^d \frac{\hat\sigma_k^2}{\lambda_k}, \qquad \hat\sigma_k^2 = \frac{1}{B-1}\sum_{i=1}^B (g_{ik}-\bar g_k)^2,$$3 with $$\mathrm{CWGD} = 2\sum_{k=1}^d \frac{\hat\sigma_k^2}{\lambda_k}, \qquad \hat\sigma_k^2 = \frac{1}{B-1}\sum_{i=1}^B (g_{ik}-\bar g_k)^2,$$4 yields approximately 23–24% improvement, described as slightly larger than in the isotropic case because CWGD down-weights the high-curvature directions that are also the noisy ones.

The convergence plots show two effects during the cosine decay phase: CWGD-Cosine reaches a 20% lower tail and does so approximately 20% faster in iteration count. A compute-normalised comparison, matching total gradient evaluations—for example, Cosine for 4020 steps versus CWGD-Cosine for 4000 steps with 20 probes—still shows an advantage of about 20% with $$\mathrm{CWGD} = 2\sum_{k=1}^d \frac{\hat\sigma_k^2}{\lambda_k}, \qquad \hat\sigma_k^2 = \frac{1}{B-1}\sum_{i=1}^B (g_{ik}-\bar g_k)^2,$$5. The paper interprets this as evidence that the gain is structural rather than a by-product of additional computation.

6. Limitations, failure modes, and open extensions

The paper is explicit about the limits of the current analysis and implementation. First, the discarded self-normalizing heuristic is degenerate: $$\mathrm{CWGD} = 2\sum_{k=1}^d \frac{\hat\sigma_k^2}{\lambda_k}, \qquad \hat\sigma_k^2 = \frac{1}{B-1}\sum_{i=1}^B (g_{ik}-\bar g_k)^2,$$6 The proposed correction is to separate curvature, represented by $$\mathrm{CWGD} = 2\sum_{k=1}^d \frac{\hat\sigma_k^2}{\lambda_k}, \qquad \hat\sigma_k^2 = \frac{1}{B-1}\sum_{i=1}^B (g_{ik}-\bar g_k)^2,$$7, from diversity, represented by $$\mathrm{CWGD} = 2\sum_{k=1}^d \frac{\hat\sigma_k^2}{\lambda_k}, \qquad \hat\sigma_k^2 = \frac{1}{B-1}\sum_{i=1}^B (g_{ik}-\bar g_k)^2,$$8. This is a methodological clarification rather than a minor implementation detail, because the degenerate form eliminates geometry dependence altogether (Hamza et al., 29 Jun 2026).

Second, the theoretical scope is restricted to diagonal strongly-convex quadratics with static Hessian. The paper notes that real losses are nonconvex, non-diagonal, and evolving. Extending the analysis to non-diagonal $$\mathrm{CWGD} = 2\sum_{k=1}^d \frac{\hat\sigma_k^2}{\lambda_k}, \qquad \hat\sigma_k^2 = \frac{1}{B-1}\sum_{i=1}^B (g_{ik}-\bar g_k)^2,$$9 would require a full-matrix preconditioner or bounds on off-diagonal perturbations, while nonconvex losses would require assumptions such as a PL-condition or local convexity.

Third, Hessian staleness is identified as the principal practical obstacle outside the quadratic regime. In nonconvex training, the diagonal of $\hat\sigma_k^2$0 can shift rapidly, so a one-off Hutchinson estimate at initialization becomes inaccurate. The paper reports that under such circumstances CWGD-Cosine can underperform standard Cosine by 2–14% in small MLPs. A natural fix proposed in the paper is an online curvature proxy, for example an EMA of squared gradients à la Adam/AdaHessian, though its theory and bias are said to require careful study to avoid degeneracy.

Fourth, empirical validation remains small-scale. All experiments use $\hat\sigma_k^2$1, and the paper states that larger-scale tests with $\hat\sigma_k^2$2 may amplify off-diagonal bias in the estimator, with Proposition 5.2(ii) indicating error growth like $\hat\sigma_k^2$3. This suggests that scalability and estimator robustness are open questions rather than resolved properties.

Taken together, these limitations define the current status of CWGD. It is established in the paper as a principled geometry-aware measure of optimization noise, rigorously beneficial on a restricted quadratic class, and empirically effective across condition numbers, batch sizes, and noise regimes in that setting. Its extension to realistic nonconvex optimization depends on lightweight and reliable online curvature estimation and on theoretical analysis beyond static diagonal Hessians.