CWGD-Cosine: Curvature-Aware Cosine Annealing

• CWGD-Cosine is a curvature-aware learning rate schedule that combines cosine annealing with gradient diversity weighted by Hessian curvature.
• It modulates the cosine envelope by scaling the learning rate based on a noise measure that penalizes directions with high curvature, ensuring effective noise control.
• Empirical results on strongly-convex quadratics show about a 20% improvement in final suboptimality, with theoretical guarantees on a reduced asymptotic noise floor.

CWGD-Cosine is a global learning-rate schedule that augments standard cosine annealing with an online scalar noise signal, Curvature-Weighted Gradient Diversity (CWGD). Introduced in "Curvature-Weighted Gradient Diversity: A Noise Measure for Geometry-Adaptive SGD Schedules" (Hamza et al., 29 Jun 2026), it is designed to measure mini-batch gradient variability after whitening by curvature, so that directions with large Hessian eigenvalues contribute less to the noise signal because stochastic gradient descent is already constrained to take small steps there. Within the paper’s supported regime—synthetic strongly-convex quadratics with diagonal or near-diagonal curvature—the method is presented as a geometry-aware alternative to scalar variance heuristics for modulating the amplitude of a cosine learning-rate envelope.

1. Conceptual basis and optimization motivation

The starting point is a criticism of the standard convergence analysis of mini-batch SGD. In that analysis, gradient noise is summarized by a single variance term that treats all parameter directions equally. The paper argues that this is too crude because it ignores the interaction between stochasticity and curvature: in a high-curvature direction with eigenvalue $\lambda_k$, stability already forces $\eta \lesssim 1/L$, so stochastic displacement there is naturally suppressed (Hamza et al., 29 Jun 2026).

This motivates a curvature-weighted notion of effective noise. Rather than measuring raw gradient variability, CWGD rescales per-sample gradient differences by the inverse square root of the Hessian. The consequence is directional asymmetry: a perturbation along eigen-direction $k$ is weighted by $1/\sqrt{\lambda_k}$, and hence contributes $1/\lambda_k$ in squared norm. In this formulation, directions with low curvature dominate the noise proxy because they are the directions in which a fixed global learning rate is least constrained.

The paper further introduces an aspirational curvature-adaptive improvement ratio,

$$\rho(\kappa) \;=\; \frac{d}{\mu\,\operatorname{tr}(H^{-1})},$$

and states that for log-spaced spectra in $[\mu,L]$, $\rho(\kappa)$ scales like $\kappa/\log\kappa$. The reported values are $\rho(\kappa)=2.0,2.5,3.1,3.9$ for $\eta \lesssim 1/L$0. The paper is explicit that this is not its theorem, but an aspirational target showing why Hessian weighting is desirable.

2. Formal definition of CWGD

Let $\eta \lesssim 1/L$1 be twice differentiable, let $\eta \lesssim 1/L$2 be positive-definite, let the mini-batch be $\eta \lesssim 1/L$3, and let per-sample gradients be $\eta \lesssim 1/L$4. Definition 1 / Eq. $\eta \lesssim 1/L$5 defines

$\eta \lesssim 1/L$6

The $\eta \lesssim 1/L$7 factor is the central construction. It makes CWGD a geometry-aware diversity measure rather than a raw dispersion statistic. In the diagonal case $\eta \lesssim 1/L$8, the paper gives the exact simplification

$\eta \lesssim 1/L$9

CWGD is therefore literally a curvature-weighted sum of per-coordinate sample variances.

Under isotropic additive gradient noise,

$k$0

Proposition 1 states

$k$1

This makes explicit that the expected signal depends on the inverse curvature spectrum rather than on a direction-agnostic variance scalar.

The contrast with the usual scalar variance floor is one of the paper’s main interpretive claims. The classic bound

$k$2

depends on the worst curvature $k$3 and ignores how noise is distributed across eigendirections. CWGD is introduced precisely to replace that isotropic view with a curvature-aware proxy for the optimization noise actually felt by a global-step SGD iterate.

3. CWGD-Cosine schedule and estimation machinery

Algorithm 1 combines CWGD with standard cosine annealing through the per-step rule

$k$4

CWGD therefore does not change the cosine phase, period, or floor directly. It multiplicatively modulates the amplitude, or instantaneous scale, of the cosine envelope. When current diversity $k$5 is high relative to the initial reference $k$6, the step size shrinks; when diversity is low, the schedule follows the cosine baseline more closely. The hyperparameter $k$7 controls modulation strength, and $k$8 recovers plain cosine annealing exactly (Hamza et al., 29 Jun 2026).

The algorithmic workflow is stated explicitly. First, estimate $k$9. Second, compute $1/\sqrt{\lambda_k}$0 on the first mini-batch. Third, at each step $1/\sqrt{\lambda_k}$1, compute per-sample gradients $1/\sqrt{\lambda_k}$2, compute $1/\sqrt{\lambda_k}$3, set $1/\sqrt{\lambda_k}$4, and update with the mini-batch mean gradient.

Because the schedule requires curvature weights, the paper uses the estimator

$1/\sqrt{\lambda_k}$5

with stabilizer $1/\sqrt{\lambda_k}$6. The diagonal Hessian entries are estimated by a Hutchinson-style construction with $1/\sqrt{\lambda_k}$7 Rademacher probes $1/\sqrt{\lambda_k}$8: $1/\sqrt{\lambda_k}$9 where $1/\lambda_k$0 is approximated by the finite difference

$1/\lambda_k$1

For quadratic objectives with diagonal $1/\lambda_k$2, this estimator is exact for any $1/\lambda_k$3 and any $1/\lambda_k$4, because $1/\lambda_k$5. For near-diagonal $1/\lambda_k$6, the paper gives

$1/\lambda_k$7

The paper also identifies and corrects a degenerate curvature estimator from an earlier draft: $1/\lambda_k$8 Because this collapses to a constant, it contains no curvature or noise information. The corrected construction separates curvature estimation from mini-batch diversity estimation and combines them only in Eq. (9).

One point remains unresolved in the paper’s presentation. The Algorithm / computational-cost discussion says Hutchinson may be refreshed periodically, quoting $1/\lambda_k$9 and refresh interval $$\rho(\kappa) \;=\; \frac{d}{\mu\,\operatorname{tr}(H^{-1})},$$0 epochs for roughly 40% overhead, whereas the compute-normalized appendix states that in the quadratic setting the Hessian is constant, so the $$\rho(\kappa) \;=\; \frac{d}{\mu\,\operatorname{tr}(H^{-1})},$$1 probes are computed once at initialization, giving only $$\rho(\kappa) \;=\; \frac{d}{\mu\,\operatorname{tr}(H^{-1})},$$2 extra compute. For the main quadratic experiments, the latter appears to be the relevant setup.

4. Theoretical guarantees on strongly-convex quadratics

The theoretical analysis is explicitly restricted to a strongly-convex quadratic with diagonal Hessian,

$$\rho(\kappa) \;=\; \frac{d}{\mu\,\operatorname{tr}(H^{-1})},$$3

with stochastic gradients

$$\rho(\kappa) \;=\; \frac{d}{\mu\,\operatorname{tr}(H^{-1})},$$4

and batch size $$\rho(\kappa) \;=\; \frac{d}{\mu\,\operatorname{tr}(H^{-1})},$$5. In the theorem section, the schedule is analyzed in the simplified form

$$\rho(\kappa) \;=\; \frac{d}{\mu\,\operatorname{tr}(H^{-1})},$$6

The main theorem states

$$\rho(\kappa) \;=\; \frac{d}{\mu\,\operatorname{tr}(H^{-1})},$$7

Hence the asymptotic floor is reduced relative to the unmodulated case $$\rho(\kappa) \;=\; \frac{d}{\mu\,\operatorname{tr}(H^{-1})},$$8 by the exact factor

$$\rho(\kappa) \;=\; \frac{d}{\mu\,\operatorname{tr}(H^{-1})},$$9

At the recommended $[\mu,L]$0, this yields the claimed factor-of-two improvement in the asymptotic error floor (Hamza et al., 29 Jun 2026).

The proof tracks the Lyapunov quantity

$[\mu,L]$1

With mini-batch averaged noise $[\mu,L]$2, the recursion becomes

$[\mu,L]$3

Near the optimum, Proposition 1 implies $[\mu,L]$4, so $[\mu,L]$5 and therefore

$[\mu,L]$6

Balancing contraction and injected noise at steady state yields

$[\mu,L]$7

The paper stresses two interpretive points. First, this theorem is weaker than the earlier aspirational curvature-adaptive factor $[\mu,L]$8. Second, it is stronger than what is observed at finite $[\mu,L]$9: the theorem gives $\rho(\kappa)$0 at $\rho(\kappa)$1, whereas experiments show about $\rho(\kappa)$2 ($\rho(\kappa)$3) in finite-horizon loss. The authors attribute the gap to finite-horizon effects and slower transient progress. They also state monotonicity in $\rho(\kappa)$4: the residual floor decreases strictly with $\rho(\kappa)$5, though in practice they recommend $\rho(\kappa)$6, with $\rho(\kappa)$7 best among tested values.

5. Empirical results on synthetic quadratics

All reported experiments are on synthetic strongly-convex quadratics of the form

$\rho(\kappa)$8

with $\rho(\kappa)$9, log-spaced eigenvalues in $\kappa/\log\kappa$0, $\kappa/\log\kappa$1, $\kappa/\log\kappa$2, and per-sample stochastic gradient $\kappa/\log\kappa$3. The default noise model is $\kappa/\log\kappa$4 with $\kappa/\log\kappa$5; the default optimization settings are $\kappa/\log\kappa$6, $\kappa/\log\kappa$7, $\kappa/\log\kappa$8, and $\kappa/\log\kappa$9 (Hamza et al., 29 Jun 2026).

Across $\rho(\kappa)=2.0,2.5,3.1,3.9$0, using 20 runs per setting, CWGD-Cosine consistently outperforms plain cosine annealing by about $\rho(\kappa)=2.0,2.5,3.1,3.9$1 in final suboptimality, with all paired $\rho(\kappa)=2.0,2.5,3.1,3.9$2-tests satisfying $\rho(\kappa)=2.0,2.5,3.1,3.9$3.

$\rho(\kappa)=2.0,2.5,3.1,3.9$4	CWGD-Cosine vs cosine annealing	Reported factor
$\rho(\kappa)=2.0,2.5,3.1,3.9$5	$\rho(\kappa)=2.0,2.5,3.1,3.9$6 vs $\rho(\kappa)=2.0,2.5,3.1,3.9$7	$\rho(\kappa)=2.0,2.5,3.1,3.9$8
$\rho(\kappa)=2.0,2.5,3.1,3.9$9	$\eta \lesssim 1/L$00 vs $\eta \lesssim 1/L$01	$\eta \lesssim 1/L$02
$\eta \lesssim 1/L$03	$\eta \lesssim 1/L$04 vs $\eta \lesssim 1/L$05	$\eta \lesssim 1/L$06
$\eta \lesssim 1/L$07	$\eta \lesssim 1/L$08 vs $\eta \lesssim 1/L$09	$\eta \lesssim 1/L$10

The gain is therefore stable at roughly $\eta \lesssim 1/L$11, substantially smaller than the theorem’s asymptotic factor-of-two floor reduction. The paper interprets this difference as a finite-horizon phenomenon rather than a contradiction of the asymptotic analysis.

The ablation results reinforce the same pattern. In the $\eta \lesssim 1/L$12 sweep at $\eta \lesssim 1/L$13, performance improves monotonically from $\eta \lesssim 1/L$14 at $\eta \lesssim 1/L$15 to $\eta \lesssim 1/L$16 at $\eta \lesssim 1/L$17. Across batch sizes $\eta \lesssim 1/L$18, the improvement remains stable at $\eta \lesssim 1/L$19 to $\eta \lesssim 1/L$20, with all $\eta \lesssim 1/L$21. Under aligned noise $\eta \lesssim 1/L$22 with $\eta \lesssim 1/L$23, gains are slightly larger, $\eta \lesssim 1/L$24 in both reported cases, than under isotropic noise, $\eta \lesssim 1/L$25.

Estimator robustness is reported as very strong in the supported setting. On diagonal quadratics, relative error is $\eta \lesssim 1/L$26. Under a $\eta \lesssim 1/L$27 rotated Hessian at $\eta \lesssim 1/L$28, the relative error in the resulting CWGD is reported as $\eta \lesssim 1/L$29 in Table 4, while the text also mentions $\eta \lesssim 1/L$30. The paper therefore contains a small inconsistency in the reported worst-case value.

The compute-normalized comparison uses a one-time $\eta \lesssim 1/L$31 initialization cost. Under that accounting, CWGD-Cosine at 4000 steps is compared against plain cosine at 4020 steps, and the result remains $\eta \lesssim 1/L$32 vs $\eta \lesssim 1/L$33, a $\eta \lesssim 1/L$34 improvement with $\eta \lesssim 1/L$35.

6. Scope, failure modes, and relation to other adaptive methods

The paper is unusually explicit about scope. Its theory applies only to strongly-convex quadratics, diagonal Hessians, and isotropic noise. It does not claim a general nonconvex result (Hamza et al., 29 Jun 2026). The principal practical concern is Hessian staleness: in nonconvex training, a curvature estimate $\eta \lesssim 1/L$36 computed at time $\eta \lesssim 1/L$37 can rapidly become uncorrelated with the true Hessian later, so the weighting in CWGD is wrong. The authors report that on nonconvex classification models this caused CWGD-Cosine to underperform plain cosine by $\eta \lesssim 1/L$38 to $\eta \lesssim 1/L$39, though no full tables are given. On Rosenbrock, CWGD-Cosine also fails because a static diagonal proxy cannot capture rapidly changing off-diagonal curvature.

Even within quadratics, the method is not presented as a universal accelerator. At extreme ill-conditioning $\eta \lesssim 1/L$40, standard cosine performs better at $\eta \lesssim 1/L$41 because optimization is still in the transient phase and CWGD’s conservative modulation slows descent before the asymptotic noise floor matters. The method is therefore framed as an asymptotic variance-reduction schedule, not a guaranteed accelerator of early training.

Its relation to alternatives is correspondingly narrow and specific. Compared to Adam, AdaGrad, and AdaHessian, CWGD-Cosine does not precondition parameters or maintain per-parameter adaptive steps; it remains a single scalar global learning-rate modulator on top of cosine annealing. Compared to gradient diversity measures such as $\eta \lesssim 1/L$42, its novelty is the curvature weighting via $\eta \lesssim 1/L$43. Compared to EigenCurve or LANTON, it uses curvature as an online noise signal, not just schedule design metadata. The paper also clearly prefers the Hutchinson-based construction over the abandoned naive estimator because the latter provably degenerates to a constant and carries no useful signal.

Within the supported regime, the practical takeaway is correspondingly restricted. CWGD-Cosine is a cosine annealing schedule multiplied by $\eta \lesssim 1/L$44, where $\eta \lesssim 1/L$45 is mini-batch gradient diversity measured after inverse-square-root Hessian whitening. In strongly-convex quadratics with diagonal or near-diagonal curvature, it is simple to implement, essentially exact with Hutchinson diagonal estimation, and reduces final loss by about $\eta \lesssim 1/L$46 empirically while enjoying a proven $\eta \lesssim 1/L$47 reduction in the asymptotic noise floor. The main caveats are finite-horizon slowdown, sensitivity to stale curvature estimates in nonconvex problems, and lack of theory beyond the quadratic setting.