Gradient-Norm-Inverse Scheduling

• Gradient-Norm-Inverse Scheduling is an optimization strategy that adapts learning rates by normalizing the current gradient norm with its historical maximum to ensure smooth decay.
• The approach underpins advanced optimizers like ZENITH and integrates with classical methods such as ISTA/FISTA, yielding improved convergence with minimal computational overhead.
• This scheduling technique is robust against regularization effects, maintaining stability and achieving flatter minima compared to naive methods.

Gradient-Norm-Inverse Scheduling refers to a family of optimization strategies in which the learning rate or step size is adaptively modulated based on the norm of the gradient or proximal subgradient. This approach underpins recent advances in stochastic optimization, such as the ZENITH optimizer in deep learning, and the convergence rate analysis of classical first-order convex algorithms (e.g., ISTA, FISTA). The central mechanic is the use of a quantity derived from the gradient norm to either directly or indirectly schedule learning rates, achieving automatic decay and often sharper convergence to flat minima with little or no additional computational or memory overhead.

1. Foundational Concepts and Mechanism

Gradient-norm-inverse scheduling fundamentally centers on the use of either the raw gradient norm $\|\nabla \mathcal{L}(\theta_t)\|_2$ (as in pure SGD settings) or the proximal subgradient norm (for composite optimization, e.g., ISTA/FISTA frameworks) as an informative signal for scheduling the optimizer step size. The gradient norm serves as a proxy for the local steepness of the loss landscape. In the context of non-composite objectives, the scheduling rule often involves normalizing the instantaneous (or smoothed) norm by a running maximum, yielding an adaptive, monotonic decay.

A canonical instantiation is provided by ZENITH, where at training iteration $t$:

• The gradient norm $g_t = \|\nabla \mathcal{L}(\theta_t)\|_2$ is maintained.
• To reduce minibatch noise, a rolling mean $H_t$ over a window of $W$ most recent $g_t$'s is computed:

$H_t = \frac{1}{W} \sum_{k=0}^{W-1} g_{t-k}$

• The "zenith" $Z_t$ is tracked as the historical maximum of $H_t$:

$Z_t = \max\{Z_{t-1}, H_t\},\quad Z_0=0$

• The learning rate schedule is set as:

$\eta_t = \eta_0 \, \frac{H_t}{Z_t}$

where $\eta_0$ is the initial learning rate hyperparameter. Thus, early in training, when $H_t \approx Z_t$, $\eta_t \approx \eta_0$; as the gradient norm decays, $\eta_t$ does so automatically in a smooth and controlled fashion (Saha, 21 Jan 2026).

2. Algorithmic Instantiations

ZENITH Optimizer

The ZENITH optimizer exemplifies gradient-norm-inverse scheduling in modern deep learning pipelines. It operates as follows:

1. Compute $g_t = \|\nabla \mathcal{L}(\theta_t)\|_2$.
2. Update the FIFO queue $Q$ of size $W$, maintaining the window of recent norms.
3. If $Q$ is full, update $H_t$ and $Z_t$, and compute $\eta_t$.
4. Update parameters with a single SGD-type step using $\eta_t$.

This mechanism introduces only a single dot product per step beyond standard SGD, with measured computational overhead of 1–2% (substantially lower than adaptive baselines, which incur up to 15–373% overhead due to per-parameter statistics). Memory overhead consists strictly of $W+2$ scalars ($W$ for the queue, $H_t, Z_t$), which stands in stark contrast to baselines requiring 2–6× model-size in auxiliary buffers (Saha, 21 Jan 2026).

Proximal Subgradient Norm Scheduling in ISTA/FISTA

Within composite optimization, such as in ISTA and FISTA, the scheduling function is embedded within the step-size analysis. For a composite objective $\Phi(x) = f(x) + g(x)$ with $f \in \mathcal{F}^1_L$, $g$ convex and possibly non-smooth, and $s$ the step size,

• The $s$–proximal operator is $$P_s(y) = \arg\min_z \{\frac{1}{2s}\|z-(y-s\nabla f(y))\|^2 + g(z)\}$$
• The proximal subgradient is $G_s(y) = \frac{y-P_s(y)}{s}$

A crucial Lyapunov-based argument shows that for $0 < s \le 1/L$ (ISTA) and $0 < s < 1/L$ (FISTA), constant step size already enforces that the squared norm decays as $O(1/k^2)$ in ISTA and $O(1/k^3)$ in FISTA, without requiring diminishing schedules. This pattern, which the authors denote as “gradient-norm inverse scheduling,” arises from sharp control of Lyapunov differences via a tightened pivotal inequality (Li et al., 2022).

3. Theoretical Underpinnings and Convergence Guarantees

ZENITH’s schedule is derived by normalizing the smoothed gradient norm to its historical maximum. This approach yields a theoretical profile characterized by:

• High step size ($\eta_t \approx \eta_0$) when gradients are large, facilitating exploration and escape from sharp basins.
• Automatic, monotonic decay of the learning rate as gradients shrink, enabling fine descent into flatter minima.
• Invariance to global scaling of the objective, ensuring compatibility with $\ell_2$ regularization and addressing a known weakness of Polyak and distance-aware rules, whose schedules can diverge or mis-estimate under regularization.

Appendix B in ZENITH proves that, under standard smoothness assumptions and provided $\eta_0 < 2/L$, the norm $\|\nabla \mathcal{L}(\theta_t)\|_2$ converges to zero, thereby guaranteeing asymptotic convergence (Saha, 21 Jan 2026).

The ISTA/FISTA theoretical analysis exploits a tightened pivotal inequality in the Lyapunov function difference. This allows for optimal rates of norm minimization: $O(1/k^2)$ for ISTA, $O(1/k^3)$ for FISTA, strictly under constant step size $s \le 1/L$ for stability (Li et al., 2022). These results establish that an implicit "inverse" schedule on the gradient norm is sufficient for fast convergence, obviating the need for adaptive or diminishing step sizes.

4. Comparative Analysis and Limitations of Naïve Approaches

Naïve gradient-norm-inverse step size rules, such as $\eta_t^{\text{inv}} \propto 1/(g_t+\varepsilon)$, lack normalization and historical context:

• They may yield unbounded or explosive learning rates when $g_t\to 0$ unless clipped or regularized.
• Learning rate evolution is neither smooth nor stable, often exhibiting oscillations and poor resemblance to cosine or polynomial decay favored in deep learning practice.
• Empirically, such naive schedules diverge or lead to sharp minima with degraded generalization, as measured by the Hessian spectrum and test accuracy.

Head-to-head comparison (e.g., on CIFAR-100) demonstrates that ZENITH converges faster, yields flatter minima, and consistently produces 2–4% higher test accuracy. Naïve approaches either fail to converge or lock into sharp minima with poor generalization (Saha, 21 Jan 2026).

5. Practical Implications and Regularization Robustness

ZENITH’s learning rate schedule, depending solely on the ratio $H_t/Z_t$, is exactly invariant to the absolute scaling of loss or gradient norms. Thus, the insertion of $\ell_2$ regularization or weight decay does not disrupt the schedule’s form or efficacy. Empirical evaluation reveals further accuracy improvements from regularization, whereas alternative schedules (such as Polyak or distance-based schemes) may become unreliable under regularization (Saha, 21 Jan 2026).

This property ensures that gradient-norm-inverse scheduling is robust for hyperparameter settings typically encountered in large-scale deep learning. Moreover, the approach is compatible with a wide range of model architectures and task domains, as evidenced by its favorable performance across classification, object detection, keypoint detection, and segmentation on MS COCO and diverse CNN architectures (Saha, 21 Jan 2026).

6. Perspectives in Composite and Accelerated Optimization

The gradient-norm-inverse concept generalizes beyond stochastic deep learning, as evidenced in the phase-space and Lyapunov theory underpinning ISTA and FISTA. Here, the decay rate of the proximal subgradient norm is dictated by the step size $s$. By fixing $s$ within the stability region, ISTA and FISTA algorithms automatically enforce inverse-square and inverse-cubic norm decay, respectively, aligning with classical optimal rates and requiring no explicit learning rate decay schedule (Li et al., 2022).

A plausible implication is that gradient-norm-informed scheduling, either explicit or implicit, constitutes a unifying principle across modern stochastic optimizers and classical convex optimization schemes, with both empirical and theoretical justification.

---

Cited Papers:

• “ZENITH: Automated Gradient Norm Informed Stochastic Optimization” (Saha, 21 Jan 2026)
• “Proximal Subgradient Norm Minimization of ISTA and FISTA” (Li et al., 2022)