OSGM-Best: Adaptive Heavy-Ball Optimizer

• OSGM-Best is a momentum-enhanced variant of the Online Scaled Gradient Methods framework that treats step-size selection as an online convex optimization problem in preconditioner space.
• It combines heavy-ball momentum, hypergradient feedback, and lookahead corrections with a Lyapunov potential to ensure monotonic progress and robust convergence.
• Empirical results show that OSGM-Best achieves quasi-Newton-like convergence on convex and nonconvex tasks while using significantly less memory and computational cost.

OSGM-Best is a practical instantiation of the Online Scaled Gradient Methods (OSGM) framework in which step-size selection is treated as an online convex optimization problem in preconditioner space rather than as a fixed hyperparameter choice. Within this framework, OSGM-Best is the momentum-enhanced variant introduced in Part II of the OSGM project. It combines hypergradient-type feedback, heavy-ball momentum, a lookahead landscape action, and online adaptation of both a matrix step-size $P_k$ and a scalar momentum coefficient $\beta_k$. Its role is to translate the theoretical guarantees developed for OSGM in smooth convex and strongly convex settings into a robust first-order method whose empirical behavior matches quasi-Newton variants while using less memory and cheaper iterations (Chu et al., 13 Sep 2025, Gao et al., 29 May 2025).

1. Position within the OSGM framework

OSGM is a family of first-order methods in which the step-size, or more generally the preconditioner, is itself learned online. The foundational construction couples an optimization process on $x$ with an online learner acting in the space of candidate step-size matrices $P_k$. The key perspective is that step-size selection is treated as an online learning problem in the step-size space, and regret guarantees are then converted into convergence guarantees for the primal iterates (Gao et al., 29 May 2025).

In Part I, two central feedback families are developed. The ratio feedback

$$r_x(P) := \frac{f(x - P \nabla f(x)) - f(x^*)}{f(x) - f(x^*)}$$

measures contraction in suboptimality, while the hypergradient feedback

$$h_x(P) := \frac{f(x - P \nabla f(x)) - f(x)}{\|\nabla f(x)\|^2}$$

measures normalized decrease in objective value. OSGM-Best is built on the hypergradient branch, specifically on the analysis of OSGM-H, but modifies that branch to address practical deficiencies identified in Part II: OSGM-H is theoretically sound yet exhibits spiky behavior and limited practical speed (Chu et al., 13 Sep 2025).

Part II therefore introduces OSGM-Best as the practical “best” variant of the framework. It couples OSGM with heavy-ball momentum, uses a potential rather than bare function decrease as the monitored quantity, employs a lookahead correction, and updates both $P_k$ and $\beta_k$ online. This suggests that OSGM-Best is best understood not as an isolated optimizer but as a specific operationalization of the more general OSGM principle.

2. Algorithmic construction

The base iteration of OSGM-Best is a heavy-ball update,

$$x_{k+1} = x_k - P_k \nabla f(x_k) + \beta_k (x_k - x_{k-1}),$$

where $P_k \in \mathbb{R}^{n\times n}$ is a matrix step-size and $\beta_k$0 is a momentum coefficient. In contrast to plain heavy-ball, both quantities are adapted online by the scheduler (Chu et al., 13 Sep 2025).

Because plain heavy-ball does not ensure monotonic decrease of $\beta_k$1, OSGM-Best uses the Lyapunov potential

$\beta_k$2

with $\beta_k$3 chosen appropriately. The state is written as $\beta_k$4, and the heavy-ball proposal is

$\beta_k$5

The landscape then applies a lookahead correction on the potential. After the scheduler proposes $\beta_k$6, the lookahead action is

$\beta_k$7

A monotone landscape action follows: $\beta_k$8 This null-step mechanism ensures that $\beta_k$9 is monotone nonincreasing (Chu et al., 13 Sep 2025).

The online feedback used by the scheduler is the heavy-ball potential-based quantity

$x$0

Negative values correspond to stronger normalized decrease. The scheduler then performs online gradient updates,

$x$1

In the theory, the parameter choices are

$x$2

and the standard initialization is

$x$3

Part II brands this heavy-ball OSGM variant with lookahead and monotone potential as OSGM-Best (Chu et al., 13 Sep 2025).

3. Feedback geometry and online adaptation

The central technical reason OSGM-Best is analyzable is that its feedback is constructed to be compatible with online convex optimization. For convex $x$4 and suitably large $x$5, the heavy-ball feedback $x$6 is jointly convex in $x$7 and $x$8-smooth with respect to the joint norm

$x$9

with dual norm

$P_k$0

This extends the Part I principle—online learning in step-size space—to a joint matrix-and-momentum space (Chu et al., 13 Sep 2025, Gao et al., 29 May 2025).

The explicit gradient is given by

$P_k$1

As in Part I, the scheduler’s update is an online gradient descent step driven by a feedback function explicitly tied to progress rather than by gradient-magnitude statistics alone.

This distinguishes OSGM-Best from AdaGrad, Adam, and RMSProp. Those methods adapt learning rates from accumulated gradient information, whereas OSGM-Best adapts $P_k$2 by minimizing a convex surrogate of potential decrease. It also distinguishes OSGM-Best from classic hypergradient descent. Part II states that the classic hypergradient ordering updates the step-size and then uses it immediately in the current primal step; on quadratics this can stabilize the method on a poor spectral orbit where the step-size is “optimal” but convergence remains slow in direction. OSGM-Best instead uses the OSGM ordering—update $P_k$3 using $P_k$4, then update $P_k$5 using feedback—and combines that with potential monotonicity so that bad orbits are unstable rather than attractors (Chu et al., 13 Sep 2025).

4. Convergence guarantees

The convergence analysis of OSGM-Best proceeds through a potential-reduction quantity

$P_k$6

For any monotone sequence $P_k$7, the heavy-ball reduction theorem converts aggregate progress in $P_k$8 into primal convergence (Chu et al., 13 Sep 2025).

In the convex case, Theorem 2.3 gives a simplified rate

$P_k$9

which is an $$r_x(P) := \frac{f(x - P \nabla f(x)) - f(x^*)}{f(x) - f(x^*)}$$0 bound. In the strongly convex case, for suitable benchmark and initialization $$r_x(P) := \frac{f(x - P \nabla f(x)) - f(x^*)}{f(x) - f(x^*)}$$1, the paper simplifies the bound to

$$r_x(P) := \frac{f(x - P \nabla f(x)) - f(x^*)}{f(x) - f(x^*)}$$2

with $$r_x(P) := \frac{f(x - P \nabla f(x)) - f(x^*)}{f(x) - f(x^*)}$$3. Part II describes this as linear convergence at the optimal gradient-descent rate for smooth strongly convex problems (Chu et al., 13 Sep 2025).

The mechanism behind these guarantees follows the OSGM logic from Part I. There, no-regret control of feedback implies that the learned step-size sequence is asymptotically no worse than the best static preconditioner in hindsight, and trajectory-dependent bounds compare against slowly varying benchmark sequences through dynamic regret (Gao et al., 29 May 2025). OSGM-Best inherits this architecture, but replaces the single-parameter step-size view with joint online learning of a matrix preconditioner and a momentum scalar under a heavy-ball Lyapunov potential. A plausible implication is that OSGM-Best serves as the practically stabilized counterpart of the more abstract OSGM-H theory.

Part II also emphasizes robustness. The monotone potential action allows nonmonotone objective values while preserving monotone Lyapunov decrease, which is precisely the setting in which momentum methods are usually difficult to control analytically.

5. Empirical behavior and comparison with related methods

The empirical study in Part II evaluates OSGM-Best on 38 LIBSVM datasets for SVM classification objectives and logistic regression, and on 47 unconstrained problems from CUTEst. Each algorithm is run with maximum 1000 gradient calls in the convex setting or 2000 in the nonconvex setting, the same starting point, and stopping criterion $$r_x(P) := \frac{f(x - P \nabla f(x)) - f(x^*)}{f(x) - f(x^*)}$$4 (Chu et al., 13 Sep 2025).

On convex SVM and logistic regression tasks, OSGM-Best’s number of solved problems is described as comparable to L-BFGS-M10. In the reported plots, its function value gap and gradient norm decay are significantly faster than GD, GD-HB, and AGD variants, much better than AdaGrad and often Adam, and often overlap with or lie slightly below L-BFGS-M5 and L-BFGS-M10 curves. On nonconvex CUTEst problems, it significantly outperforms Adam and AdaGrad and is comparable to L-BFGS with larger memory in both function-value and gradient-norm reduction (Chu et al., 13 Sep 2025).

Memory and per-iteration cost are central to the method’s positioning. Practical OSGM-Best uses about 7 vectors of length $$r_x(P) := \frac{f(x - P \nabla f(x)) - f(x^*)}{f(x) - f(x^*)}$$5, similar to L-BFGS with memory 1 and much less than L-BFGS-M10. Its numerical cost is dominated by gradient computation, and updates of $$r_x(P) := \frac{f(x - P \nabla f(x)) - f(x^*)}{f(x) - f(x^*)}$$6 and $$r_x(P) := \frac{f(x - P \nabla f(x)) - f(x^*)}{f(x) - f(x^*)}$$7 are cheap scalar or vector operations without matrix factorizations or line searches. The paper therefore characterizes the method as matching quasi-Newton variants while requiring less memory and cheaper iterations (Chu et al., 13 Sep 2025).

A common misconception is to regard OSGM-Best as a quasi-Newton method in disguise. Part II does not support that characterization. OSGM-Best does not build secant-based curvature approximations and does not form Hessians or quasi-Newton matrices. Its preconditioning is produced by online gradient steps on a scalar-valued convex feedback function. Another misconception is that it is merely heavy-ball with an adaptive learning rate. The defining feature is instead the joint use of online convex optimization, a Lyapunov potential, a lookahead correction, and a monotone landscape action.

6. Scope, extensions, and open problems

Part II extends OSGM to smooth nonconvex optimization by regularizing the hypergradient feedback,

$$r_x(P) := \frac{f(x - P \nabla f(x)) - f(x^*)}{f(x) - f(x^*)}$$8

so that it becomes convex in step-size space when $$r_x(P) := \frac{f(x - P \nabla f(x)) - f(x^*)}{f(x) - f(x^*)}$$9 dominates the local weak convexity induced by nonconvexity. Under this construction, OSGM-H with regularized feedback achieves the standard nonconvex stationarity bound

$$h_x(P) := \frac{f(x - P \nabla f(x)) - f(x)}{\|\nabla f(x)\|^2}$$0

and the PL-case linear bound

$$h_x(P) := \frac{f(x - P \nabla f(x)) - f(x)}{\|\nabla f(x)\|^2}$$1

relative to the baseline $$h_x(P) := \frac{f(x - P \nabla f(x)) - f(x)}{\|\nabla f(x)\|^2}$$2 (Chu et al., 13 Sep 2025).

For OSGM-Best itself, the nonconvex heavy-ball extension is not fully re-analyzed. Part II explicitly states that the robust behavior observed in experiments suggests more is possible, but that a full heavy-ball nonconvex theory remains open. This delimits the scope of current guarantees: the main construction of OSGM-Best is in the convex setting, and the strongest formal guarantees are convex and strongly convex rather than fully general nonconvex ones.

The paper identifies several open directions. One is whether OSGM-R can achieve the same $$h_x(P) := \frac{f(x - P \nabla f(x)) - f(x)}{\|\nabla f(x)\|^2}$$3 complexity without knowledge of $$h_x(P) := \frac{f(x - P \nabla f(x)) - f(x)}{\|\nabla f(x)\|^2}$$4. Another is whether computable potentials for accelerated methods such as AGD can be generated systematically and then incorporated into the OSGM design. Additional questions concern composite and nonsmooth optimization, the Barzilai–Borwein connection viewed through implicit online learning, and a complete nonconvex analysis for heavy-ball OSGM variants (Chu et al., 13 Sep 2025).

Within the OSGM program as a whole, OSGM-Best therefore occupies a specific position: it is the practical, heavy-ball, potential-based realization of online learned preconditioning, designed to preserve the framework’s regret-driven adaptivity while addressing the stability and performance requirements of large-scale deterministic optimization.