LMO Framework & Hessian-Corrected Momentum

• The LMO framework is a method that leverages linear minimization oracles combined with Hessian corrections to achieve optimal convergence in stochastic optimization and MCMC.
• It incorporates second-order Hessian information into momentum schemes, reducing estimator bias and variance for improved sampling and optimization efficiency.
• The framework demonstrates robust performance in nonconvex settings, distributed learning, and high-dimensional Bayesian inference with proven convergence rates.

Hessian-Corrected Momentum (HCM) refers to a class of algorithms that incorporate second-order (Hessian) information into momentum schemes for stochastic optimization, Markov chain Monte Carlo (MCMC), and related computational methods. Rather than relying solely on first-order gradient statistics, HCM introduces explicit corrections using Hessian or Hessian-vector products to adaptively shape both momentum and proposal distributions according to local curvature, execute variance reduction, or accelerate convergence. This approach has led to statistically efficient sampling in Bayesian inference, optimal rates in stochastic optimization, and robust mixing in high-dimensional problems.

1. Foundational Principles and Mathematical Formulation

The central premise of HCM is that naive momentum schemes exhibit an intrinsic bias, as momentum is a lagging estimate based on past gradients, which neglects the change in curvature between successive iterates. Mathematically, for an objective $F(x)$ with momentum buffer $m_{t-1} \approx \nabla F(x_{t-1})$ at time $t$, a Taylor expansion yields

$$\nabla F(x_t) \approx \nabla F(x_{t-1}) + \nabla^2 F(x_{t-1}) (x_t - x_{t-1}).$$

Classical momentum thus incurs an $O(\|x_t - x_{t-1}\|)$ error. HCM introduces a Hessian-vector correction term $\nabla^2 F(x_t)(x_t - x_{t-1})$, yielding an update of the form

$$\hat g_t = (1-\alpha) \left[ \hat g_{t-1} + \nabla^2 f(x_t, z_t)(x_t - x_{t-1}) \right] + \alpha \nabla f(x_t, z_t),$$

which reduces bias to second order and achieves lower estimator variance (Tran et al., 2021, Khirirat et al., 15 Dec 2025, Sadiev et al., 18 Nov 2025). In MCMC contexts, second-order corrections are integrated into the proposal distribution’s mean and covariance, resulting in locally adapted transitions and improved mixing (House, 2015, Karimi et al., 2023, House, 2017).

2. HCM in Stochastic Optimization and Momentum Schemes

HCM has yielded significant advances in stochastic optimization, notably in algorithms for nonconvex objectives and distributed machine learning. Key forms include:

• Heavy-ball and Nesterov acceleration with Hessian-driven damping (Hadjisavvas et al., 18 Jun 2025): Time discretization of a continuous inertial system with Hessian damping produces a two-step recurrence,

$$y_k = x_k + \alpha (x_k - x_{k-1}) - \theta (\nabla h(x_k) - \nabla h(x_{k-1})), \quad x_{k+1} = y_k - \beta \nabla h(x_k),$$

which adaptively damps oscillations in high-curvature directions, ensuring robust convergence in strongly quasiconvex settings. Linear convergence is established through discrete Lyapunov arguments.

• Stochastic Cubic Newton with Momentum (Chayti et al., 25 Oct 2024, Yang et al., 17 Jul 2025): Momentum variants stabilize noisy gradient and Hessian estimates. For Hessian-momentum recursion,

$$H_t = (1-\beta) H_{t-1} + \beta \nabla^2 f_{\xi^h_t}(x_t),$$

variance reduction is achieved by reusing past curvature information, and the cubic regularization subproblem can be solved using these stabilized Hessians to reach second-order stationary points, even with single-sample estimates per iteration.

• HCM in distributed stochastic optimization (Sadiev et al., 18 Nov 2025): In communication-efficient error feedback, Hessian corrections are integrated client-side, producing the update

$v_i^{t+1} = (1-\eta_t) (v_i^t + h_i) + \eta_t g_i,$

with $$h_i = \nabla^2 f_i(x^{t+1};\xi_i^{t+1})(x^{t+1}-x^t)$$. This technique attains the theoretical minimum rate $O(1/T^{1/3})$ for mean gradient norm under standard smoothness and Hessian-Lipschitz conditions.

• HCM in general norm settings/LinMinOracle frameworks (Khirirat et al., 15 Dec 2025): Robustness has been extended beyond Euclidean geometry, showing that HCM can attain optimal $O(1/K^{1/3})$ rates in arbitrary Banach spaces via norm-equivalent updates and Hessian-vector transport.

3. Applications in Markov Chain Monte Carlo and Bayesian Inference

HCM-based corrections form the backbone of advanced MCMC methods. Major families include:

• Hessian-corrected Metropolis Adjusted Langevin Algorithm (HMALA) (House, 2015): Diffusion proposals are locally linearized with second-order Taylor expansion. The resulting SDE is Gaussian with closed-form mean and covariance:

$$m = (e^{½H\delta} - I) H^{-1} v, \qquad S = (e^{H\delta} - I) H^{-1},$$

yielding proposal $x_{t+1} \sim \mathcal N(x_t + m, S)$. A Metropolis-Hastings correction ensures $\pi$-invariance and geometric ergodicity, with sharp improvements in mixing and robustness at saddle points and multimodal targets.

• Hessian-informed Hamiltonian Monte Carlo (H(HMC)) (Karimi et al., 2023, House, 2017): The kinetic term employs a mass matrix parameterized by either the local or global Hessian of the negative log-posterior. For instance, in local HCM, $M = \nabla^2J(\theta_k)^{-1}$ per iteration, used as constant during each trajectory. This approach is shown to dramatically accelerate mixing in high dimensions compared to vanilla HMC, while avoiding the full computational burden of Riemannian manifold HMC.
• Ensemble Quasi-Newton HMC (Jin et al., 2019): Multiple chains are coupled by a data-driven, L-BFGS-based approximation of the inverse Hessian built from recent ensemble history. This preconditioning of the kinetic term preserves reversibility and detailed balance, leading to accelerated sampling especially for stiff and anisotropic models.

4. Estimator Variance Reduction, Complexity Bounds, and Convergence

HCM’s appeal derives from its capacity to drastically reduce estimator variance and bias, producing provable efficiencies:

• Variance reduction and bias correction (Chayti et al., 25 Oct 2024, Sadiev et al., 18 Nov 2025, Khirirat et al., 15 Dec 2025): The third-moment of the Hessian estimator error $\Delta_t$ satisfies

$$\frac{1}{T}\sum_t \mathbb{E}\|\Delta_t\|^3 = O(\beta^{3/2} \tilde \sigma_h^3(\beta) + \cdots),$$

with $\beta$ the momentum parameter. As $\beta \to 0$, the variance component vanishes, and residual bias is absorbed into higher-order progress terms.

• Complexity bounds: HCM-based momentum methods reach the lower bound $O(\epsilon^{-3})$ for finding $\epsilon$-critical points (Tran et al., 2021), matching second-order oracle methods and surpassing classical momentum ($O(\epsilon^{-4})$).
• Global convergence: Under relaxed smoothness, HCM achieves the optimal $O(1/T^{1/3})$ rate in mean gradient norm. In reinforcement learning, NPG-HM attains $O(\epsilon^{-2})$ sample complexity for $\epsilon$-optimality under Fisher nondegeneracy (Feng et al., 2 Jan 2024).

5. Algorithmic Design, Pseudocode, and Implementation Considerations

HCM algorithms are modular and integrate seamlessly into existing frameworks. Core pseudocode constructs (first-order optimization (Tran et al., 2021), cubic Newton (Chayti et al., 25 Oct 2024), Langevin MCMC (House, 2015)) follow the general template:

for t in range(T): grad = compute_gradient(x_t, z_t) hvp = compute_hessian_vector_product(x_t, z_t, x_t - x_{t-1}) temp = m_{t-1} + hvp m_t = (1 - alpha) * temp + alpha * grad x_{t+1} = x_t - eta_t * m_t

Efficient computation of Hessian-vector products (Pearlmutter’s trick) is essential. For cubic Newton or LMO-based frameworks, Hessian-momentum updates resemble Polyak or SPIDER recursions, and can exploit sparsity or low-rank structure.

Computational cost is typically $O(d^2)$ per Hessian evaluation or $O(d^3)$ for full matrix exponentiation/inversion (as in SDE-based MALA/HMC (House, 2015)). Krylov subspace tricks, diagonal/low-rank approximations, or ensemble-based approximators mitigate expense for large-scale problems. Empirical studies show overheads of 1.3–1.7× over pure first-order methods are typical, but often justified by several-fold improvements in mixing and convergence speed.

6. Empirical Evaluation and Theoretical Impact

Empirical results across domains validate HCM’s efficiency gains:

• Sampling efficiency: HMALA yields a four-fold increase in effective sample size over tuned random walk proposals (House, 2015); H(HMC) achieves rapid autocorrelation decay even in $d\gtrsim 1000$ (Karimi et al., 2023).
• Stochastic optimization: SGDHess matches or outperforms SGD/Adam/AdaHessian benchmarks in image classification and neural machine translation, often at modest additional cost (Tran et al., 2021).
• Nonconvex optimization and deep learning: On MLP and LSTM tasks, HCM under LMO outpaces classic and extrapolated momentum in both gradient norm and loss (Khirirat et al., 15 Dec 2025); in error-feedback training, EF21-HM attains minimal convergence rates with parameter-agnostic stepsize schedules (Sadiev et al., 18 Nov 2025).
• Reinforcement learning: NPG-HM achieves the best known last-iterate $\epsilon$-optimality and fastest convergence on standard Mujoco settings (Feng et al., 2 Jan 2024).

These results demonstrate HCM’s robustness to ill-conditioning, saddle points, and oscillatory dynamics, its scalability in distributed and high-dimensional settings, and its practical efficiency in real-world learning tasks.

7. Historical Development and Contextual Positioning

The introduction of second-order corrections to momentum dates to advanced MCMC methods (Hessian-corrected HMC (House, 2017), HMALA (House, 2015)), where precise local geometry adaptation was essential for high-dimensional Bayesian inference. Modern stochastic optimization has integrated these principles—sometimes under alternate nomenclature (e.g., Hessian-aided momentum, implicit transport, heavy-ball with Hessian damping)—to overcome limitations of first-order momentum and stalling in nonconvex regimes.

Recent works have unified HCM techniques under generalized frameworks for variance reduction (momentum recursions, cubic Newton, error feedback, LMO) and have established theoretical lower bounds matching the best possible rates for nonconvex stochastic optimization under mild smoothness and Hessian regularity assumptions. Extensions to arbitrary norm geometries and linear minimization oracles further underscore the generality of the approach (Khirirat et al., 15 Dec 2025).

In sum, Hessian-Corrected Momentum constitutes a broad family of algorithmic tools that meld momentum principles with curvature adaptation via second-order information—conferring provable, optimal efficiency and wide applicability in modern machine learning, Bayesian computation, and optimization theory.