Adaptive Quasi-Newton Hessian Preconditioner

• Adaptive Quasi-Newton Hessian Preconditioner is a method that updates matrix approximations to reshape gradient directions based on recycled curvature information.
• It leverages multi-secant, block, and nonlinear formulations to handle evolving problem geometries and noisy data efficiently.
• Empirical results demonstrate reduced measurement and computational costs with faster convergence in applications like VQAs, PDE-constrained problems, and deep learning.

Searching arXiv for the core paper and closely related quasi-Newton preconditioning work to ground the article in published sources. arXiv search query: (Ramôa et al., 2024) Adaptive quasi-Newton Hessian preconditioning denotes a class of methods that maintain and update an approximation of a Hessian, inverse Hessian, or closely related Jacobian, and use it to precondition optimization steps or linear solves as the problem, model, or active subspace changes. Contemporary instances range from adaptive variational quantum algorithms that recycle inverse Hessians across ansatz-growth steps, to nonlinear Schwarz-preconditioned limited-memory quasi-Newton schemes, Bayesian and probabilistic Hessian inference under noisy secant data, and Newton–CG preconditioners built from action constraints on Krylov subspaces (Ramôa et al., 2024, Kothari, 2022, Carlon et al., 2022, Gower et al., 2014).

1. Conceptual scope

At the most general level, quasi-Newton preconditioning uses a matrix $H_k$ or $G_k^{-1}$ to reshape the gradient direction, as in $x_{k+1}=x_k-\alpha_k H_k \nabla f(x_k)$ or $x^{k+1}=x^k-\alpha^k (G^k)^{-1}\nabla f(x^k)$. In Newton–CG and PDE-constrained settings, the same idea appears as a left preconditioner for a sequence of symmetric linear systems or Hessian systems rather than as an explicit optimizer step. This suggests a unifying viewpoint: the preconditioner is an evolving second-order surrogate whose role is to encode curvature already revealed by prior iterates, Hessian-vector products, or local subproblem solves (Du et al., 2023, Gower et al., 2014, Hart et al., 28 Aug 2025).

In nonlinear-preconditioned frameworks, the target of approximation is not always the raw Hessian. For the nonlinear restricted additive Schwarz strategy, the quasi-Newton matrix approximates the Jacobian of the preconditioned residual $\widetilde F_L(x)=x-G(x)$ or $\widetilde F_R(x)=F(G(x))$, and standard secant pairs must therefore be replaced by preconditioned ones. This is why some authors characterize the construction more precisely as an adaptive approximation built on top of a nonlinear Schwarz preconditioner, rather than as a fixed Hessian preconditioner for the original objective (Kothari, 2022).

2. Algebraic foundations

In ordinary BFGS, the quasi-Newton search direction is

$p_k^{(\mathrm{QN})}=-H_k \nabla f(x_k),$

with inverse-Hessian update

$$H_{k+1}=(I-\rho_k s_k y_k^T)H_k(I-\rho_k y_k s_k^T)+\rho_k s_k s_k^T,\qquad \rho_k=\frac{1}{y_k^T s_k},$$

where

$$s_k=x_{k+1}-x_k,\qquad y_k=\nabla f(x_{k+1})-\nabla f(x_k).$$

Canonical restarts typically set $H_0=I$, so all previously learned curvature information is discarded when a new optimization begins (Ramôa et al., 2024).

Generalizations replace the single secant equation by multisecant or subspace action constraints. Using $G_k^{-1}$0 and $G_k^{-1}$1, generalized multisecant conditions are $G_k^{-1}$2 or $G_k^{-1}$3, while the inverse quNac formulation enforces $G_k^{-1}$4. These constructions are derived from least-change or regularized least-squares principles and make BFGS and DFP special cases of broader symmetric or action-constrained updates (Scieur et al., 2020, Gower et al., 2014).

When Hessian samples $G_k^{-1}$5 are available from forward-mode AD, block SR1 and block PSB updates impose the inverse secant condition $G_k^{-1}$6 on a sampled subspace, permit indefinite approximations, and feed a trust-region QCQP rather than a line-search step. A plausible implication is that adaptive preconditioning need not be tied to positive-definite line-search geometry; it can also be designed to preserve negative curvature information (Azzam et al., 2022).

3. Recycling across related subproblems

In adaptive VQAs, the clearest recycling mechanism appears when an $G_k^{-1}$7-parameter optimum $G_k^{-1}$8 and inverse Hessian approximation $G_k^{-1}$9 are transferred to the next iteration through

$x_{k+1}=x_k-\alpha_k H_k \nabla f(x_k)$0

BFGS then proceeds normally, but the final inverse-Hessian state is saved before the convergence check and reused after ansatz growth. The resulting algorithm has the flavor of a continuous optimization where the dimension of the search space is augmented when the gradient norm falls below a threshold; in simulation, Hessian recycling often achieved superlinear convergence, reduced VQE-subroutine measurement cost by roughly one order of magnitude, yielded about $x_{k+1}=x_k-\alpha_k H_k \nabla f(x_k)$1 reduction in optimization measurement cost in many cases, and achieved up to $x_{k+1}=x_k-\alpha_k H_k \nabla f(x_k)$2 reduction in total function evaluations for some qubit-pool cases (Ramôa et al., 2024).

The same continuity principle appears in slowly changing Newton systems and parameterized inverse problems. In inverse quNac, Krylov directions collected while solving $x_{k+1}=x_k-\alpha_k H_k \nabla f(x_k)$3 are converted into an estimate $x_{k+1}=x_k-\alpha_k H_k \nabla f(x_k)$4 and used as a left preconditioner for the next Newton–CG or PCG call, exploiting the fact that the matrices are “slowly changing.” In pseudo-time continuation for PDE-constrained inverse problems, a parameter-aware BFGS update uses $x_{k+1}=x_k-\alpha_k H_k \nabla f(x_k)$5 and

$x_{k+1}=x_k-\alpha_k H_k \nabla f(x_k)$6

while an additional block BFGS update is extracted from PCG search directions; the reported method outperformed a direct optimization solve for the perturbed problem in total linear-solve cost (Gower et al., 2014, Hart et al., 28 Aug 2025).

4. Nonlinear and probabilistic formulations

Nonlinear Schwarz preconditioning changes the residual map before quasi-Newton updating. The left-preconditioned residual is $x_{k+1}=x_k-\alpha_k H_k \nabla f(x_k)$7, the right-preconditioned residual is $x_{k+1}=x_k-\alpha_k H_k \nabla f(x_k)$8, and the secant pairs become

$x_{k+1}=x_k-\alpha_k H_k \nabla f(x_k)$9

or

$x^{k+1}=x^k-\alpha^k (G^k)^{-1}\nabla f(x^k)$0

The quasi-Newton matrix therefore approximates the Jacobian of the transformed residual, not directly $x^{k+1}=x^k-\alpha^k (G^k)^{-1}\nabla f(x^k)$1 (Kothari, 2022).

A related but more general nonlinear-preconditioning viewpoint applies Newton’s method to

$x^{k+1}=x^k-\alpha^k (G^k)^{-1}\nabla f(x^k)$2

with preconditioned Hessian

$x^{k+1}=x^k-\alpha^k (G^k)^{-1}\nabla f(x^k)$3

The resulting smoothness assumption is Lipschitz continuity of $x^{k+1}=x^k-\alpha^k (G^k)^{-1}\nabla f(x^k)$4, not of $x^{k+1}=x^k-\alpha^k (G^k)^{-1}\nabla f(x^k)$5. Under this model, local Q-superlinear and Q-quadratic convergence are established, a regularized isotropic variant attains $x^{k+1}=x^k-\alpha^k (G^k)^{-1}\nabla f(x^k)$6 iteration complexity, and an adaptive version removes the need to know the smoothness constant and allows inexact subproblem solutions while preserving the same complexity order (Bodard et al., 12 May 2026).

In stochastic settings, several papers replace exact secant matching by probabilistic inference. One Bayesian formulation computes a MAP estimator $x^{k+1}=x^k-\alpha^k (G^k)^{-1}\nabla f(x^k)$7 from noisy secant pairs, weighted precision matrices, a Frobenius prior centered at the previous Hessian estimate, and log-barrier eigenvalue constraints; the inverse Hessian is then used to precondition SGD, with the strongest gains reported on ill-conditioned problems and with the lower eigenvalue bound playing a central role in preventing noise amplification. Another probabilistic approach infers a low-rank Hessian from noisy Hessian-vector products and turns it into a preconditioner $x^{k+1}=x^k-\alpha^k (G^k)^{-1}\nabla f(x^k)$8, while noting that in very high-dimensional problems the construction effectively becomes an automatic learning-rate adaptation scheme (Carlon et al., 2022, Roos et al., 2019).

5. Indefinite, cubic-regularized, and sampled-curvature variants

Some adaptive quasi-Newton preconditioners deliberately abandon positive-definite line-search geometry. Using SIMD-parallel forward-mode AD, block Hessian samples $x^{k+1}=x^k-\alpha^k (G^k)^{-1}\nabla f(x^k)$9 can be generated whenever the gradient is evaluated, and block SR1 or block PSB updates can enforce the inverse secant condition $\widetilde F_L(x)=x-G(x)$0 on sampled directions. The step is then computed from a moderate-dimensional trust-region QCQP in the span of the gradient, the quasi-Newton direction, and the sampled curvature directions, allowing negative curvature to be exploited rather than filtered away (Azzam et al., 2022).

In deep learning, this strategy reappears in limited-memory SR1 with adaptive cubic regularization. The SR1 update

$\widetilde F_L(x)=x-G(x)$1

is accepted only when

$\widetilde F_L(x)=x-G(x)$2

and the cubic model is solved in closed form after transforming to the eigenbasis of $\widetilde F_L(x)=x-G(x)$3. Because $\widetilde F_L(x)=x-G(x)$4 is not required to be positive definite, the method can represent and use directions of negative curvature (Ranganath et al., 17 Feb 2025).

Cubic regularization has also been used to globalize classical quasi-Newton directions without abandoning the quasi-Newton geometry. In the Cubically Enhanced Quasi-Newton formulation, $\widetilde F_L(x)=x-G(x)$5 is recovered from a local model with quadratic and cubic terms measured in the $\widetilde F_L(x)=x-G(x)$6-norm, and the explicit stepsize is

$\widetilde F_L(x)=x-G(x)$7

Under a relative inexactness condition for the Hessian approximation, the method has a global $\widetilde F_L(x)=x-G(x)$8 rate for convex functions, improves to $\widetilde F_L(x)=x-G(x)$9 when the inexactness is controlled within a prescribed relative accuracy, and admits an adaptive variant that adjusts to the function’s curvature while retaining the global convergence guarantees of the non-adaptive algorithm (Agafonov et al., 27 Aug 2025).

6. Large-scale, distributed, and application-specific implementations

Large-scale implementations emphasize cheap curvature acquisition, bounded memory, and communication efficiency. The adaptive greedy quasi-Newton method maintains $\widetilde F_R(x)=F(G(x))$0, updates it through multiple greedy quasi-Newton refinements based on Hessian-vector products, and uses

$\widetilde F_R(x)=F(G(x))$1

with an explicit adaptive stepsize chosen so that the Hessian approximation error remains uniformly bounded. Its master–worker extension DAGQN communicates $\widetilde F_R(x)=F(G(x))$2 scalars per worker per iteration, a cost described as comparable with that of first-order methods. Related action-constrained inverse quNac updates were implemented in full-memory and limited-memory forms and described as the first parallel quasi-Newton preconditioners (Du et al., 2023, Gower et al., 2014).

Other large-scale variants trade matrix richness against robustness. adaQN trains RNNs with a stochastic L-BFGS update, an Adagrad-like diagonal initialization

$\widetilde F_R(x)=F(G(x))$3

aggregated curvature pairs, and memory flushing when a monitoring-set test deteriorates. Apollo uses a parameter-wise weak secant condition to update a diagonal Hessian approximation, rectifies it by $\widetilde F_R(x)=F(G(x))$4, and thereby retains linear time and memory complexity. LODO replaces analytic update formulas by a neural-network parameterization of the preconditioner, learned online by hypergradients during optimization rather than by offline meta-training (Keskar et al., 2015, Ma, 2020, Liao et al., 2022).

The same preconditioning logic extends beyond deterministic optimization. Ensemble quasi-Newton HMC constructs an approximate inverse Hessian of the action from differences across an ensemble of Markov chains and inserts it as the mass matrix in the kinetic term of Hamiltonian dynamics. In the reported $\widetilde F_R(x)=F(G(x))$5 $\widetilde F_R(x)=F(G(x))$6D compact $\widetilde F_R(x)=F(G(x))$7 test, the method reduced autocorrelation cost for the average plaquette, but did not show improvement for topological charge (Jin et al., 2019).

7. Convergence properties, performance, and limitations

Across the literature, convergence guarantees range from Q-linear to Q-superlinear and Q-quadratic local behavior, finite termination on quadratics, and global non-asymptotic rates. In adaptive VQAs, Hessian recycling was associated with faster saturation of the line-search step size to $\widetilde F_R(x)=F(G(x))$8 and with convergence traces consistent with the condition

$\widetilde F_R(x)=F(G(x))$9

In inverse quNac, quadratic hereditary implies exact termination on a convex quadratic after a total of $p_k^{(\mathrm{QN})}=-H_k \nabla f(x_k),$0 inner CG steps. In AGQN and DAGQN, explicit global convergence bounds coexist with local superlinear rates; in nonlinear-preconditioned Newton methods, local superlinear and quadratic guarantees are phrased in terms of the preconditioned Hessian rather than the raw Hessian (Ramôa et al., 2024, Gower et al., 2014, Du et al., 2023, Bodard et al., 12 May 2026).

Reported computational benefits are similarly heterogeneous but consistent in direction. The VQA recycling scheme reduced measurement cost and classical simulation time, while nonlinear Schwarz-preconditioned L-BFGS and AA-I reduced iteration counts dramatically on the minimal surface problem and were roughly two times faster than standard L-BFGS in wall-clock time. On logistic SVM benchmarks, inverse quNac often converged in wall clock time before unpreconditioned Newton–CG. This suggests that the practical value of adaptive preconditioning is greatest when second-order information is expensive, search directions are repeatedly recomputed, or the problem geometry changes more slowly than the outer iteration index (Ramôa et al., 2024, Kothari, 2022, Gower et al., 2014).

Several recurrent cautions delimit the concept. First, an adaptive quasi-Newton Hessian preconditioner is not always an approximation of the original Hessian; it may target a preconditioned residual or a transformed optimality map. Second, positive definiteness is method-dependent: BFGS-style recycling, Apollo’s rectified diagonal update, and Bayesian eigenvalue barriers are designed to preserve or enforce positivity, whereas SR1-based methods intentionally allow indefiniteness to exploit negative curvature. Third, stochastic and ensemble methods can amplify noise or fail on certain observables; the Bayesian stochastic formulation identifies lower eigenvalue control as central to avoiding noise amplification, and ensemble QNHMC did not automatically cure topological slowing down. Finally, the benefit of recycling depends on how much the local curvature changes between successive subproblems; when the landscape changes substantially, a better initial Hessian matters less than the ability to track a difficult path (Kothari, 2022, Ma, 2020, Carlon et al., 2022, Jin et al., 2019, Ramôa et al., 2024).