---
title: Adaptive Quasi-Newton Hessian Preconditioner
url: https://www.emergentmind.com/topics/adaptive-quasi-newton-hessian-preconditioner
type: topic
---

# Adaptive Quasi-Newton Hessian Preconditioner

Searching arXiv for the core paper and closely related quasi-Newton preconditioning work to ground the article in published sources.
arXiv search query: 2401.05172
Adaptive quasi-Newton Hessian preconditioning denotes a class of methods that maintain and update an approximation of a Hessian, inverse Hessian, or closely related Jacobian, and use it to precondition optimization steps or linear solves as the problem, model, or active subspace changes. Contemporary instances range from adaptive variational quantum algorithms that recycle inverse Hessians across ansatz-growth steps, to nonlinear Schwarz-preconditioned limited-memory quasi-Newton schemes, Bayesian and probabilistic Hessian inference under noisy secant data, and Newton–CG preconditioners built from action constraints on Krylov subspaces [2401.05172] [2211.14403] [2208.00441] [1412.8045].

## 1. Conceptual scope

At the most general level, quasi-Newton preconditioning uses a matrix \(H_k\) or \(G_k^{-1}\) to reshape the gradient direction, as in \(x_{k+1}=x_k-\alpha_k H_k \nabla f(x_k)\) or \(x^{k+1}=x^k-\alpha^k (G^k)^{-1}\nabla f(x^k)\). In Newton–CG and PDE-constrained settings, the same idea appears as a left preconditioner for a sequence of symmetric linear systems or Hessian systems rather than as an explicit optimizer step. This suggests a unifying viewpoint: the preconditioner is an evolving second-order surrogate whose role is to encode curvature already revealed by prior iterates, Hessian-vector products, or local subproblem solves [2311.18210] [1412.8045] [2508.21155].

In nonlinear-preconditioned frameworks, the target of approximation is not always the raw Hessian. For the nonlinear restricted additive Schwarz strategy, the quasi-Newton matrix approximates the Jacobian of the preconditioned residual \(\widetilde F_L(x)=x-G(x)\) or \(\widetilde F_R(x)=F(G(x))\), and standard secant pairs must therefore be replaced by preconditioned ones. This is why some authors characterize the construction more precisely as an adaptive approximation built on top of a nonlinear Schwarz preconditioner, rather than as a fixed Hessian preconditioner for the original objective [2211.14403].

## 2. Algebraic foundations

In ordinary BFGS, the quasi-Newton search direction is
\[
p_k^{(\mathrm{QN})}=-H_k \nabla f(x_k),
\]
with inverse-Hessian update
\[
H_{k+1}=(I-\rho_k s_k y_k^T)H_k(I-\rho_k y_k s_k^T)+\rho_k s_k s_k^T,\qquad \rho_k=\frac{1}{y_k^T s_k},
\]
where
\[
s_k=x_{k+1}-x_k,\qquad y_k=\nabla f(x_{k+1})-\nabla f(x_k).
\]
Canonical restarts typically set \(H_0=I\), so all previously learned curvature information is discarded when a new optimization begins [2401.05172].

Generalizations replace the single secant equation by multisecant or subspace action constraints. Using \(A=XC\) and \(D=GC\), generalized multisecant conditions are \(BA=D\) or \(HD=A\), while the inverse quNac formulation enforces \(H_{k+1}(Q_{k+1}\mathcal S_k)=\mathcal S_k\). These constructions are derived from least-change or regularized least-squares principles and make BFGS and DFP special cases of broader symmetric or action-constrained updates [2011.03358] [1412.8045].

When Hessian samples \(Y=\nabla^2 f(x)S\) are available from forward-mode AD, block SR1 and block PSB updates impose the inverse secant condition \(H_{k+1}V=U\) on a sampled subspace, permit indefinite approximations, and feed a trust-region QCQP rather than a line-search step. A plausible implication is that adaptive preconditioning need not be tied to positive-definite line-search geometry; it can also be designed to preserve negative curvature information [2201.02608].

## 3. Recycling across related subproblems

In adaptive VQAs, the clearest recycling mechanism appears when an \((n-1)\)-parameter optimum \(x_{(n-1)}^*\) and inverse Hessian approximation \(H_{n-1\times n-1}^*\) are transferred to the next iteration through
\[
x_0 \gets \{x_{(n-1)}^*,0\},\qquad
H_0 \gets
\begin{pmatrix}
H_{n-1\times n-1}^* & 0\\
0 & 1
\end{pmatrix}.
\]
BFGS then proceeds normally, but the final inverse-Hessian state is saved before the convergence check and reused after ansatz growth. The resulting algorithm has the flavor of a continuous optimization where the dimension of the search space is augmented when the gradient norm falls below a threshold; in simulation, Hessian recycling often achieved superlinear convergence, reduced VQE-subroutine measurement cost by roughly one order of magnitude, yielded about \(70\text{–}90\%\) reduction in optimization measurement cost in many cases, and achieved up to \(84\%\) reduction in total function evaluations for some qubit-pool cases [2401.05172].

The same continuity principle appears in slowly changing Newton systems and parameterized inverse problems. In inverse quNac, Krylov directions collected while solving \(Q_k d_k=b_k\) are converted into an estimate \(H_k\approx Q_k^{-1}\) and used as a left preconditioner for the next Newton–CG or PCG call, exploiting the fact that the matrices are “slowly changing.” In pseudo-time continuation for PDE-constrained inverse problems, a parameter-aware BFGS update uses \(z_k=m_k-m_{k-1}\) and
\[
y_k=\nabla_m J(m_k,\theta_k)-\nabla_m J(m_{k-1},\theta_{k-1})-\Delta t\,B(m_{k-1},\theta_{k-1})\dot\theta,
\]
while an additional block BFGS update is extracted from PCG search directions; the reported method outperformed a direct optimization solve for the perturbed problem in total linear-solve cost [1412.8045] [2508.21155].

## 4. Nonlinear and probabilistic formulations

Nonlinear Schwarz preconditioning changes the residual map before quasi-Newton updating. The left-preconditioned residual is \(\widetilde F_L(x)=x-G(x)\), the right-preconditioned residual is \(\widetilde F_R(x)=F(G(x))\), and the secant pairs become
\[
s_L^{(k)}=x^{(k+1)}-x^{(k)},\qquad
y_L^{(k)}=\widetilde F_L(x^{(k+1)})-\widetilde F_L(x^{(k)}),
\]
or
\[
s_R^{(k)}=x^{(k+1)}-x^{(+)},\qquad
y_R^{(k)}=F(x^{(k+1)})-F(x^{(+)}).
\]
The quasi-Newton matrix therefore approximates the Jacobian of the transformed residual, not directly \(\nabla^2\Psi(x)\) [2211.14403].

A related but more general nonlinear-preconditioning viewpoint applies Newton’s method to
\[
g(x)=\nabla\phi^*(\nabla f(x)),
\]
with preconditioned Hessian
\[
H(x)=\nabla^2\phi^*(\nabla f(x))\,\nabla^2 f(x).
\]
The resulting smoothness assumption is Lipschitz continuity of \(H\), not of \(\nabla^2 f\). Under this model, local Q-superlinear and Q-quadratic convergence are established, a regularized isotropic variant attains \(O(\varepsilon^{-3/2})\) iteration complexity, and an adaptive version removes the need to know the smoothness constant and allows inexact subproblem solutions while preserving the same complexity order [2605.12666].

In stochastic settings, several papers replace exact secant matching by probabilistic inference. One Bayesian formulation computes a MAP estimator \(B_k^*=\arg\min_B L_k(B;\beta)\) from noisy secant pairs, weighted precision matrices, a Frobenius prior centered at the previous Hessian estimate, and log-barrier eigenvalue constraints; the inverse Hessian is then used to precondition SGD, with the strongest gains reported on ill-conditioned problems and with the lower eigenvalue bound playing a central role in preventing noise amplification. Another probabilistic approach infers a low-rank Hessian from noisy Hessian-vector products and turns it into a preconditioner \(P\), while noting that in very high-dimensional problems the construction effectively becomes an automatic learning-rate adaptation scheme [2208.00441] [1902.07557].

## 5. Indefinite, cubic-regularized, and sampled-curvature variants

Some adaptive quasi-Newton preconditioners deliberately abandon positive-definite line-search geometry. Using SIMD-parallel forward-mode AD, block Hessian samples \(Y=\nabla^2 f(x)S\) can be generated whenever the gradient is evaluated, and block SR1 or block PSB updates can enforce the inverse secant condition \(H_{k+1}V=U\) on sampled directions. The step is then computed from a moderate-dimensional trust-region QCQP in the span of the gradient, the quasi-Newton direction, and the sampled curvature directions, allowing negative curvature to be exploited rather than filtered away [2201.02608].

In deep learning, this strategy reappears in limited-memory SR1 with adaptive cubic regularization. The SR1 update
\[
B_{k+1}=B_k+\frac{(y_k-B_k s_k)(y_k-B_k s_k)^\top}{s_k^\top (y_k-B_k s_k)}
\]
is accepted only when
\[
\left|s_k^\top (y_k-B_k s_k)\right|>\varepsilon \|s_k\|_2 \|y_k-B_k s_k\|_2,
\]
and the cubic model is solved in closed form after transforming to the eigenbasis of \(B_k\). Because \(B_k\) is not required to be positive definite, the method can represent and use directions of negative curvature [2502.12298].

Cubic regularization has also been used to globalize classical quasi-Newton directions without abandoning the quasi-Newton geometry. In the Cubically Enhanced Quasi-Newton formulation, \(x_{k+1}=x_k-\eta_k B_k \nabla f(x_k)\) is recovered from a local model with quadratic and cubic terms measured in the \(B_k\)-norm, and the explicit stepsize is
\[
\eta_k=\frac{2}{\theta+\sqrt{\theta^2+4L\|\nabla f(x_k)\|_{B_k}^*}}.
\]
Under a relative inexactness condition for the Hessian approximation, the method has a global \(O(1/k)\) rate for convex functions, improves to \(O(1/k^2)\) when the inexactness is controlled within a prescribed relative accuracy, and admits an adaptive variant that adjusts to the function’s curvature while retaining the global convergence guarantees of the non-adaptive algorithm [2508.19712].

## 6. Large-scale, distributed, and application-specific implementations

Large-scale implementations emphasize cheap curvature acquisition, bounded memory, and communication efficiency. The adaptive greedy quasi-Newton method maintains \(G^k\approx \nabla^2 f(x^k)\), updates it through multiple greedy quasi-Newton refinements based on Hessian-vector products, and uses
\[
x^{k+1}=x^k-\alpha^k (G^k)^{-1}\nabla f(x^k)
\]
with an explicit adaptive stepsize chosen so that the Hessian approximation error remains uniformly bounded. Its master–worker extension DAGQN communicates \(O((\tau+1)n)\) scalars per worker per iteration, a cost described as comparable with that of first-order methods. Related action-constrained inverse quNac updates were implemented in full-memory and limited-memory forms and described as the first parallel quasi-Newton preconditioners [2311.18210] [1412.8045].

Other large-scale variants trade matrix richness against robustness. adaQN trains RNNs with a stochastic L-BFGS update, an Adagrad-like diagonal initialization
\[
[H_k^{(0)}]_{ii}=\frac{1}{\sqrt{\sum_{j=0}^k[\hat{\nabla}f(w_j)]_i^2+\epsilon}},
\]
aggregated curvature pairs, and memory flushing when a monitoring-set test deteriorates. Apollo uses a parameter-wise weak secant condition to update a diagonal Hessian approximation, rectifies it by \(D_t=\max(|B_t|,\sigma)\), and thereby retains linear time and memory complexity. LODO replaces analytic update formulas by a neural-network parameterization of the preconditioner, learned online by hypergradients during optimization rather than by offline meta-training [1511.01169] [2009.13586] [2210.06171].

The same preconditioning logic extends beyond deterministic optimization. Ensemble quasi-Newton HMC constructs an approximate inverse Hessian of the action from differences across an ensemble of Markov chains and inserts it as the mass matrix in the kinetic term of Hamiltonian dynamics. In the reported \(24\times 24\) \(2\)D compact \(U(1)\) test, the method reduced autocorrelation cost for the average plaquette, but did not show improvement for topological charge [1904.10039].

## 7. Convergence properties, performance, and limitations

Across the literature, convergence guarantees range from Q-linear to Q-superlinear and Q-quadratic local behavior, finite termination on quadratics, and global non-asymptotic rates. In adaptive VQAs, Hessian recycling was associated with faster saturation of the line-search step size to \(1\) and with convergence traces consistent with the condition
\[
\lim_{k\to\infty}\frac{\|(B_k-\nabla^2 f(x^*))p_k\|_F}{\|p_k\|_F}=0.
\]
In inverse quNac, quadratic hereditary implies exact termination on a convex quadratic after a total of \(n\) inner CG steps. In AGQN and DAGQN, explicit global convergence bounds coexist with local superlinear rates; in nonlinear-preconditioned Newton methods, local superlinear and quadratic guarantees are phrased in terms of the preconditioned Hessian rather than the raw Hessian [2401.05172] [1412.8045] [2311.18210] [2605.12666].

Reported computational benefits are similarly heterogeneous but consistent in direction. The VQA recycling scheme reduced measurement cost and classical simulation time, while nonlinear Schwarz-preconditioned L-BFGS and AA-I reduced iteration counts dramatically on the minimal surface problem and were roughly two times faster than standard L-BFGS in wall-clock time. On logistic SVM benchmarks, inverse quNac often converged in wall clock time before unpreconditioned Newton–CG. This suggests that the practical value of adaptive preconditioning is greatest when second-order information is expensive, search directions are repeatedly recomputed, or the problem geometry changes more slowly than the outer iteration index [2401.05172] [2211.14403] [1412.8045].

Several recurrent cautions delimit the concept. First, an adaptive quasi-Newton Hessian preconditioner is not always an approximation of the original Hessian; it may target a preconditioned residual or a transformed optimality map. Second, positive definiteness is method-dependent: BFGS-style recycling, Apollo’s rectified diagonal update, and Bayesian eigenvalue barriers are designed to preserve or enforce positivity, whereas SR1-based methods intentionally allow indefiniteness to exploit negative curvature. Third, stochastic and ensemble methods can amplify noise or fail on certain observables; the Bayesian stochastic formulation identifies lower eigenvalue control as central to avoiding noise amplification, and ensemble QNHMC did not automatically cure topological slowing down. Finally, the benefit of recycling depends on how much the local curvature changes between successive subproblems; when the landscape changes substantially, a better initial Hessian matters less than the ability to track a difficult path [2211.14403] [2009.13586] [2208.00441] [1904.10039] [2401.05172].

Source: https://www.emergentmind.com/topics/adaptive-quasi-newton-hessian-preconditioner