---
title: Anderson Acceleration Overview
url: https://www.emergentmind.com/topics/anderson-acceleration
type: topic
---

# Anderson Acceleration Overview

Anderson acceleration is a multisecant mixing technique for accelerating fixed-point iterations of the form \(x_{k+1}=G(x_k)\). Writing the residual as \(F(x)=G(x)-x\), the method uses a short history of iterates or residual differences to solve a small least-squares problem and extrapolate a new iterate with reduced residual. In linear settings it is essentially equivalent to GMRES up to a stagnation index; in nonlinear settings it is closely connected to multisecant quasi-Newton updates, Pulay/DIIS mixing, and residual-minimizing extrapolation [1102.0796][2410.03970].

## 1. Core formulation and standard algorithmic forms

The common starting point is a fixed-point equation \(x=G(x)\), with residual \(r(x)=G(x)-x\). One widely used residual-difference form introduces
\[
\Delta x_j=x_{j+1}-x_j,\qquad \Delta r_j=r_{j+1}-r_j,
\]
and, for a window \(m_k=\min(m,k)\),
\[
X_k=[\Delta x_{k-m_k},\ldots,\Delta x_{k-1}],\qquad
R_k=[\Delta r_{k-m_k},\ldots,\Delta r_{k-1}].
\]
The coefficients are obtained from
\[
\theta^{(k)}=\arg\min_{\theta\in\mathbb{R}^{m_k}}\|r_k-R_k\theta\|_2^2,
\]
or, with Tikhonov regularization,
\[
\theta_\lambda^{(k)}=\arg\min_\theta \|r_k-R_k\theta\|_2^2+\lambda\|\theta\|_2^2,\qquad \lambda\ge 0.
\]
The corresponding update can be written as
\[
x_{k+1}=x_k+r_k-(X_k+R_k)\theta^{(k)}.
\]
This form is explicit in optimization-oriented analyses of Type-II AA and in several later implementations [2512.21269].

An alternative affine-combination form minimizes the norm of a linear combination of recent residuals subject to a sum-to-one constraint. For value iteration, for example, with residuals \(\delta_i=T v_i-v_i\) and \(\Delta_k=[\delta_{k-m_k},\ldots,\delta_k]\), the coefficients satisfy
\[
\alpha^{k+1}=\arg\min_{\alpha\in\mathbb{R}^{m_k+1}}\|\Delta_k\alpha\|
\quad\text{subject to}\quad
\sum_{i=0}^{m_k}\alpha_i=1,
\]
with closed form in the \(\ell_2\) norm
\[
\alpha^{k+1}=\frac{(\Delta_k^\top\Delta_k)^{-1}\mathbf{1}}{\mathbf{1}^\top(\Delta_k^\top\Delta_k)^{-1}\mathbf{1}},
\]
and update
\[
v_{k+1}=\sum_{i=0}^{m_k}\alpha_i^{k+1}T v_{k-m_k+i}.
\]
Closely related constrained least-squares forms are also used in proximal-gradient adaptations of AA [1809.09501][1910.08590].

Across these formulations, the essential mechanism is the same: AA constructs a small local model from recent residual information, then chooses coefficients that minimize a residual surrogate in an affine subspace. The literature cited here uses both constrained affine-combination and residual-difference least-squares formulations; the distinction is often one of algebraic representation rather than underlying principle [2512.21269][2410.03970].

## 2. Linear-algebraic structure, Krylov equivalences, and sharp convergence characterizations

For linear problems, the behavior of AA can be characterized almost completely. Given a linear system \(Ax+b=0\), define the residual \(r(x)=Ax+b\) and the grade \(\nu(A,r_0)\) of the initial residual with respect to \(A\). If full-memory AA does not stagnate up to \(\nu(A,r_0)\), then its iterates converge to the exact solution; if it stagnates earlier, then it converges to the wrong solution. Up to the index where AA begins to stagnate, AA and GMRES are essentially equivalent, and this remains true for the optimized-mixing variant in which the mixing parameter is chosen to minimize the current residual [1102.0796].

This linear equivalence extends along several directions. In the CROP framework, CROP without truncation is equivalent to GMRES, while truncated CROP(\(m\)) is equivalent to ORTHOMIN(\(m-1\)); for symmetric matrices, CROP(1) coincides with MINRES and CROP(2) with CR. The same work establishes explicit equivalence relations between AA, CROP, and CROP-Anderson under matched initialization and no truncation, and it interprets both AA and CROP as multisecant Jacobian-free methods [2410.03970]. A different structural reformulation, Anderson acceleration with truncated Gram-Schmidt (AATGS), is equivalent to classical AA in the linear case and acquires a three-term recurrence for symmetric linear problems. In that setting AATGS(3) is equivalent to full-depth AATGS, reducing storage from \(O(nm)\) to \(O(n)\) and making the method akin to CG or CR in memory profile [2403.14961].

For quadratic optimization, AA admits stronger rate statements when the mixing is scheduled through Chebyshev polynomials. In particular, for \(f(x)=\tfrac12 x^\top A x-b^\top x\) with \(\mu I\preceq A\preceq LI\), Anderson-Chebyshev acceleration achieves the optimal iteration complexity
\[
O\!\left(\sqrt{\kappa}\,\ln\frac{1}{\epsilon}\right),
\qquad \kappa=L/\mu,
\]
improving the \(O(\kappa\ln(1/\epsilon))\) rate obtained for fixed-\(\beta\) AA on quadratics. The method chooses
\[
\beta_t=\frac{1}{\frac{L+\mu}{2}+\frac{L-\mu}{2}\cos\!\left(\frac{(2t-1)\pi}{2T}\right)},
\]
which implements Chebyshev-optimal polynomial filtering of the iteration operator [1809.02341].

## 3. Optimization interpretations and continuous-time dynamics

When AA is applied to gradient-descent fixed-point maps,
\[
x_{k+1}=x_k-h\nabla f(x_k)=:g(x_k),
\qquad r(x)=-h\nabla f(x),
\]
its inertial structure becomes explicit. Introducing the auxiliary gradient step
\[
y_{k+1}:=x_k-h\nabla f(x_k),
\]
Type-II AA can be rewritten exactly as
\[
x_{k+1}=y_{k+1}+\sum_{j=1}^{m}\gamma_k^{(j)}\big(y_{k+1}-y_{k-j+1}\big),
\]
with adaptive momentum coefficients \(\gamma_k^{(j)}\) obtained from the mixing coefficients \(\theta^{(k)}\). This representation shows AA as an adaptive multistep momentum method rather than merely a heuristic residual mixer [2512.21269].

Under inertial scaling \(t_k=k\sqrt{h}\) and \(h\to 0\), the same analysis yields a second-order high-resolution ODE with variable effective mass,
\[
M_{\mathrm{eff}}(t)\,\ddot x(t)+c(t)\,\dot x(t)+\nabla f(x(t))
+\sqrt{h}\,\beta(t)\nabla^2 f(x(t))\dot x(t)=0.
\]
The effective mass is
\[
M_{\mathrm{eff}}(t)=\tfrac12\Big(1+\sum_{j=1}^m j^2\gamma^{(j)}(t)\Big),
\]
while \(c(t)\) is determined by the first-moment consistency deviation. The \(\sqrt{h}\,\nabla^2 f(x)\dot x\) term acts as an implicit Hessian-driven damping and stabilizes stiff directions. A Lyapunov analysis gives
\[
\frac{d\mathcal E}{dt}
=
-\Big(c(t)-\tfrac12\dot M_{\mathrm{eff}}(t)\Big)\|\dot x(t)\|^2
-\sqrt{h}\,\langle \dot x(t),\nabla^2 f(x(t))\dot x(t)\rangle,
\]
so unchecked growth in effective mass acts as negative damping and injects energy into the dynamics. This analysis differentiates AA sharply from the continuous limits of Nesterov acceleration and heavy-ball momentum, which have constant mass and fixed or time-decaying friction [2512.21269].

A complementary optimization analysis quantifies the one-step improvement of AA over gradient descent through the projection operator
\[
\Pi_k=I-U_k(U_k^\top U_k+\lambda I)^{-1}U_k^\top
\]
and the gain factor
\[
\delta_k=\frac{\|\Pi_k\nabla f(x_k)\|}{\|\nabla f(x_k)\|}\le 1.
\]
For quadratic objectives, AA-GD satisfies
\[
\frac{\|\nabla f(x_{k+1})\|}{\|\nabla f(x_k)\|}
\le \delta_k(1-\eta\mu),
\]
which isolates the AA contribution as a multiplicative contraction beyond the baseline gradient-descent factor [2211.08578].

## 4. Stability mechanisms, safeguards, and numerical linear algebra

The practical effectiveness of AA depends heavily on stabilization. One explicit negative result is that native AA does not, in general, admit a global convergence guarantee for proximal-gradient methods: a smooth strongly convex one-dimensional counterexample yields cycling for AA-GD with \(m=1\). To recover global behavior, a guarded AA-PGA scheme accepts an accelerated step only if it satisfies the descent condition
\[
f(x_{\mathrm{test}})\le f(x_k)-\frac{\gamma}{2}\|\nabla f(x_k)\|_2^2,
\]
and otherwise falls back to the proximal-gradient step. Under strong convexity and smoothness, this preserves the global linear rate of proximal gradient while retaining AA’s local acceleration [1910.08590].

Monotonicity-based accept/reject rules recur in other domains. In accelerated Lloyd iterations for \(K\)-means, the accelerated centroid iterate is accepted only if it decreases the \(K\)-means energy; otherwise the method reverts to the unaccelerated assignment-update step. That work also introduces a dynamic history rule based on
\[
r^t=\frac{E^{t-1}-E^t}{E^{t-2}-E^{t-1}},
\]
with default thresholds \(\varepsilon_1=0.02\), \(\varepsilon_2=0.5\), and cap \(\bar m=30\), decreasing or increasing the memory parameter according to recent energy reduction [1805.10638]. In geometry optimization and physics simulation, a similarly simple safeguard accepts an AA step only when it decreases the target energy, otherwise reverting to the local-global step; this guarantees a monotone energy sequence and preserves the global stability of the underlying solver [1805.05715].

Several stabilization mechanisms operate directly on the least-squares problem. Approximate-calculation variants analyze perturbations in the residual matrix and right-hand side, enforce monotonicity of the residual at AA steps, and propose reduced projected least-squares solves whose subspace dimension is adapted dynamically by computable heuristic indicators [2206.03915]. In stochastic deep learning, mini-batch noise degrades residual coherence; an adaptive moving average is therefore triggered when the relative standard deviation
\[
\|\sqrt{S(\mathbf w^k)}\|_\infty > \varepsilon \|\mathbf r^k\|_\infty
\]
signals excessive oscillation, and AA steps are safeguarded by accepting them only when the residual norm decreases [2110.14813]. For perturbed Newton methods and Levenberg–Marquardt, \(\gamma\)-safeguarding scales or rejects the AA correction according to the ratio \(\eta_{k+1}=\|w_{k+1}\|/\|w_k\|\), guaranteeing local linear convergence in singular 2-regular settings and avoiding degradation of superlinear order when the base method is already fast [2508.12513]. In the dynamical framework, Energy-Guarded Anderson Acceleration (EG-AA) imposes a bound
\[
\Delta M_k=M_k-M_{k-1}\le \delta \sqrt{h}
\]
on effective-mass growth and adds geometric damping
\[
D_k=-\eta\sqrt{\beta}\,\Delta g_{k-1},
\]
thereby enforcing the discrete analogue of the dissipation condition \(c(t)\ge \tfrac12 \dot M_{\mathrm{eff}}(t)\) [2512.21269].

## 5. Structured variants and algorithmic generalizations

A substantial body of work modifies AA’s internal representation rather than its external use. Preconditioned Anderson acceleration replaces the basic fixed-point map by
\[
g_{M_k}(x)=x-M_k^{-1}f(x),
\]
so that \(M_k=I\) recovers Picard iteration, while \(M_k\approx J_f(x_k)\) recovers Newton or quasi-Newton behavior. In this framework AA acts on the preconditioned residuals, and delayed updates of \(M_k\) reduce Jacobian-related cost while preserving the acceleration effect [2310.04034].

The truncated Gram-Schmidt variant AATGS replaces repeated least-squares solves by a paired truncated orthogonalization of residual and iterate differences. In exact arithmetic and full depth, \(F_j=Q_jS_j\) and \(X_j=U_jS_j\), so the method is equivalent to classical AA. For symmetric linear systems, the resulting basis obeys a three-term recurrence, and AATGS(3) is equivalent to full-depth AATGS, yielding an \(O(n)\)-storage method [2403.14961].

Another recent direction eliminates restart. Adjusted Anderson Acceleration (AAA) maintains an approximate Jacobian \(B_k\), performs Newton-like steps
\[
x_{k+1}=x_k-B_k^{-1}F(x_k),
\]
and updates \(B_k\) by a rank-one correction selected either greedily or randomly. The random version satisfies
\[
\frac{\|x_{k+n+1}-x_*\|}{\|x_k-x_*\|^2}
=
\mathcal O\!\left(\left(1-\frac{1}{n}\right)^{kn}\right),
\]
an \(n\)-step super quadratic rate that, according to the cited work, is new for Anderson-type and quasi-Newton methods [2403.16734].

CROP and CROP-Anderson form another branch of development. CROP was introduced as an alternative to classical AA with less storage and, in the reported study, better performance. The same work establishes exact equivalence between CROP and GMRES without truncation, between truncated CROP(\(m\)) and ORTHOMIN(\(m-1\)), and between AA, CROP, and CROP-Anderson under the appropriate no-truncation identifications [2410.03970].

## 6. Application domains and empirical behavior

AA has been applied across a wide range of computational regimes. For scientific computing on function spaces, replacing the internal least-squares norm by an \(\mathcal H^{-s}\) Sobolev norm biases AA toward low-frequency residual content. In particular, AA based on the \(\mathcal H^{-2}\) norm is reported as well-suited for fixed-point operators derived from second-order elliptic differential operators, including Helmholtz, and it outperformed both \(L^2\)-based AA and restarted GMRES in the WaveHoltz experiments summarized in the cited work [2002.03694]. In another scientific-computing line, AA with approximate calculations was analyzed rigorously for linear problems, and reduced projected least-squares variants were tested on Richardson iterations for benchmark matrices and on nonlinear time-dependent Boltzmann equations [2206.03915].

In optimization and inverse problems, AA has been used to accelerate steepest descent for seismic full-waveform inversion and least-squares reverse-time migration. Treating steepest descent as the fixed-point map \(G(p)=p-\eta \nabla J(p)\), the method requires only a very low-dimensional least-squares problem in addition to gradient evaluations, and on Marmousi benchmarks it produced fast convergence and competitive reconstructions relative to quasi-Newton methods such as L-BFGS [2008.11778]. In feasible sequential linear programming, AA(\(d\))-FSLP reduced the mean number of constraint evaluations from \(448.84\) for FSLP to \(274.69\), \(196.57\), and \(193.98\) for \(d=1,5,15\), reduced the mean number of outer iterations from \(49.68\) to \(32.31\), \(27.62\), and \(27.70\), and reduced mean wall time from \(5.01\) seconds to \(2.914\), \(2.24\), and \(2.34\) seconds on a SCARA robot motion-planning benchmark [2212.02718].

In machine learning and data analysis, the range of applications is broad. For \(K\)-means clustering, AA-based acceleration of Lloyd’s algorithm outperformed comparison methods in \(106\) out of \(120\) test cases, with mean computational time decrease ratio greater than \(33\%\) [1805.10638]. In reinforcement learning, Anderson-accelerated value iteration on Garnet problems \(\mathrm G(|S|=100, |A|=4, b=3)\) with \(\gamma=0.99\) consistently sped up value-function convergence, although the cited study emphasizes that earlier improvement in value error need not translate immediately into better greedy-policy quality [1809.09501]. In a bioinformatics classifier trained on sequence embeddings, a depth-1 Anderson-like update reduced the iterations to convergence for the Genome dataset with Spike2Vec embeddings from about \(600\) to about \(300\) [2302.00347]. In stochastic deep learning, a stabilized alternating AA scheme with adaptive moving average was demonstrated on MLPs, PINNs for \(2\)d and \(100\)d Burgers equations, and ResNet50 on ImageNet1k, with large-scale experiments run on up to \(1{,}536\) NVIDIA V100 GPUs on Summit [2110.14813].

Distributed optimization has also been a recent target. For constrained optimization over time-varying networks, FDGM-AA embeds local AA subroutines into pairwise Fenchel-dual subproblems and, with a safeguard scheme, achieves \(O(1/\sqrt{k})\) convergence for the primal sequence and \(O(1/k)\) convergence for the dual sequence [2601.12398].

## 7. Assumptions, limitations, and open directions

Most available theory remains local or problem-class specific. The dynamical analysis of AA as adaptive momentum with variable effective mass assumes smooth convex objectives with \(L\)-Lipschitz gradients and, for some bounds, \(L_H\)-Lipschitz Hessians [2512.21269]. Local AA-PGA theory relies on strong local curvature of \(f\), twice epi-differentiability of \(g\), and bounded mixing coefficients [1910.08590]. For perturbed Newton and Levenberg–Marquardt methods, the convergence analysis is formulated under 2-regularity, smoothness assumptions on \(f''\), and a starlike local domain of convergence [2508.12513]. Approximate-calculation theory for AA is rigorous for linear problems and assumes full rank of the residual history matrices, explicitly excluding stagnation [2206.03915].

Several open directions are identified in the cited works. For the energy-based dynamical program, principled automatic tuning of the hyperparameters \(\delta_{\max}\) and \(\eta\) in EG-AA remains open, as does extending energy guarding to nonsmooth composite problems and operator-splitting methods such as ADMM [2512.21269]. Anderson-Chebyshev establishes optimal rates for quadratics, but extending optimal acceleration guarantees to broader nonlinear classes remains challenging [1809.02341]. For AA with approximate calculations, rigorous nonlinear theory, truncated-memory analysis, and improved computable surrogates for the least-squares conditioning are still unresolved [2206.03915]. In stochastic and approximate settings, function-approximation noise, mini-batch variance, and distributability constraints limit the direct transfer of classical AA guarantees, which explains the central role of safeguards, damping, and projection-based reductions in recent variants [2110.14813][2601.12398].

Taken together, these developments present Anderson acceleration not as a single algorithm but as a family of residual-minimizing multisecant procedures. Its classical fixed-point form, Krylov equivalences, momentum and ODE interpretations, safeguard mechanisms, and numerous structured variants make it a central acceleration framework spanning numerical linear algebra, optimization, scientific computing, machine learning, and distributed algorithms [2512.21269][2410.03970].

Source: https://www.emergentmind.com/topics/anderson-acceleration