Anderson Acceleration Overview

• Anderson Acceleration is a multisecant technique that improves convergence by extrapolating new iterates from a short history of residual differences.
• It offers both fixed-point and affine-combination formulations, showing equivalence with GMRES in linear scenarios and adaptive momentum properties in nonlinear contexts.
• Recent developments focus on robust stabilization, safeguard mechanisms, and structured variants to enhance performance in optimization, scientific computing, and machine learning.

Anderson acceleration is a multisecant mixing technique for accelerating fixed-point iterations of the form $x_{k+1}=G(x_k)$. Writing the residual as $F(x)=G(x)-x$, the method uses a short history of iterates or residual differences to solve a small least-squares problem and extrapolate a new iterate with reduced residual. In linear settings it is essentially equivalent to GMRES up to a stagnation index; in nonlinear settings it is closely connected to multisecant quasi-Newton updates, Pulay/DIIS mixing, and residual-minimizing extrapolation (Potra et al., 2011, Wan et al., 2024).

1. Core formulation and standard algorithmic forms

The common starting point is a fixed-point equation $x=G(x)$, with residual $r(x)=G(x)-x$. One widely used residual-difference form introduces

$$\Delta x_j=x_{j+1}-x_j,\qquad \Delta r_j=r_{j+1}-r_j,$$

and, for a window $m_k=\min(m,k)$,

$$X_k=[\Delta x_{k-m_k},\ldots,\Delta x_{k-1}],\qquad R_k=[\Delta r_{k-m_k},\ldots,\Delta r_{k-1}].$$

The coefficients are obtained from

$$\theta^{(k)}=\arg\min_{\theta\in\mathbb{R}^{m_k}}\|r_k-R_k\theta\|_2^2,$$

or, with Tikhonov regularization,

$$\theta_\lambda^{(k)}=\arg\min_\theta \|r_k-R_k\theta\|_2^2+\lambda\|\theta\|_2^2,\qquad \lambda\ge 0.$$

The corresponding update can be written as

$x_{k+1}=x_k+r_k-(X_k+R_k)\theta^{(k)}.$

This form is explicit in optimization-oriented analyses of Type-II AA and in several later implementations (Chen et al., 24 Dec 2025).

An alternative affine-combination form minimizes the norm of a linear combination of recent residuals subject to a sum-to-one constraint. For value iteration, for example, with residuals $F(x)=G(x)-x$0 and $F(x)=G(x)-x$1, the coefficients satisfy

$F(x)=G(x)-x$2

with closed form in the $F(x)=G(x)-x$3 norm

$F(x)=G(x)-x$4

and update

$F(x)=G(x)-x$5

Closely related constrained least-squares forms are also used in proximal-gradient adaptations of AA (Geist et al., 2018, Mai et al., 2019).

Across these formulations, the essential mechanism is the same: AA constructs a small local model from recent residual information, then chooses coefficients that minimize a residual surrogate in an affine subspace. The literature cited here uses both constrained affine-combination and residual-difference least-squares formulations; the distinction is often one of algebraic representation rather than underlying principle (Chen et al., 24 Dec 2025, Wan et al., 2024).

2. Linear-algebraic structure, Krylov equivalences, and sharp convergence characterizations

For linear problems, the behavior of AA can be characterized almost completely. Given a linear system $F(x)=G(x)-x$6, define the residual $F(x)=G(x)-x$7 and the grade $F(x)=G(x)-x$8 of the initial residual with respect to $F(x)=G(x)-x$9. If full-memory AA does not stagnate up to $x=G(x)$0, then its iterates converge to the exact solution; if it stagnates earlier, then it converges to the wrong solution. Up to the index where AA begins to stagnate, AA and GMRES are essentially equivalent, and this remains true for the optimized-mixing variant in which the mixing parameter is chosen to minimize the current residual (Potra et al., 2011).

This linear equivalence extends along several directions. In the CROP framework, CROP without truncation is equivalent to GMRES, while truncated CROP($x=G(x)$1) is equivalent to ORTHOMIN($x=G(x)$2); for symmetric matrices, CROP(1) coincides with MINRES and CROP(2) with CR. The same work establishes explicit equivalence relations between AA, CROP, and CROP-Anderson under matched initialization and no truncation, and it interprets both AA and CROP as multisecant Jacobian-free methods (Wan et al., 2024). A different structural reformulation, Anderson acceleration with truncated Gram-Schmidt (AATGS), is equivalent to classical AA in the linear case and acquires a three-term recurrence for symmetric linear problems. In that setting AATGS(3) is equivalent to full-depth AATGS, reducing storage from $x=G(x)$3 to $x=G(x)$4 and making the method akin to CG or CR in memory profile (Tang et al., 2024).

For quadratic optimization, AA admits stronger rate statements when the mixing is scheduled through Chebyshev polynomials. In particular, for $x=G(x)$5 with $x=G(x)$6, Anderson-Chebyshev acceleration achieves the optimal iteration complexity

$x=G(x)$7

improving the $x=G(x)$8 rate obtained for fixed-$x=G(x)$9 AA on quadratics. The method chooses

$r(x)=G(x)-x$0

which implements Chebyshev-optimal polynomial filtering of the iteration operator (Li et al., 2018).

3. Optimization interpretations and continuous-time dynamics

When AA is applied to gradient-descent fixed-point maps,

$r(x)=G(x)-x$1

its inertial structure becomes explicit. Introducing the auxiliary gradient step

$r(x)=G(x)-x$2

Type-II AA can be rewritten exactly as

$r(x)=G(x)-x$3

with adaptive momentum coefficients $r(x)=G(x)-x$4 obtained from the mixing coefficients $r(x)=G(x)-x$5. This representation shows AA as an adaptive multistep momentum method rather than merely a heuristic residual mixer (Chen et al., 24 Dec 2025).

Under inertial scaling $r(x)=G(x)-x$6 and $r(x)=G(x)-x$7, the same analysis yields a second-order high-resolution ODE with variable effective mass,

$r(x)=G(x)-x$8

The effective mass is

$r(x)=G(x)-x$9

while $$\Delta x_j=x_{j+1}-x_j,\qquad \Delta r_j=r_{j+1}-r_j,$$0 is determined by the first-moment consistency deviation. The $$\Delta x_j=x_{j+1}-x_j,\qquad \Delta r_j=r_{j+1}-r_j,$$1 term acts as an implicit Hessian-driven damping and stabilizes stiff directions. A Lyapunov analysis gives

$$\Delta x_j=x_{j+1}-x_j,\qquad \Delta r_j=r_{j+1}-r_j,$$2

so unchecked growth in effective mass acts as negative damping and injects energy into the dynamics. This analysis differentiates AA sharply from the continuous limits of Nesterov acceleration and heavy-ball momentum, which have constant mass and fixed or time-decaying friction (Chen et al., 24 Dec 2025).

A complementary optimization analysis quantifies the one-step improvement of AA over gradient descent through the projection operator

$$\Delta x_j=x_{j+1}-x_j,\qquad \Delta r_j=r_{j+1}-r_j,$$3

and the gain factor

$$\Delta x_j=x_{j+1}-x_j,\qquad \Delta r_j=r_{j+1}-r_j,$$4

For quadratic objectives, AA-GD satisfies

$$\Delta x_j=x_{j+1}-x_j,\qquad \Delta r_j=r_{j+1}-r_j,$$5

which isolates the AA contribution as a multiplicative contraction beyond the baseline gradient-descent factor (Liu et al., 2022).

4. Stability mechanisms, safeguards, and numerical linear algebra

The practical effectiveness of AA depends heavily on stabilization. One explicit negative result is that native AA does not, in general, admit a global convergence guarantee for proximal-gradient methods: a smooth strongly convex one-dimensional counterexample yields cycling for AA-GD with $$\Delta x_j=x_{j+1}-x_j,\qquad \Delta r_j=r_{j+1}-r_j,$$6. To recover global behavior, a guarded AA-PGA scheme accepts an accelerated step only if it satisfies the descent condition

$$\Delta x_j=x_{j+1}-x_j,\qquad \Delta r_j=r_{j+1}-r_j,$$7

and otherwise falls back to the proximal-gradient step. Under strong convexity and smoothness, this preserves the global linear rate of proximal gradient while retaining AA’s local acceleration (Mai et al., 2019).

Monotonicity-based accept/reject rules recur in other domains. In accelerated Lloyd iterations for $$\Delta x_j=x_{j+1}-x_j,\qquad \Delta r_j=r_{j+1}-r_j,$$8-means, the accelerated centroid iterate is accepted only if it decreases the $$\Delta x_j=x_{j+1}-x_j,\qquad \Delta r_j=r_{j+1}-r_j,$$9-means energy; otherwise the method reverts to the unaccelerated assignment-update step. That work also introduces a dynamic history rule based on

$m_k=\min(m,k)$0

with default thresholds $m_k=\min(m,k)$1, $m_k=\min(m,k)$2, and cap $m_k=\min(m,k)$3, decreasing or increasing the memory parameter according to recent energy reduction (Zhang et al., 2018). In geometry optimization and physics simulation, a similarly simple safeguard accepts an AA step only when it decreases the target energy, otherwise reverting to the local-global step; this guarantees a monotone energy sequence and preserves the global stability of the underlying solver (Peng et al., 2018).

Several stabilization mechanisms operate directly on the least-squares problem. Approximate-calculation variants analyze perturbations in the residual matrix and right-hand side, enforce monotonicity of the residual at AA steps, and propose reduced projected least-squares solves whose subspace dimension is adapted dynamically by computable heuristic indicators (Pasini et al., 2022). In stochastic deep learning, mini-batch noise degrades residual coherence; an adaptive moving average is therefore triggered when the relative standard deviation

$m_k=\min(m,k)$4

signals excessive oscillation, and AA steps are safeguarded by accepting them only when the residual norm decreases (Pasini et al., 2021). For perturbed Newton methods and Levenberg–Marquardt, $m_k=\min(m,k)$5-safeguarding scales or rejects the AA correction according to the ratio $m_k=\min(m,k)$6, guaranteeing local linear convergence in singular 2-regular settings and avoiding degradation of superlinear order when the base method is already fast (Dallas, 17 Aug 2025). In the dynamical framework, Energy-Guarded Anderson Acceleration (EG-AA) imposes a bound

$m_k=\min(m,k)$7

on effective-mass growth and adds geometric damping

$m_k=\min(m,k)$8

thereby enforcing the discrete analogue of the dissipation condition $m_k=\min(m,k)$9 (Chen et al., 24 Dec 2025).

5. Structured variants and algorithmic generalizations

A substantial body of work modifies AA’s internal representation rather than its external use. Preconditioned Anderson acceleration replaces the basic fixed-point map by

$$X_k=[\Delta x_{k-m_k},\ldots,\Delta x_{k-1}],\qquad R_k=[\Delta r_{k-m_k},\ldots,\Delta r_{k-1}].$$0

so that $$X_k=[\Delta x_{k-m_k},\ldots,\Delta x_{k-1}],\qquad R_k=[\Delta r_{k-m_k},\ldots,\Delta r_{k-1}].$$1 recovers Picard iteration, while $$X_k=[\Delta x_{k-m_k},\ldots,\Delta x_{k-1}],\qquad R_k=[\Delta r_{k-m_k},\ldots,\Delta r_{k-1}].$$2 recovers Newton or quasi-Newton behavior. In this framework AA acts on the preconditioned residuals, and delayed updates of $$X_k=[\Delta x_{k-m_k},\ldots,\Delta x_{k-1}],\qquad R_k=[\Delta r_{k-m_k},\ldots,\Delta r_{k-1}].$$3 reduce Jacobian-related cost while preserving the acceleration effect (Chen et al., 2023).

The truncated Gram-Schmidt variant AATGS replaces repeated least-squares solves by a paired truncated orthogonalization of residual and iterate differences. In exact arithmetic and full depth, $$X_k=[\Delta x_{k-m_k},\ldots,\Delta x_{k-1}],\qquad R_k=[\Delta r_{k-m_k},\ldots,\Delta r_{k-1}].$$4 and $$X_k=[\Delta x_{k-m_k},\ldots,\Delta x_{k-1}],\qquad R_k=[\Delta r_{k-m_k},\ldots,\Delta r_{k-1}].$$5, so the method is equivalent to classical AA. For symmetric linear systems, the resulting basis obeys a three-term recurrence, and AATGS(3) is equivalent to full-depth AATGS, yielding an $$X_k=[\Delta x_{k-m_k},\ldots,\Delta x_{k-1}],\qquad R_k=[\Delta r_{k-m_k},\ldots,\Delta r_{k-1}].$$6-storage method (Tang et al., 2024).

Another recent direction eliminates restart. Adjusted Anderson Acceleration (AAA) maintains an approximate Jacobian $$X_k=[\Delta x_{k-m_k},\ldots,\Delta x_{k-1}],\qquad R_k=[\Delta r_{k-m_k},\ldots,\Delta r_{k-1}].$$7, performs Newton-like steps

$$X_k=[\Delta x_{k-m_k},\ldots,\Delta x_{k-1}],\qquad R_k=[\Delta r_{k-m_k},\ldots,\Delta r_{k-1}].$$8

and updates $$X_k=[\Delta x_{k-m_k},\ldots,\Delta x_{k-1}],\qquad R_k=[\Delta r_{k-m_k},\ldots,\Delta r_{k-1}].$$9 by a rank-one correction selected either greedily or randomly. The random version satisfies

$$\theta^{(k)}=\arg\min_{\theta\in\mathbb{R}^{m_k}}\|r_k-R_k\theta\|_2^2,$$0

an $$\theta^{(k)}=\arg\min_{\theta\in\mathbb{R}^{m_k}}\|r_k-R_k\theta\|_2^2,$$1-step super quadratic rate that, according to the cited work, is new for Anderson-type and quasi-Newton methods (Ye et al., 2024).

CROP and CROP-Anderson form another branch of development. CROP was introduced as an alternative to classical AA with less storage and, in the reported study, better performance. The same work establishes exact equivalence between CROP and GMRES without truncation, between truncated CROP($$\theta^{(k)}=\arg\min_{\theta\in\mathbb{R}^{m_k}}\|r_k-R_k\theta\|_2^2,$$2) and ORTHOMIN($$\theta^{(k)}=\arg\min_{\theta\in\mathbb{R}^{m_k}}\|r_k-R_k\theta\|_2^2,$$3), and between AA, CROP, and CROP-Anderson under the appropriate no-truncation identifications (Wan et al., 2024).

6. Application domains and empirical behavior

AA has been applied across a wide range of computational regimes. For scientific computing on function spaces, replacing the internal least-squares norm by an $$\theta^{(k)}=\arg\min_{\theta\in\mathbb{R}^{m_k}}\|r_k-R_k\theta\|_2^2,$$4 Sobolev norm biases AA toward low-frequency residual content. In particular, AA based on the $$\theta^{(k)}=\arg\min_{\theta\in\mathbb{R}^{m_k}}\|r_k-R_k\theta\|_2^2,$$5 norm is reported as well-suited for fixed-point operators derived from second-order elliptic differential operators, including Helmholtz, and it outperformed both $$\theta^{(k)}=\arg\min_{\theta\in\mathbb{R}^{m_k}}\|r_k-R_k\theta\|_2^2,$$6-based AA and restarted GMRES in the WaveHoltz experiments summarized in the cited work (Yang et al., 2020). In another scientific-computing line, AA with approximate calculations was analyzed rigorously for linear problems, and reduced projected least-squares variants were tested on Richardson iterations for benchmark matrices and on nonlinear time-dependent Boltzmann equations (Pasini et al., 2022).

In optimization and inverse problems, AA has been used to accelerate steepest descent for seismic full-waveform inversion and least-squares reverse-time migration. Treating steepest descent as the fixed-point map $$\theta^{(k)}=\arg\min_{\theta\in\mathbb{R}^{m_k}}\|r_k-R_k\theta\|_2^2,$$7, the method requires only a very low-dimensional least-squares problem in addition to gradient evaluations, and on Marmousi benchmarks it produced fast convergence and competitive reconstructions relative to quasi-Newton methods such as L-BFGS (Yang, 2020). In feasible sequential linear programming, AA($$\theta^{(k)}=\arg\min_{\theta\in\mathbb{R}^{m_k}}\|r_k-R_k\theta\|_2^2,$$8)-FSLP reduced the mean number of constraint evaluations from $$\theta^{(k)}=\arg\min_{\theta\in\mathbb{R}^{m_k}}\|r_k-R_k\theta\|_2^2,$$9 for FSLP to $$\theta_\lambda^{(k)}=\arg\min_\theta \|r_k-R_k\theta\|_2^2+\lambda\|\theta\|_2^2,\qquad \lambda\ge 0.$$0, $$\theta_\lambda^{(k)}=\arg\min_\theta \|r_k-R_k\theta\|_2^2+\lambda\|\theta\|_2^2,\qquad \lambda\ge 0.$$1, and $$\theta_\lambda^{(k)}=\arg\min_\theta \|r_k-R_k\theta\|_2^2+\lambda\|\theta\|_2^2,\qquad \lambda\ge 0.$$2 for $$\theta_\lambda^{(k)}=\arg\min_\theta \|r_k-R_k\theta\|_2^2+\lambda\|\theta\|_2^2,\qquad \lambda\ge 0.$$3, reduced the mean number of outer iterations from $$\theta_\lambda^{(k)}=\arg\min_\theta \|r_k-R_k\theta\|_2^2+\lambda\|\theta\|_2^2,\qquad \lambda\ge 0.$$4 to $$\theta_\lambda^{(k)}=\arg\min_\theta \|r_k-R_k\theta\|_2^2+\lambda\|\theta\|_2^2,\qquad \lambda\ge 0.$$5, $$\theta_\lambda^{(k)}=\arg\min_\theta \|r_k-R_k\theta\|_2^2+\lambda\|\theta\|_2^2,\qquad \lambda\ge 0.$$6, and $$\theta_\lambda^{(k)}=\arg\min_\theta \|r_k-R_k\theta\|_2^2+\lambda\|\theta\|_2^2,\qquad \lambda\ge 0.$$7, and reduced mean wall time from $$\theta_\lambda^{(k)}=\arg\min_\theta \|r_k-R_k\theta\|_2^2+\lambda\|\theta\|_2^2,\qquad \lambda\ge 0.$$8 seconds to $$\theta_\lambda^{(k)}=\arg\min_\theta \|r_k-R_k\theta\|_2^2+\lambda\|\theta\|_2^2,\qquad \lambda\ge 0.$$9, $x_{k+1}=x_k+r_k-(X_k+R_k)\theta^{(k)}.$0, and $x_{k+1}=x_k+r_k-(X_k+R_k)\theta^{(k)}.$1 seconds on a SCARA robot motion-planning benchmark (Kiessling et al., 2022).

In machine learning and data analysis, the range of applications is broad. For $x_{k+1}=x_k+r_k-(X_k+R_k)\theta^{(k)}.$2-means clustering, AA-based acceleration of Lloyd’s algorithm outperformed comparison methods in $x_{k+1}=x_k+r_k-(X_k+R_k)\theta^{(k)}.$3 out of $x_{k+1}=x_k+r_k-(X_k+R_k)\theta^{(k)}.$4 test cases, with mean computational time decrease ratio greater than $x_{k+1}=x_k+r_k-(X_k+R_k)\theta^{(k)}.$5 (Zhang et al., 2018). In reinforcement learning, Anderson-accelerated value iteration on Garnet problems $x_{k+1}=x_k+r_k-(X_k+R_k)\theta^{(k)}.$6 with $x_{k+1}=x_k+r_k-(X_k+R_k)\theta^{(k)}.$7 consistently sped up value-function convergence, although the cited study emphasizes that earlier improvement in value error need not translate immediately into better greedy-policy quality (Geist et al., 2018). In a bioinformatics classifier trained on sequence embeddings, a depth-1 Anderson-like update reduced the iterations to convergence for the Genome dataset with Spike2Vec embeddings from about $x_{k+1}=x_k+r_k-(X_k+R_k)\theta^{(k)}.$8 to about $x_{k+1}=x_k+r_k-(X_k+R_k)\theta^{(k)}.$9 (Ali et al., 2023). In stochastic deep learning, a stabilized alternating AA scheme with adaptive moving average was demonstrated on MLPs, PINNs for $F(x)=G(x)-x$00d and $F(x)=G(x)-x$01d Burgers equations, and ResNet50 on ImageNet1k, with large-scale experiments run on up to $F(x)=G(x)-x$02 NVIDIA V100 GPUs on Summit (Pasini et al., 2021).

Distributed optimization has also been a recent target. For constrained optimization over time-varying networks, FDGM-AA embeds local AA subroutines into pairwise Fenchel-dual subproblems and, with a safeguard scheme, achieves $F(x)=G(x)-x$03 convergence for the primal sequence and $F(x)=G(x)-x$04 convergence for the dual sequence (Liu et al., 18 Jan 2026).

7. Assumptions, limitations, and open directions

Most available theory remains local or problem-class specific. The dynamical analysis of AA as adaptive momentum with variable effective mass assumes smooth convex objectives with $F(x)=G(x)-x$05-Lipschitz gradients and, for some bounds, $F(x)=G(x)-x$06-Lipschitz Hessians (Chen et al., 24 Dec 2025). Local AA-PGA theory relies on strong local curvature of $F(x)=G(x)-x$07, twice epi-differentiability of $F(x)=G(x)-x$08, and bounded mixing coefficients (Mai et al., 2019). For perturbed Newton and Levenberg–Marquardt methods, the convergence analysis is formulated under 2-regularity, smoothness assumptions on $F(x)=G(x)-x$09, and a starlike local domain of convergence (Dallas, 17 Aug 2025). Approximate-calculation theory for AA is rigorous for linear problems and assumes full rank of the residual history matrices, explicitly excluding stagnation (Pasini et al., 2022).

Several open directions are identified in the cited works. For the energy-based dynamical program, principled automatic tuning of the hyperparameters $F(x)=G(x)-x$10 and $F(x)=G(x)-x$11 in EG-AA remains open, as does extending energy guarding to nonsmooth composite problems and operator-splitting methods such as ADMM (Chen et al., 24 Dec 2025). Anderson-Chebyshev establishes optimal rates for quadratics, but extending optimal acceleration guarantees to broader nonlinear classes remains challenging (Li et al., 2018). For AA with approximate calculations, rigorous nonlinear theory, truncated-memory analysis, and improved computable surrogates for the least-squares conditioning are still unresolved (Pasini et al., 2022). In stochastic and approximate settings, function-approximation noise, mini-batch variance, and distributability constraints limit the direct transfer of classical AA guarantees, which explains the central role of safeguards, damping, and projection-based reductions in recent variants (Pasini et al., 2021, Liu et al., 18 Jan 2026).

Taken together, these developments present Anderson acceleration not as a single algorithm but as a family of residual-minimizing multisecant procedures. Its classical fixed-point form, Krylov equivalences, momentum and ODE interpretations, safeguard mechanisms, and numerous structured variants make it a central acceleration framework spanning numerical linear algebra, optimization, scientific computing, machine learning, and distributed algorithms (Chen et al., 24 Dec 2025, Wan et al., 2024).