---
title: Generalized Alternating Anderson Acceleration
url: https://www.emergentmind.com/topics/generalized-alternating-anderson-acceleration
type: topic
---

# Generalized Alternating Anderson Acceleration

Generalized Alternating Anderson Acceleration denotes a class of fixed-point acceleration schemes that interleave plain fixed-point iterations with Anderson-accelerated updates in a periodic or composite pattern. In its most explicit formulation, it is “a periodic scheme composed of \(t\) fixed-point iteration steps, interleaved with \(s\) steps of Anderson acceleration with window size \(m\),” and it generalizes both windowed Anderson acceleration and alternating Anderson acceleration [2508.10158]. Closely related formulations include Alternating Anderson–Picard for nonlinear fixed-point problems, alternating Anderson–Richardson for linear systems, composite dynamic-window AA, and problem-specific extensions using preconditioning, sketching, or active-manifold structure [2407.10472].

## 1. Conceptual scope and historical lineages

The central object is a fixed-point map \(q\) or \(G\), with iteration \(x_{k+1}=q(x_k)\) and residual \(r(x)=q(x)-x\). Standard Anderson acceleration applies a least-squares mixing step at every iteration. Generalized alternating variants relax that schedule: they insert AA only periodically, or compose AA modules with different window sizes, damping rules, or baseline iterations. This periodicity is the defining structural distinction from stationary or windowed AA [2508.10158].

Several lineages converge on this viewpoint. Alternating Anderson–Richardson (AAR) alternates \(p-1\) preconditioned Richardson steps with one Anderson step, using history matrices of iterate and residual differences; with complete history, it is equivalent to GMRES without restart in exact arithmetic, while retaining a low-communication profile on parallel architectures [1606.08740]. Alternating Anderson–Picard (AAP) performs \(m\) Picard iterations followed by one AA step and is explicitly identified as “an instance of generalized alternating AA,” with memory length \(m\), number of baseline steps \(p=m\), damping \(\beta\), and optional restarts or filtering [2407.10472]. A further expansion appears in composite non-stationary AA, where one alternates AA modules with different window sizes and optimized damping by additive or multiplicative composition; that paper states that these schemes “can be viewed as a generalized alternating Anderson acceleration” [2203.14627].

The term also appears in nonsmooth optimization, where a “Generalized Alternating Anderson Acceleration” template is used for composite maps or stacked block updates once an active manifold has been identified, so that the locally relevant iteration becomes \(C^2\) with Lipschitz Jacobian [2410.09420]. More recent extensions include two-level sketching AAP for multiphysics PDEs [2505.08587] and preconditioned-alternating Anderson acceleration for generalized Sylvester equations [2604.15766]. This body of work suggests that the unifying idea is not a single algorithmic formula but a design principle: alternate inexpensive baseline iterations with selective multisecant corrections.

## 2. Fixed-point formulation and periodic schedules

In the general fixed-point setting, one solves \(x=q(x)\) with residual \(r(x)=q(x)-x\). The generalized periodic schedule uses three primary parameters: the AA window size \(m\), the number \(t\) of plain fixed-point steps per cycle, and the number \(s\) of AA steps per cycle. One cycle has length \(p=s+t\), and the iteration rule is periodic: if \(\operatorname{mod}(k-1,s+t)<t\), perform a plain fixed-point step; otherwise perform an AA step [2508.10158].

For the type-I Walker–Ni formulation used in that paper, AA\((m)\) computes coefficients \(\gamma^{(k)}\) from a least-squares problem on residual differences,
\[
\min_{\gamma\in\mathbb{R}^{m_k}}
\left\| r(x_k)+\sum_{i=1}^{m_k}\gamma_i^{(k)}\big[r(x_k)-r(x_{k-i})\big]\right\|_2^2,
\]
with update
\[
x_{k+1}=q(x_k)+\sum_{i=1}^{m_k}\gamma_i^{(k)}\big[q(x_k)-q(x_{k-i})\big],
\qquad m_k=\min\{m,k\}.
\]
The MATLAB-like pseudocode maintains rolling matrices of residuals and map evaluations to assemble the least-squares matrices \(B_k\) and \(C_k\) efficiently, and then either accepts the plain iterate \(\bar x_k=q(x_{k-1})\) or the accelerated iterate \(\bar x_k+C_k\gamma^{(k)}\) depending on the phase of the cycle [2508.10158].

AAP specializes this structure to nonlinear fixed-point problems \(x=g(x)\) with \(F(x)=g(x)-x\). At global iteration \(t\), it performs \(m\) Picard steps
\[
x_t^{0}:=x_t,\qquad x_t^{\ell}:=g(x_t^{\ell-1}),\quad \ell=1,\dots,m,
\]
then solves a constrained least-squares problem over coefficients \(\alpha_t\) satisfying \(\sum_{\ell=0}^{m}\alpha_\ell=1\), and updates
\[
x_{t+1}=(1-\beta_t)\sum_{\ell=0}^{m}\alpha_\ell x_t^\ell+\beta_t\sum_{\ell=0}^{m}\alpha_\ell g(x_t^\ell).
\]
The corresponding difference matrices \(S_t\) and \(Y_t\) reduce the constrained problem to an unconstrained least-squares in \(z_t\), and the resulting AA step can be written as a quasi-Newton update \(x_{t+1}=x_t-H_t f_t\) [2407.10472].

The main special cases are straightforward.

| Variant | Schedule | Relation |
|---|---|---|
| AA\((m)\) | AA at every iteration | recovered by \(t=0\) |
| restarted AA(1) | one FP step, one AA step, window \(m=1\) | \(m=s=t=1\) |
| alternating Anderson with one AA per cycle | \(s=1\), general \(t\) | fits GAAA but differs step-by-step from prior AAR schedules |
| AAP\((m)\) | \(m\) Picard steps, then one AA step | generalized alternating AA with \(p=m\) |
| AAR | \(p-1\) Richardson steps, then one Anderson step | generalized AAJ with preconditioning |

Composite non-stationary schemes widen the schedule design space further. Additive composition computes one AA\((m)\) proposal and one AA\((n)\) proposal from the same iterate and averages them, while multiplicative composition uses AA\((m)\) in an outer loop and AA\((n)\) in a short inner loop; fully non-stationary versions replace one module by AAoptD, in which the damping \(\beta_k\) is selected by a residual-minimization model [2203.14627].

## 3. Multisecant, Krylov, and Newton interpretations

A decisive feature of generalized alternating AA is that its apparently ad hoc periodic mixing admits precise multisecant and Krylov interpretations. In AA-II form, with
\[
\Delta X_k=[x_{k-m+1}-x_{k-m},\dots,x_k-x_{k-1}],\qquad
\Delta R_k=[r_{k-m+1}-r_{k-m},\dots,r_k-r_{k-1}],
\]
the update
\[
s_k=-r_k-\Delta X_k\gamma_k,\qquad x_{k+1}=x_k+\beta s_k
\]
has a multisecant quasi-Newton representation
\[
H_k=-\beta I+(\Delta X_k+\beta\Delta R_k)\Delta R_k^\dagger,
\qquad H_k\Delta R_k=\Delta X_k,
\qquad s_k=-H_k r_k
\]
[2407.10472].

For AAP, the multisecant structure becomes more explicit. With \(S_t=[s_t^0,\dots,s_t^{m-1}]\) and \(Y_t=[y_t^0,\dots,y_t^{m-1}]\), the approximate inverse Jacobian
\[
H_t=-\beta_t I+(S_t+\beta_t Y_t)Y_t^\dagger
\]
satisfies the inverse multisecant condition \(H_tY_t=S_t\). Equivalently, one may define a direct multisecant matrix \(B_t\) by \(B_tS_t=Y_t\), so that the AA direction is precisely a quasi-Newton step on a multisecant linearization [2407.10472].

The same paper proves that AAP is equivalent to solving a multisecant linear system with GMRES at each AA iteration. For any \(B_t\) in the multisecant set \(\{B\in\mathbb{R}^{d\times d}\mid BS_t=Y_t\}\), there exists a vector \(p_t\in K_m(B_t,f_t)\) solving
\[
p_t=\arg\min_{p\in K_m(B_t,f_t)}\|B_t p-f_t\|_2,
\]
such that
\[
\sum_{\ell=0}^{m}\alpha_\ell x_t^\ell = x_t-p_t,
\qquad
x_{t+1}=g_t(x_t-p_t),
\]
with \(g_t\) a damped linearized Picard map. The constrained LS over \(\alpha\), the unconstrained LS over \(z_t\), and the Krylov residual minimization are therefore equivalent descriptions of the same update [2407.10472].

In the small-residual regime, AAP approaches Newton–GMRES. Under Lipschitz differentiability of \(g'\) and rank and conditioning assumptions on \(S_t\) and \(Y_t\), the closest multisecant matrix \(B_t^*\) converges to the Jacobian \(J_t=f'(x_t)\), the Krylov blocks \(S_t\) and \(Y_t\) converge to their Jacobian-generated limits, the search direction \(p_t\) converges to the Newton–GMRES direction \(p_t^N\), and the AAP optimization gain
\[
\theta_t=\frac{\|(I-P_{Y_t})f_t\|}{\|f_t\|}
\]
converges to the normalized Jacobian–GMRES residual [2407.10472].

Linear alternating schemes admit related Krylov interpretations. For \(s=1\), GAAA applied to Richardson iteration has a periodic GMRES equivalence: if \(p=t+1\) and GMRES residuals strictly decrease up to \(k_0\), then \(x_{jp}=q(x_{jp-1}^G)\) for \(jp<k_0\). When \(t=0\), this reduces to the classical AA\((\infty)\)–GMRES equivalence. AAR with complete history is likewise equivalent to GMRES without restart in exact arithmetic [2508.10158, 1606.08740].

## 4. Convergence theory

The convergence theory of generalized alternating AA is stratified by problem class. In the linear contractive case, the theory is simplest: if the Richardson matrix \(M\) satisfies \(\|M\|=c<1\), then for any initial \(x_0\), GAAA \(aAA(m)[s]\)–FP\([t]\) converges, and every step—plain or accelerated—satisfies
\[
\|r_{k+1}\|\le c\|r_k\|,
\qquad
\|r_k\|\le c^k\|r_0\|\to 0
\]
[2508.10158].

For diagonalizable but noncontractive Richardson iterations, the same paper derives Chebyshev-based one-step and per-cycle bounds. If the spectrum of \(M\) lies in \([a,b]\subset\mathbb{R}\) not containing \(0\) or \(1\), then an AA\((m)\) step satisfies
\[
\|r_{k+1}\|\le C(a,b,m)\kappa_2(W)\|M\|\cdot \|r_k\|,
\]
where
\[
C(a,b,m)=\left|T_m\!\left(\frac{2ab-a-b}{b-a}\right)\right|^{-1}.
\]
For GAAA \(aAA(m)[s]\)–FP\([t]\) with \(m\le t\), the residuals at cycle endpoints obey
\[
\|r_{jp}\|\le [C(a,b,m)\kappa_2(W)]^j\|M\|^{jp}\|r_0\|.
\]
A sufficient convergence condition is then obtained even when the base fixed-point iteration is divergent but diagonalizable [2508.10158].

For nonlinear fixed-point problems, AAP furnishes the most detailed local theory. Under Assumption 1.1—\(g\) nonexpansive with Lipschitz \(g'\)—together with full-rank and conditioning assumptions on \(S_t\) and \(Y_t\), the one-step residual bound is
\[
\|f_{t+1}\|
\le
[(1-\beta_t)+\beta_t K]\theta_t\|f_t\|
+
\big(\gamma\sqrt{1-\theta_t^2}\|B_t^{-1}\|+\|E_t\|\big)\|f_t\|^2.
\]
If \(g\) is a contraction, \(\beta_t\ge \bar\beta>0\), and
\[
\sup_t[(1-\beta_t)+\beta_t K]\theta_t<p<1,
\]
then AAP achieves local linear residual convergence,
\[
\|f_{t+1}\|\le p\|f_t\|,
\qquad
\|x_t-x^*\|\le (1+K)p^t\|f_0\|,
\]
and it satisfies an inexact Newton condition with forcing term
\[
\eta_t=[(1-\beta_t)+\beta_t K]\theta_t+O(\|f_t\|).
\]
A sufficient local convergence condition is \(\sup_t\eta_t<1\) [2407.10472].

For nonsmooth problems, the local theory proceeds through active-manifold identification. Once an algorithm such as PPA, PGA, PLA, cyclic PCD, DRS/ADMM, or IRL1 identifies an active manifold \(\mathcal M\), the fixed-point map becomes smooth locally by sensitivity analysis. Under local contraction and bounded AA coefficients, AA then achieves R-linear local convergence:
\[
\|r(x_k)\|\le \hat\gamma^k\|r(x_0)\|,
\qquad
\|x_k-x_*\|\le C\hat\gamma^k\|x_0-x_*\|,
\qquad
\hat\gamma\in(\gamma,1)
\]
[2410.09420].

Later variants extend the theory rather than replace it. Two-level sketching AAP establishes a backward stability criterion under Lipschitz continuity of \(T\): if the RHS perturbation is controlled and the projectors satisfy the stated stability bound involving \(L\), \(\sigma_{\min}(\hat R^k)\), and \(\|\Delta x^j\|\), then the LS perturbation \(\delta^k\) is uniformly bounded by \(C\) [2505.08587]. For generalized Sylvester equations, the preconditioner in P-aAA is based on a first-order Neumann approximation, and the fixed-point error bound improves from
\[
\|E_k\|\le \|\widehat{\mathcal L}^{-1}\widehat\Pi\|^k\|E_0\|
\]
to
\[
\|E_k\|\le \|\widehat{\mathcal L}^{-1}\widehat\Pi\|^{2k}\|E_0\|,
\]
so the preconditioned fixed-point part squares the contraction factor [2604.15766].

## 5. Variants and application domains

The linear and nonlinear application record is broad. In nonsymmetric linear systems from Matrix Market, GAAA accelerating Jacobi or Richardson iterations often reduced both iteration count and CPU time relative to windowed AA and GMRES. On fidap008, GMRES required \(2182\) iter \((22.9\ \text{s})\), AA(5) solved in \(56\) iter \((0.034\ \text{s})\), and \(aAA(5)[1]\)–FP[5] solved in \(31\) iter \((0.015\ \text{s})\). On fidapm37, AA(5) failed because of rank-deficient LSQ, whereas \(aAA(10)[1]\)–FP[6] solved in \(50\) iter \((0.162\ \text{s})\) and \(aAA(10)[3]\)–FP[5] in \(47\) iter \((0.180\ \text{s})\) [2508.10158].

In nonlinear PDEs, alternating variants have been used with Picard maps, multiphysics couplings, and sketching. For incompressible Navier–Stokes, AA(20) reached the target residual reduction \(10^{-10}\) in \(37\) iter and \(\sim 10{,}495\ \text{s}\), while \(aAA(20)[25]\)–FP[20] also used \(37\) iter but \(\sim 10{,}278\ \text{s}\); AA\((\infty)\) required \(39\) iter and \(\sim 11{,}281\ \text{s}\) [2508.10158]. The two-level sketching extension of AAP reported “up to 50% time-to-solution reductions compared to standard Anderson acceleration—without degrading convergence rates” on Stokes, \(p\)-Laplacian, Bidomain, and Navier–Stokes benchmarks. In the largest Stokes case, baseline AA(10,1) used \(119\) iterations and \(385.12\ \text{s}\), while the best adapted variant used \(60\) iterations and \(175.30\ \text{s}\); in the largest Bidomain case, the best alternation-plus-randomized adaptivity variant improved from \(45\) iterations and \(2207.21\ \text{s}\) to \(27\) iterations and \(1247.51\ \text{s}\) [2505.08587].

Optimization and splitting methods form another major application class. In ADMM experiments reported for total variation, Lasso, and nonnegative least squares, carefully chosen \(aAA(m)[s]\)–FP\([t]\) configurations frequently reduced both iteration count and CPU time relative to windowed AA and ADMM alone; for regularized logistic regression on covtype, GD failed at \(1000\) iterations and \(1122\ \text{s}\), AA(7) used \(64\) iterations and \(72.3\ \text{s}\), and the best GAAA configuration \(aAA(5)[1]\)–FP[2] used \(28\) iterations and \(30.6\ \text{s}\) [2508.10158]. In full-waveform inversion, the Anderson-accelerated augmented Lagrangian method reduced the BP salt case from \(580\) iterations for plain IR-WRI to \(371\) iterations for AA-IR-WRI, while also improving early kinematic recovery [2106.14065]. For nonconvex ADMM based on Douglas–Rachford splitting, accelerating the lower-dimensional DRS fixed-point in the constraint space was shown to be more effective than accelerating concatenated ADMM variables, with convergence guarantees under nonconvex DRS assumptions [2006.14539]. Stationary AA applied to ADMM was also analyzed spectrally: for a ridge-regression example with \(\rho(q'(z^*))=0.833\), the optimal sAA(1) factor was \(0.592\), improving over the baseline \(0.833\), and sAA(2) and sAA(3) further reduced the asymptotic factor to approximately \(0.516\) and \(0.4837\) [2007.02916].

The framework extends beyond standard vector fixed-point problems. For multilinear systems with nonsingular \(\mathcal M\)-tensors, tensor AAR periodically interleaves tensor Richardson steps with tensor Anderson–Richardson steps on the power vector \(x^{[m-1]}\); across seven tested sizes, TAAR converged in \(13\)–\(19\) iterations, whereas baseline tensor splitting methods required on the order of \(500\)–\(1700\) iterations [2401.07602]. For generalized Sylvester equations, P-aAA consistently outperformed fixed-point, Neumann, AA/aAA, and Krylov baselines. In one generalized Lyapunov example with \(n=900\) and \(\rho(\mathcal L^{-1}\Pi)=0.93\), P-aAA achieved \(2.21\ \text{s}\) versus \(3.40\ \text{s}\) for AA, \(2.36\ \text{s}\) for aAA, \(14.5\ \text{s}\) for FP, and \(66.7\ \text{s}\) for GLEK; in a case with \(\rho=1.22>1\), AA, GLEK, Neumann, and FP failed, while aAA converged in \(20.1\ \text{s}\) and P-aAA in \(7.34\ \text{s}\) [2604.15766].

## 6. Parameterization, implementation, and limitations

The practical design variables are the history length \(m\), the numbers \(s\) and \(t\) of accelerated and baseline steps per cycle, the damping, and the restart or filtering policy. In GAAA, larger \(m\) usually accelerates more because the Chebyshev factor \(C(a,b,m)\) decays rapidly with \(m\), but it also increases cost, storage, and the risk of LS ill-conditioning; the paper notes that \(m\in[3,20]\) is common in practice, and for linear problems \(s=1\) or \(2\) with \(t\approx m/2\) often work well, whereas for nonlinear, ADMM, and gradient-descent problems \(s\in[3,10]\) with small \(t\in\{1,2,3\}\) typically gave the best CPU/iteration trade-offs [2508.10158]. In AAP, moderate \(m\) such as \(3\)–\(7\) is described as often effective, because larger \(m\) enriches multisecant information but can degrade conditioning of \(S_t\) and \(Y_t\) and magnify \(\|E_t\|\) through \(\operatorname{cond}(S_t)\) [2407.10472].

Damping and stabilization are recurrent themes. AAP treats \(\beta_t\) as a step size in the residual direction; smaller \(\beta_t\) improves stability when residuals are large, while \(\beta_t\) near \(1\) is suitable in the small-residual regime [2407.10472]. Composite AAoptD chooses \(\beta_k\) by a per-iteration residual model and then applies safeguards such as \(\hat\beta_k=\max\{\beta_k,\eta\}\) or the alternative flip rule when \(\beta_k<\eta\), with \(0<\eta<0.5\) [2203.14627]. In nonsmooth problems, regularized constrained least-squares, residual-descent checks, damping \(x_{k+1}\leftarrow x_k+\eta(\cdot-x_k)\), and restarts are standard safeguards, because bounded AA coefficients are assumed in the local theory but not always provable a priori [2410.09420].

Implementation is dominated by the small least-squares solve and by the cost of evaluating the baseline map. Robust solvers use QR or SVD for the LSQ, often with Tikhonov regularization or a Moore–Penrose pseudoinverse when the history becomes nearly rank deficient [2508.10158, 1606.08740]. AAR reduces global communication by performing one global reduction every \(p\) iterations rather than every iteration, which explains its strong and weak scaling advantages on large processor counts [1606.08740]. Two-level sketching AAP pushes this further by reducing the LS problem dimension through a static physics-based projection \(\Pi_1\) and an adaptive algebraic sketch \(\Pi_2\); the paper’s cache-aware guidance is that masked matrix–vector products are faster than full sequential access only when at most \(10\%\) of rows are retained, while QR factorization remains faster than full only when at most \(80\%\) of rows are retained [2505.08587].

The limitations are likewise consistent across the literature. Global convergence is not guaranteed in general for AA, and several theories are explicitly local. The active-manifold results hinge on identification and nondegeneracy; if the structure does not settle, the local smooth theory may not apply [2410.09420]. Large windows can trigger ill-conditioning or rank-deficient least-squares systems [2508.10158]. Excessive sketching or poor variable selection can be harmful: in Bidomain, intracellular masking was detrimental, and \(p=4\) caused divergence [2505.08587]. A plausible implication is that generalized alternating AA is most reliable when alternation is used not as a purely heuristic slowdown of AA, but as a mechanism for improving local relevance of the secant information, controlling conditioning, and matching the structure of the underlying fixed-point map.

Source: https://www.emergentmind.com/topics/generalized-alternating-anderson-acceleration