Generalized Alternating Anderson Acceleration

• Generalized Alternating Anderson Acceleration is a fixed-point acceleration scheme that alternates standard iterations with Anderson-accelerated updates in a periodic or composite pattern.
• It employs least-squares mixing, multisecant quasi-Newton updates, and Krylov interpretations to improve convergence rates in various linear and nonlinear problem settings.
• The method is applied in domains such as PDEs, ADMM optimization, and tensor computations, often reducing CPU time and iteration counts compared to traditional approaches.

Generalized Alternating Anderson Acceleration denotes a class of fixed-point acceleration schemes that interleave plain fixed-point iterations with Anderson-accelerated updates in a periodic or composite pattern. In its most explicit formulation, it is “a periodic scheme composed of $t$ fixed-point iteration steps, interleaved with $s$ steps of Anderson acceleration with window size $m$,” and it generalizes both windowed Anderson acceleration and alternating Anderson acceleration (He et al., 13 Aug 2025). Closely related formulations include Alternating Anderson–Picard for nonlinear fixed-point problems, alternating Anderson–Richardson for linear systems, composite dynamic-window AA, and problem-specific extensions using preconditioning, sketching, or active-manifold structure (Feng et al., 2024).

1. Conceptual scope and historical lineages

The central object is a fixed-point map $q$ or $G$, with iteration $x_{k+1}=q(x_k)$ and residual $r(x)=q(x)-x$. Standard Anderson acceleration applies a least-squares mixing step at every iteration. Generalized alternating variants relax that schedule: they insert AA only periodically, or compose AA modules with different window sizes, damping rules, or baseline iterations. This periodicity is the defining structural distinction from stationary or windowed AA (He et al., 13 Aug 2025).

Several lineages converge on this viewpoint. Alternating Anderson–Richardson (AAR) alternates $p-1$ preconditioned Richardson steps with one Anderson step, using history matrices of iterate and residual differences; with complete history, it is equivalent to GMRES without restart in exact arithmetic, while retaining a low-communication profile on parallel architectures (Suryanarayana et al., 2016). Alternating Anderson–Picard (AAP) performs $m$ Picard iterations followed by one AA step and is explicitly identified as “an instance of generalized alternating AA,” with memory length $m$, number of baseline steps $s$0, damping $s$1, and optional restarts or filtering (Feng et al., 2024). A further expansion appears in composite non-stationary AA, where one alternates AA modules with different window sizes and optimized damping by additive or multiplicative composition; that paper states that these schemes “can be viewed as a generalized alternating Anderson acceleration” (Chen et al., 2022).

The term also appears in nonsmooth optimization, where a “Generalized Alternating Anderson Acceleration” template is used for composite maps or stacked block updates once an active manifold has been identified, so that the locally relevant iteration becomes $s$2 with Lipschitz Jacobian (Li et al., 2024). More recent extensions include two-level sketching AAP for multiphysics PDEs (Barnafi et al., 13 May 2025) and preconditioned-alternating Anderson acceleration for generalized Sylvester equations (Chen et al., 17 Apr 2026). This body of work suggests that the unifying idea is not a single algorithmic formula but a design principle: alternate inexpensive baseline iterations with selective multisecant corrections.

2. Fixed-point formulation and periodic schedules

In the general fixed-point setting, one solves $s$3 with residual $s$4. The generalized periodic schedule uses three primary parameters: the AA window size $s$5, the number $s$6 of plain fixed-point steps per cycle, and the number $s$7 of AA steps per cycle. One cycle has length $s$8, and the iteration rule is periodic: if $s$9, perform a plain fixed-point step; otherwise perform an AA step (He et al., 13 Aug 2025).

For the type-I Walker–Ni formulation used in that paper, AA$m$0 computes coefficients $m$1 from a least-squares problem on residual differences,

$m$2

with update

$m$3

The MATLAB-like pseudocode maintains rolling matrices of residuals and map evaluations to assemble the least-squares matrices $m$4 and $m$5 efficiently, and then either accepts the plain iterate $m$6 or the accelerated iterate $m$7 depending on the phase of the cycle (He et al., 13 Aug 2025).

AAP specializes this structure to nonlinear fixed-point problems $m$8 with $m$9. At global iteration $q$0, it performs $q$1 Picard steps

$q$2

then solves a constrained least-squares problem over coefficients $q$3 satisfying $q$4, and updates

$q$5

The corresponding difference matrices $q$6 and $q$7 reduce the constrained problem to an unconstrained least-squares in $q$8, and the resulting AA step can be written as a quasi-Newton update $q$9 (Feng et al., 2024).

The main special cases are straightforward.

Variant	Schedule	Relation
AA$G$0	AA at every iteration	recovered by $G$1
restarted AA(1)	one FP step, one AA step, window $G$2	$G$3
alternating Anderson with one AA per cycle	$G$4, general $G$5	fits GAAA but differs step-by-step from prior AAR schedules
AAP$G$6	$G$7 Picard steps, then one AA step	generalized alternating AA with $G$8
AAR	$G$9 Richardson steps, then one Anderson step	generalized AAJ with preconditioning

Composite non-stationary schemes widen the schedule design space further. Additive composition computes one AA$x_{k+1}=q(x_k)$0 proposal and one AA$x_{k+1}=q(x_k)$1 proposal from the same iterate and averages them, while multiplicative composition uses AA$x_{k+1}=q(x_k)$2 in an outer loop and AA$x_{k+1}=q(x_k)$3 in a short inner loop; fully non-stationary versions replace one module by AAoptD, in which the damping $x_{k+1}=q(x_k)$4 is selected by a residual-minimization model (Chen et al., 2022).

3. Multisecant, Krylov, and Newton interpretations

A decisive feature of generalized alternating AA is that its apparently ad hoc periodic mixing admits precise multisecant and Krylov interpretations. In AA-II form, with

$x_{k+1}=q(x_k)$5

the update

$x_{k+1}=q(x_k)$6

has a multisecant quasi-Newton representation

$x_{k+1}=q(x_k)$7

(Feng et al., 2024).

For AAP, the multisecant structure becomes more explicit. With $x_{k+1}=q(x_k)$8 and $x_{k+1}=q(x_k)$9, the approximate inverse Jacobian

$r(x)=q(x)-x$0

satisfies the inverse multisecant condition $r(x)=q(x)-x$1. Equivalently, one may define a direct multisecant matrix $r(x)=q(x)-x$2 by $r(x)=q(x)-x$3, so that the AA direction is precisely a quasi-Newton step on a multisecant linearization (Feng et al., 2024).

The same paper proves that AAP is equivalent to solving a multisecant linear system with GMRES at each AA iteration. For any $r(x)=q(x)-x$4 in the multisecant set $r(x)=q(x)-x$5, there exists a vector $r(x)=q(x)-x$6 solving

$r(x)=q(x)-x$7

such that

$r(x)=q(x)-x$8

with $r(x)=q(x)-x$9 a damped linearized Picard map. The constrained LS over $p-1$0, the unconstrained LS over $p-1$1, and the Krylov residual minimization are therefore equivalent descriptions of the same update (Feng et al., 2024).

In the small-residual regime, AAP approaches Newton–GMRES. Under Lipschitz differentiability of $p-1$2 and rank and conditioning assumptions on $p-1$3 and $p-1$4, the closest multisecant matrix $p-1$5 converges to the Jacobian $p-1$6, the Krylov blocks $p-1$7 and $p-1$8 converge to their Jacobian-generated limits, the search direction $p-1$9 converges to the Newton–GMRES direction $m$0, and the AAP optimization gain

$m$1

converges to the normalized Jacobian–GMRES residual (Feng et al., 2024).

Linear alternating schemes admit related Krylov interpretations. For $m$2, GAAA applied to Richardson iteration has a periodic GMRES equivalence: if $m$3 and GMRES residuals strictly decrease up to $m$4, then $m$5 for $m$6. When $m$7, this reduces to the classical AA$m$8–GMRES equivalence. AAR with complete history is likewise equivalent to GMRES without restart in exact arithmetic (He et al., 13 Aug 2025, Suryanarayana et al., 2016).

4. Convergence theory

The convergence theory of generalized alternating AA is stratified by problem class. In the linear contractive case, the theory is simplest: if the Richardson matrix $m$9 satisfies $m$0, then for any initial $m$1, GAAA $m$2–FP$m$3 converges, and every step—plain or accelerated—satisfies

$m$4

(He et al., 13 Aug 2025).

For diagonalizable but noncontractive Richardson iterations, the same paper derives Chebyshev-based one-step and per-cycle bounds. If the spectrum of $m$5 lies in $m$6 not containing $m$7 or $m$8, then an AA$m$9 step satisfies

$s$00

where

$s$01

For GAAA $s$02–FP$s$03 with $s$04, the residuals at cycle endpoints obey

$s$05

A sufficient convergence condition is then obtained even when the base fixed-point iteration is divergent but diagonalizable (He et al., 13 Aug 2025).

For nonlinear fixed-point problems, AAP furnishes the most detailed local theory. Under Assumption 1.1—$s$06 nonexpansive with Lipschitz $s$07—together with full-rank and conditioning assumptions on $s$08 and $s$09, the one-step residual bound is

$s$10

If $s$11 is a contraction, $s$12, and

$s$13

then AAP achieves local linear residual convergence,

$s$14

and it satisfies an inexact Newton condition with forcing term

$s$15

A sufficient local convergence condition is $s$16 (Feng et al., 2024).

For nonsmooth problems, the local theory proceeds through active-manifold identification. Once an algorithm such as PPA, PGA, PLA, cyclic PCD, DRS/ADMM, or IRL1 identifies an active manifold $s$17, the fixed-point map becomes smooth locally by sensitivity analysis. Under local contraction and bounded AA coefficients, AA then achieves R-linear local convergence: $s$18 (Li et al., 2024).

Later variants extend the theory rather than replace it. Two-level sketching AAP establishes a backward stability criterion under Lipschitz continuity of $s$19: if the RHS perturbation is controlled and the projectors satisfy the stated stability bound involving $s$20, $s$21, and $s$22, then the LS perturbation $s$23 is uniformly bounded by $s$24 (Barnafi et al., 13 May 2025). For generalized Sylvester equations, the preconditioner in P-aAA is based on a first-order Neumann approximation, and the fixed-point error bound improves from

$s$25

to

$s$26

so the preconditioned fixed-point part squares the contraction factor (Chen et al., 17 Apr 2026).

5. Variants and application domains

The linear and nonlinear application record is broad. In nonsymmetric linear systems from Matrix Market, GAAA accelerating Jacobi or Richardson iterations often reduced both iteration count and CPU time relative to windowed AA and GMRES. On fidap008, GMRES required $s$27 iter $s$28, AA(5) solved in $s$29 iter $s$30, and $s$31–FP[5] solved in $s$32 iter $s$33. On fidapm37, AA(5) failed because of rank-deficient LSQ, whereas $s$34–FP[6] solved in $s$35 iter $s$36 and $s$37–FP[5] in $s$38 iter $s$39 (He et al., 13 Aug 2025).

In nonlinear PDEs, alternating variants have been used with Picard maps, multiphysics couplings, and sketching. For incompressible Navier–Stokes, AA(20) reached the target residual reduction $s$40 in $s$41 iter and $s$42, while $s$43–FP[20] also used $s$44 iter but $s$45; AA$s$46 required $s$47 iter and $s$48 (He et al., 13 Aug 2025). The two-level sketching extension of AAP reported “up to 50% time-to-solution reductions compared to standard Anderson acceleration—without degrading convergence rates” on Stokes, $s$49-Laplacian, Bidomain, and Navier–Stokes benchmarks. In the largest Stokes case, baseline AA(10,1) used $s$50 iterations and $s$51, while the best adapted variant used $s$52 iterations and $s$53; in the largest Bidomain case, the best alternation-plus-randomized adaptivity variant improved from $s$54 iterations and $s$55 to $s$56 iterations and $s$57 (Barnafi et al., 13 May 2025).

Optimization and splitting methods form another major application class. In ADMM experiments reported for total variation, Lasso, and nonnegative least squares, carefully chosen $s$58–FP$s$59 configurations frequently reduced both iteration count and CPU time relative to windowed AA and ADMM alone; for regularized logistic regression on covtype, GD failed at $s$60 iterations and $s$61, AA(7) used $s$62 iterations and $s$63, and the best GAAA configuration $s$64–FP[2] used $s$65 iterations and $s$66 (He et al., 13 Aug 2025). In full-waveform inversion, the Anderson-accelerated augmented Lagrangian method reduced the BP salt case from $s$67 iterations for plain IR-WRI to $s$68 iterations for AA-IR-WRI, while also improving early kinematic recovery (Aghazade et al., 2021). For nonconvex ADMM based on Douglas–Rachford splitting, accelerating the lower-dimensional DRS fixed-point in the constraint space was shown to be more effective than accelerating concatenated ADMM variables, with convergence guarantees under nonconvex DRS assumptions (Ouyang et al., 2020). Stationary AA applied to ADMM was also analyzed spectrally: for a ridge-regression example with $s$69, the optimal sAA(1) factor was $s$70, improving over the baseline $s$71, and sAA(2) and sAA(3) further reduced the asymptotic factor to approximately $s$72 and $s$73 (Wang et al., 2020).

The framework extends beyond standard vector fixed-point problems. For multilinear systems with nonsingular $s$74-tensors, tensor AAR periodically interleaves tensor Richardson steps with tensor Anderson–Richardson steps on the power vector $s$75; across seven tested sizes, TAAR converged in $s$76–$s$77 iterations, whereas baseline tensor splitting methods required on the order of $s$78–$s$79 iterations (Niu et al., 2024). For generalized Sylvester equations, P-aAA consistently outperformed fixed-point, Neumann, AA/aAA, and Krylov baselines. In one generalized Lyapunov example with $s$80 and $s$81, P-aAA achieved $s$82 versus $s$83 for AA, $s$84 for aAA, $s$85 for FP, and $s$86 for GLEK; in a case with $s$87, AA, GLEK, Neumann, and FP failed, while aAA converged in $s$88 and P-aAA in $s$89 (Chen et al., 17 Apr 2026).

6. Parameterization, implementation, and limitations

The practical design variables are the history length $s$90, the numbers $s$91 and $s$92 of accelerated and baseline steps per cycle, the damping, and the restart or filtering policy. In GAAA, larger $s$93 usually accelerates more because the Chebyshev factor $s$94 decays rapidly with $s$95, but it also increases cost, storage, and the risk of LS ill-conditioning; the paper notes that $s$96 is common in practice, and for linear problems $s$97 or $s$98 with $s$99 often work well, whereas for nonlinear, ADMM, and gradient-descent problems $m$00 with small $m$01 typically gave the best CPU/iteration trade-offs (He et al., 13 Aug 2025). In AAP, moderate $m$02 such as $m$03–$m$04 is described as often effective, because larger $m$05 enriches multisecant information but can degrade conditioning of $m$06 and $m$07 and magnify $m$08 through $m$09 (Feng et al., 2024).

Damping and stabilization are recurrent themes. AAP treats $m$10 as a step size in the residual direction; smaller $m$11 improves stability when residuals are large, while $m$12 near $m$13 is suitable in the small-residual regime (Feng et al., 2024). Composite AAoptD chooses $m$14 by a per-iteration residual model and then applies safeguards such as $m$15 or the alternative flip rule when $m$16, with $m$17 (Chen et al., 2022). In nonsmooth problems, regularized constrained least-squares, residual-descent checks, damping $m$18, and restarts are standard safeguards, because bounded AA coefficients are assumed in the local theory but not always provable a priori (Li et al., 2024).

Implementation is dominated by the small least-squares solve and by the cost of evaluating the baseline map. Robust solvers use QR or SVD for the LSQ, often with Tikhonov regularization or a Moore–Penrose pseudoinverse when the history becomes nearly rank deficient (He et al., 13 Aug 2025, Suryanarayana et al., 2016). AAR reduces global communication by performing one global reduction every $m$19 iterations rather than every iteration, which explains its strong and weak scaling advantages on large processor counts (Suryanarayana et al., 2016). Two-level sketching AAP pushes this further by reducing the LS problem dimension through a static physics-based projection $m$20 and an adaptive algebraic sketch $m$21; the paper’s cache-aware guidance is that masked matrix–vector products are faster than full sequential access only when at most $m$22 of rows are retained, while QR factorization remains faster than full only when at most $m$23 of rows are retained (Barnafi et al., 13 May 2025).

The limitations are likewise consistent across the literature. Global convergence is not guaranteed in general for AA, and several theories are explicitly local. The active-manifold results hinge on identification and nondegeneracy; if the structure does not settle, the local smooth theory may not apply (Li et al., 2024). Large windows can trigger ill-conditioning or rank-deficient least-squares systems (He et al., 13 Aug 2025). Excessive sketching or poor variable selection can be harmful: in Bidomain, intracellular masking was detrimental, and $m$24 caused divergence (Barnafi et al., 13 May 2025). A plausible implication is that generalized alternating AA is most reliable when alternation is used not as a purely heuristic slowdown of AA, but as a mechanism for improving local relevance of the secant information, controlling conditioning, and matching the structure of the underlying fixed-point map.