Extrapolation-Accelerated Framework

• The paper's main contribution is showing that combining past iterates via polynomial or nonlinear extrapolation cancels dominant errors to accelerate convergence.
• It demonstrates the application of extrapolation techniques across optimization, simulation, and statistical estimation, achieving superlinear or optimal rates.
• Practical implementations emphasize adaptive window sizes, regularization, and restart mechanisms to balance acceleration with numerical stability.

An extrapolation-accelerated framework encompasses algorithmic schemes in which extrapolation is used to transform sequences of iterates from an underlying algorithm—optimization, fixed-point iteration, simulation, or statistical estimation—to achieve provably or empirically faster convergence toward the desired solution. The core mechanism is to combine multiple recent iterates (often by polynomial, rational, or nonlinear combinations) or to use predicted future directions, such that dominant error modes in the sequence are canceled or suppressed. This yields superlinear or optimal convergence rates in a variety of deterministic, stochastic, convex, non-convex, discrete, or continuous settings.

1. Key Principles and General Algorithmic Structure

The extrapolation-accelerated paradigm centers on producing new estimates from windows of past iterates, with the aim of eliminating leading error components. In the scalar setting, classical methods such as Richardson extrapolation (for sequences with algebraic error expansion) and Aitken's Δ² process (for geometric/exponential error) achieve this by explicit algebraic manipulation of the sequence. In the vector case relevant for iterative solvers and optimization, polynomial and ε-algorithm generalizations, such as Minimal Polynomial Extrapolation (MPE) and Reduced Rank Extrapolation (RRE), as well as Anderson Acceleration and cyclic methods, are central.

A unifying high-level algorithmic template is as follows (Jbilou, 1 Feb 2026):

for cycle ℓ = 0, 1, 2, ... Generate a window of iterates {xₙ} Compute extrapolated output t = Extrapolate({xₙ}, params) If stopping criterion met, break Possibly restart or update state with t end

The specific choice of extrapolation operator, parameters (window size, regularization), and restart mechanism distinguishes among frameworks and determines practical effectiveness.

2. Accelerated Gradient Schemes with Extrapolation

Extrapolation-accelerated first-order algorithms address both convex and non-convex minimization. Gradient Descent with Extrapolation (GDE), for instance, employs a two-step update at each iteration:

• Compute $x_t = z_{t-1} - \eta g_{t-1}$
• Evaluate gradient $g_t$
• Update $z_t = z_{t-1} - \eta g_t$

Under $L$-smoothness and finite initial gap assumptions, GDE achieves:

$$\min_{1 \leq t \leq T} \|\nabla f(x_t)\|^2 \leq \frac{8 [f(x_0) - f(x^*)]}{\eta T} - \frac{1}{\eta^2 T} \sum_{t=0}^{T-1} \|x_{t+1} - x_t\|^2$$

Matching the $O(1/\epsilon^2)$ iteration complexity of standard GD but with an extra negative telescoping term that quantifies practical acceleration (Xu et al., 2019).

Stochastic extensions (SGDE) similarly achieve the best known stochastic stationary-point complexity, with acceleration evident in the improved theoretical bound structure (additional negative $\sum \|x_{t+1}-x_t\|^2$ term).

Relaxed or weakly accelerated methods generalize this framework, permitting extrapolation parameter sequences more flexible than classical Nesterov recurrence and yielding unified convergence rates for both convex and strongly convex settings (Li et al., 9 Apr 2025).

3. Vector Extrapolation for Iterative Solvers and Nonlinear Problems

A central domain for extrapolation acceleration is linear and nonlinear fixed-point problems, including least-squares, matrix factorization, and large-scale PDEs. Prominent polynomial-based techniques include MPE and RRE; the latter, for example, combines $k+1$ consecutive iterates $s_n$ via optimal coefficients $c$ to minimize

$$\| \sum_{j=0}^k c_j (s_{n+j+1} - s_{n+j}) \| \quad \text{subject to} \quad \sum c_j = 1$$

Substantial acceleration is obtained due to annihilation of the first $k$ dominant error modes. Vector $\varepsilon$-algorithms generalize Shanks/Aitken processes to the vector case, providing comparable acceleration properties (Jbilou, 1 Feb 2026, Mouhssine, 31 Jan 2026).

In nonlinear least-squares, hybrid PGD or SGD methods with RRE/MPE or VEA extrapolation yield convergence and solution accuracy improvements; typical window sizes $q=5$–$10$ provide optimal balance between acceleration and numerical stability (Mouhssine, 31 Jan 2026).

4. Extrapolation-Accelerated Block and Coordinate Methods

Block-coordinate and alternating direction methods have also been significantly advanced through extrapolation. For example, extrapolation-accelerated ADMM variants embed Nesterov or Tseng-style steps directly within the block updates, achieving non-ergodic $O(1/k^2)$ rates under semi-strong convexity plus smoothness (He et al., 2023).

Similarly, accelerated block proximal frameworks with adaptive momentum recognize the practical risk of divergence with extrapolation in nonconvex, nonsmooth settings. These frameworks integrate an explicit compare-reject scheme and momentum schedule, retaining monotonic objective decrease and guaranteeing convergence to critical points under the Kurdyka–Łojasiewicz property (Yang et al., 2023).

Cyclic and adaptive order extrapolation (e.g., ACX$^{3,2}$) achieve robust, low-memory acceleration for black-box solvers, often outperforming quasi-Newton or Krylov subspace methods in high-dimensional and discontinuous regimes (Lepage-Saucier, 2021).

5. Statistical Inference and Simulation Through Extrapolation

In statistics, extrapolation-accelerated frameworks address classical and modern estimation challenges. Accelerated SIMEX methods in measurement error models analytically replace the computationally expensive Monte Carlo simulation-averaging with closed-form or low-dimension numerical integration of estimating functions, maintaining bias correction and asymptotic normality but at order-of-magnitude lower computational cost (Ayub et al., 2021).

In NAS accuracy prediction, extrapolation-accelerated regression leverages linear and nonlinear transforms to support out-of-distribution prediction—an essential requirement not met by tree-based regression algorithms—resulting in predictive frameworks that both reduce validation cost (by $3\times$) and monotonicity violations (by $2\times$) compared to previous baselines (Hakim, 2022).

6. Applications in Stochastic Algorithms and Scientific Computing

Extrapolation-accelerated frameworks play a powerful role in stochastic optimization, simulation, and scientific computation. Nonlinear acceleration applied to stochastic first-order iterations, such as SGD, SAGA, or SVRG, optimally combines $k$ iterates with regularized coefficients to minimize the norm of the residuals:

$\min_{c: \sum c_i=1} \|R c\|^2 + \lambda \|c\|^2$

where $R$ is the residual matrix of $k$ recent iterates. This yields best-known accelerated rates away from the noise floor and smooth interpolation to the optimal variance level as noise dominates; practical speedups of $1.5\times$–$3\times$ are typical compared to baseline stochastic solvers (Scieur et al., 2017).

In large-scale matrix and tensor computing, such as multilinear PageRank, eigenproblems, and PDE discretizations, vector extrapolation is a crucial component for accelerating Newton–GMRES-type solvers, Picard and power methods, and finite-element schemes via Richardson/multilevel extrapolation (Boubekraoui et al., 27 Sep 2025, Gyöngy et al., 2018, Jbilou, 1 Feb 2026).

7. Theoretical, Practical, and Implementation Considerations

Convergence acceleration via extrapolation is rigorously analyzed through spectral and Lyapunov arguments, Chebyshev or minimal polynomial approximations, or energy-based contractivity proofs. Superlinear local rates are achieved in problems aligned with the error model, and $O(1/k^2)$ rates are established for convex problems by embedding extrapolation in appropriate frameworks (e.g., predictor–corrector flows, dual averaging).

Practical deployment requires careful window-size, regularization, and restart/tuning mechanisms to balance acceleration and numerical robustness—especially in high dimensions or in the presence of stochasticity or strong nonlinearity. For vector schemes, ill-conditioning of extrapolation matrices or denominator breakdowns in recursions must be monitored and guarded. Extrapolation-based acceleration generally adds small extra storage (proportional to window size and problem dimension) and negligible computational overhead compared to baseline iteration costs for medium to large-scale problems (Jbilou, 1 Feb 2026).

Extrapolation-accelerated frameworks are now standard tools for advanced optimization, simulation, and data analysis tasks requiring rapid convergence and scalable performance across broad problem classes.