Damped Landweber with Nesterov Acceleration

• The paper demonstrates that combining damping with Nesterov momentum significantly accelerates convergence in solving linear and nonlinear inverse problems.
• It employs a recursive scheme with spectral polynomial filters and Gegenbauer representations to achieve near-optimal regularization properties and control variance.
• Numerical experiments confirm substantial speed-ups in applications like computed tomography and PDE parameter identification, emphasizing the balance between rapid bias decay and noise amplification.

The damped Landweber method with Nesterov acceleration is a class of iterative regularization algorithms for solving linear and nonlinear ill-posed inverse problems, especially in Hilbert and Banach space settings. By combining classical Landweber iteration (a gradient-descent type method) with both damping (via step-size control or explicit inertia) and momentum/extrapolation strategies inspired by Nesterov, these methods achieve accelerated convergence rates and optimal or near-optimal regularization properties. The approach is characterized by iterate recursions involving weighted averaging, polynomial filters connected to ultraspherical polynomials, and—where applicable—extensions to convex penalty functionals and inexact solvers.

1. Mathematical Formulation and Core Algorithm

Given a linear inverse problem $A x = y$ with $A: X \to Y$ (between Hilbert spaces or, more generally, Banach spaces) and possibly noisy data $\widehat{y}$, the classic Landweber iteration is:

$$x_{k+1} = x_{k} + \tau A^* \left( \widehat{y} - A x_{k} \right),$$

where $\tau > 0$ is the (possibly damped) step size.

The damped Landweber method with Nesterov acceleration (in the Hilbert space, linear setting) enhances this as follows (Kindermann, 2021, Pagliana et al., 2019, Zhu, 1 Nov 2025):

• Select $\tau > 0$ such that $\tau \|A\|^2 \leq 1$ (“damping”);
• Use a momentum parameter sequence $\beta_k$, e.g., $\beta_k = \frac{k-1}{k+\beta}$ with $\beta > -1$.

The recursion is: $$\begin{aligned} & x_0 = 0, \ & x_1 = \tau A^* \widehat{y}, \ & \text{For } k\geq1: \ & \quad y_k = x_k + \beta_k (x_k - x_{k-1}), \ & \quad x_{k+1} = y_k + \tau A^* (\widehat{y} - A y_k). \end{aligned}$$ This is referred to as Nesterov-enabled Landweber or "damped Nesterov–Landweber".

Extensions to Banach spaces allow for nonlinear forward operators $F_i$, convex penalties $\Theta$, and inexact inner solvers using Bregman distances and $\varepsilon$-subdifferential calculus; these generalizations retain a similar momentum strategy for both primal and dual variables (Jin, 2016).

2. Spectral, Polynomial, and Filter Perspectives

The accelerated Landweber iteration admits an explicit polynomial representation of the residual error, which is crucial for analyzing convergence and regularization properties. In the linear Hilbert space case, the residual operator after $k$ iterations is:

$$R_k = I - A^*A\,g_k(A^*A), \quad x_k = g_k(A^*A)A^* \widehat{y},$$

where $g_k$ is a filter polynomial defined recursively: $$r_{k+1}(\lambda) = (1 - \tau \lambda)\,[ r_k(\lambda) + \beta_k(r_k(\lambda) - r_{k-1}(\lambda)) ], \quad r_0(\lambda)=1,\, r_1(\lambda)=1 - \tau \lambda.$$ For Nesterov-type $\beta_k$, $r_k(\lambda)$ can be expressed in terms of Gegenbauer polynomials $C_n^{(\alpha)}$:

$$r_k(\lambda) = (1-\tau \lambda)^{(k+1)/2}\frac{C_{k-1}^{((\beta+1)/2)}(\sqrt{1-\tau \lambda})}{C_{k-1}^{((\beta+1)/2)}(1)}$$

(Kindermann, 2021). This representation underpins the derivation of optimal (and semi-saturated/suboptimal) convergence rates under classical source conditions.

Within the spectral filtering viewpoint, the convergence rate and bias–variance tradeoff are governed by qualification (maximal $q$ such that $|\lambda^q r_k(\lambda)|$ decays at the correct rate across the spectrum of $A^*A$) and the polynomial degree (Pagliana et al., 2019).

3. Convergence Theory and Optimality

Assuming the source condition $x^\ast \in \operatorname{Range}((A^*A)^\mu)$, $\mu > 0$, the main convergence theorems for damped Landweber–Nesterov indicate (Kindermann, 2021):

• A priori stopping: For $\mu \leq (\beta+1)/4$, termination at $k(\delta) \asymp \delta^{-1/(2\mu+1)}$ achieves

$$\|x_{k(\delta)} - x^\ast\| = O(\delta^{2\mu/(2\mu+1)}),$$

which is the optimal order in the absence of saturation. For $\mu > (\beta+1)/4$, convergence slows (semi-saturation).

• Discrepancy principle: With stopping when $\|A x_k - \widehat{y}\| \leq \tau_0 \delta$, the same order is optimal for $\mu + 1/2 \leq (\beta + 1)/4$, but is suboptimal beyond this regime.

For the basic Nesterov-accelerated Landweber (APDFP specialization), functional error obeys $O(1/k^2)$ convergence. In learning theory, bias decays at $O(k^{-4r})$ (for source index $r$), but variance grows as $O(k^2/n)$, reducing stability; thus, early stopping (via a discrepancy or balancing rule) is essential to maintain regularization (Pagliana et al., 2019, Zhu, 1 Nov 2025).

4. Extensions: Nonlinear Operators, Banach Spaces, and Inexact Solvers

The damped Landweber–Kaczmarz method generalizes the framework to Banach spaces, nonlinear forward operators $F_i$, systems of equations, and general convex penalty functionals $\Theta$ (Jin, 2016). The method utilizes:

• Extrapolated dual and primal updates, with $\alpha_k = k/(k+\alpha)$, $\alpha \geq 3$;
• Inexact inner solves for the convex functional minimization induced by $\Theta$ via an $\varepsilon$-subdifferential calculus, ensuring robust convergence even when inner subproblems are not solved to high precision.

Convergence is ensured under uniform convexity, tangential cone conditions on $F_i$, and summability of error/damping sequences. Strong convergence or regularization are established for the sequence of iterates and Bregman distances.

5. Practical Parameter Choices and Implementation

Critical parameter guidelines include (Kindermann, 2021, Jin, 2016, Zhu, 1 Nov 2025):

• Step size ($\tau$ or $\lambda$): Chosen to satisfy $\tau \|A\|^2 \leq 1$; empirical selection is possible, e.g., via the power method for spectral norm estimation. Conservative underestimation improves numerical stability.
• Momentum parameter ($\beta$, $\alpha$): For a priori stopping, select $\beta > 4\mu - 1$; for discrepancy, $\beta > 4\mu + 1$. For Banach space versions, $\alpha \sim 3$–$5$ is standard for momentum schedule $\alpha_k = k/(k + \alpha)$.
• Damping: Can be achieved via decaying step size, e.g., $\eta_k = \eta/(k+1)^\alpha$, $\alpha \in [0,1]$ (Pagliana et al., 2019).
• Inner solver tolerances ($\varepsilon_k$): Should decrease rapidly and sum to a finite value for rigorous convergence guarantees in the inexact Banach-space method.

Typical implementations use these recursions for each iteration, optionally incorporating restarts (resetting momentum when progress stalls) and early stopping criteria based on the discrepancy principle.

6. Numerical Performance and Stability

Numerical experiments consistently demonstrate substantial acceleration relative to pure Landweber:

• In computed tomography (CT), Nesterov acceleration reduces the number of iterations by a factor of 7–10, and total CPU time by up to 14$\times$ (Jin, 2016, Zhu, 1 Nov 2025).
• For PDE parameter identification, similar speed-ups are observed (Jin, 2016).
• Acceleration amplifies the variance component, and thus can increase solution instability, especially in the presence of data noise. Damping, together with early stopping, is required to balance rapid bias decay with controlled variance growth, preserving statistical optimality (Pagliana et al., 2019).

Empirically, using acceleration allows the same final accuracy to be reached in a fraction of the iterations compared to unaccelerated Landweber, but care must be taken when approaching convergence, where momentum and noise amplification may cause oscillatory or divergent behavior.

7. Connections to Continuous Dynamics, Alternatives, and Theoretical Insights

Recent work has interpreted the damped Landweber–Nesterov scheme as a discrete approximation to inertial continuous-time dynamics with both viscous and Hessian-driven damping (Attouch et al., 2021): $$\ddot{x}(t) + \frac{\alpha}{t} \dot{x}(t) + \beta \nabla^2 f(x(t)) \dot{x}(t) + b\nabla f(x(t)) = 0,$$ with specializations recovering Nesterov's method for $\gamma(t) = \alpha/t$.

The addition of Hessian-driven damping attenuates iterates' oscillations, and discrete analogues yield schemes with explicit control over inertia and damping terms. The Lyapunov-functional approach in the convergence proofs connects fast functional decrease ($O(1/k^2)$) with control over the velocity and gradient magnitude, yielding both value and iterate convergence.

Compared to heavy-ball or classical Landweber, the Nesterov-damped variant achieves faster decay of the bias, but at the price of increased variance and heightened sensitivity to noise, a finding confirmed in learning-theoretic bias-variance analyses (Pagliana et al., 2019).

---

Principal References:

• "Optimal-order convergence of Nesterov acceleration for linear ill-posed problems" (Kindermann, 2021)
• "Landweber-Kaczmarz method in Banach spaces with inexact inner solvers" (Jin, 2016)
• "Accelerated primal dual fixed point algorithm" (Zhu, 1 Nov 2025)
• "Convergence of iterates for first-order optimization algorithms with inertia and Hessian driven damping" (Attouch et al., 2021)
• "Implicit Regularization of Accelerated Methods in Hilbert Spaces" (Pagliana et al., 2019)