Iteratively Reweighted ℓ₁ Methods

• Iteratively reweighted ℓ₁ methods are iterative algorithms that transform nonconvex sparse regularization into a sequence of convex weighted ℓ₁ minimizations.
• They utilize strategies such as weight updating, extrapolation, and Anderson acceleration to speed up convergence in applications like compressed sensing and imaging.
• Theoretical guarantees via the Kurdyka–Łojasiewicz property ensure global convergence and predictable rates, while practical implementations benefit from efficient inner solvers and GPU acceleration.

Iteratively reweighted ℓ₁ methods are a class of algorithms that solve sparse regularization problems by replacing the challenging, often nonconvex or nonsmooth, regularization term with a sequence of convex weighted ℓ₁ minimization problems. These methods have achieved widespread use in signal processing, statistics, optimization, and machine learning, especially for inducing sparsity in inverse problems and compressed sensing, as well as for tackling non-Lipschitz, nonconvex objectives.

1. Problem Formulation and Motivation

The prototypical optimization model addressed by iteratively reweighted ℓ₁ (IRL₁) methods is

$$\min_{x\in\mathbb{R}^n} \; F(x) := f(x) + \lambda \|x\|_p^{p} \qquad \text{with} \; 0 < p < 1, \;\lambda>0,$$

where $f:\mathbb{R}^n\to\mathbb{R}$ is typically a differentiable loss with Lipschitz-continuous gradient, and $\|x\|_p^p = \sum_{i=1}^n |x_i|^p$ acts as a nonconvex, sparsity-inducing regularizer (Wang et al., 2021, Wang et al., 2020, Wang et al., 2019).

For $p < 1$, the regularizer is nonconvex and non-Lipschitz at the origin—direct minimization is computationally intractable. The IRL₁ approach replaces the hard regularizer with a sequence of convex surrogates, each majorizing the original at the current iterate, such that each subproblem is a (weighted) ℓ₁ minimization.

2. Algorithmic Framework and Variants

The iteratively reweighted ℓ₁ methodology encompasses several algorithmic forms, united by alternating between solving a convex weighted ℓ₁-regularized subproblem and updating the weights:

• General IRL₁ subproblem: At iteration $k$, given weights $w^k$, solve

$$x^{k+1} = \arg\min_x \, Q_k(x) + \lambda \sum_{i=1}^n w_i^k |x_i|$$

where $Q_k(x)$ is a strongly convex local model of $f$—commonly a proximal quadratic (first-order) approximation:

$$Q_k(x) = \nabla f(x^k)^\top (x-x^k) + \frac{\beta}{2}\|x-x^k\|_2^2, \quad \beta > L_f$$

• Weight update: For $\ell_p$ penalties with smoothing $\epsilon^k\in\mathbb{R}^n_{++}$,

$w_i^k = p(|x_i^k|+\epsilon_i^k)^{p-1}$

The smoothing vector $\epsilon^k$ is typically reduced at each step: $\epsilon^{k+1}_i \leq \mu \epsilon^k_i$, $\mu\in(0,1)$ (Wang et al., 2021, Wang et al., 2020, Wang et al., 2019).

• Extrapolation/Acceleration: The EIRL₁ algorithm combines IRL₁ with an inertial step:

$$y^k = x^k + \alpha^k(x^k - x^{k-1}), \qquad 0 \leq \alpha^k < 1,$$

solving the subproblem at $y^k$ rather than $x^k$ to accelerate convergence (Wang et al., 2021).

• Special cases: For composite analysis operators, e.g., with multiple subdictionaries $\Psi_d$, generalized reweighted ℓ₁ penalties take the form

$$R(x;\{\lambda_d\}) = \sum_{d=1}^D \lambda_d \|W_d \Psi_d x\|_1,$$

and the Co-IRW-L1 algorithm performs joint reweighting both across and within subdictionaries (Ahmad et al., 2015).

3. Theoretical Guarantees: Convergence and Complexity

Modern IRL₁ analysis leverages the Kurdyka–Łojasiewicz (KL) property on the objective, which is satisfied for a broad class of functions, including semi-algebraic penalties such as $\ell_p^p$ for rational $p$ (Wang et al., 2020, Wang et al., 2021). Under these conditions:

• Global convergence: The sequence $\{x^k\}$ is bounded, $\|x^{k+1}-x^k\|\to0$, and every limit $x^*$ is a stationary point of the nonconvex problem. In fact, the iterates $\{x^k\}$ converge to a unique $x^*$ (Wang et al., 2020, Wang et al., 2021).
• Local convergence rate: The KL exponent $\theta$ determines the rate:
  • $\theta=0$: finite convergence in finite steps.
  • $0<\theta\leq1/2$: linear (geometric) rate, $\|x^k-x^*\|\leq c_1\gamma^k-c_2\|\epsilon^k\|_1$.
  • $1/2<\theta<1$: sublinear (polynomial) rate (Wang et al., 2021, Wang et al., 2020).
• Stable support and sign: After a finite number of steps, the support and sign pattern of iterates stabilize and remain fixed. The limiting behavior is locally equivalent to solving a smooth (strongly convex) problem on the active set (Wang et al., 2019).
• Support lower bound: Nonzero entries at stationarity are uniformly bounded away from zero, explicitly in terms of the regularization parameter $\lambda$, $p$, and the gradient modulus of $f$ (Lu, 2012).
• No restriction on smoothing decay: Methods such as EIRL₁, PIRL₁, and modern IRL₁ variants allow $\epsilon^k\to0$, with no requirement that smoothing be bounded away from zero (Wang et al., 2021, Wang et al., 2020).

4. Practical Accelerations and Implementation

Extensive efforts have been devoted to accelerating IRL₁ algorithms, as basic IRL₁ can be slow near solutions:

• Extrapolation: The EIRL₁ approach with momentum-like inertial term delivers faster convergence and lower mean-squared error in sparse signal recovery tasks compared to standard IRL₁ and IRL₂ (Wang et al., 2021).
• Anderson acceleration: Recent advances utilize Anderson acceleration for IRL₁, constructing linear combinations of previous iterates to speed up fixed-point convergence. Notably, Anderson-accelerated IRL₁ achieves provable local linear convergence even in nonsmooth scenarios and does not require the KL property; global convergence is ensured by a nonmonotone line search (Li, 12 Mar 2024).
• Efficient inner solves: In high dimensions, the weighted ℓ₁ or weighted least squares subproblems are solved efficiently via conjugate gradient, flexible Krylov methods, or warm restarts. For large-scale inverse problems, IRL₁ can be embedded in iterative refinement frameworks with warm or memory-limited Krylov bases (Onisk et al., 4 Feb 2025, Fornasier et al., 2015).
• GPU implementation: For specific applications (e.g., $L^1$ phase unwrapping), IRL₁ is highly parallelizable, and solving quadratic subproblems by preconditioned CG with Sylvester or Laplacian blocks leads to order-of-magnitude runtime reductions (Dubois-Taine et al., 18 Jan 2024).

5. Applications and Numerical Behavior

IRL₁ methods are central in applications demanding enhanced sparsity or model selection capability:

Application Area	Effect / Strength of IRL₁	Key Reference
Compressed sensing	Reduces error bounds, accelerates recovery, robust to noise	(0904.3780)
Cardinality minimization	Outperforms $\ell_1$ for moderate-to-large sparsity, robust support recovery	(Abdi, 2013)
Inverse problems & imaging	Flexible Krylov-IRL₁ yields accurate reconstructions with modest memory	(Onisk et al., 4 Feb 2025)
Composite sparsity	Adaptive sparsity across subdictionaries, higher SNR	(Ahmad et al., 2015)
Phase unwrapping, InSAR	GPU-IRL₁ is scalable, numerically superior to graph-based methods	(Dubois-Taine et al., 18 Jan 2024)

IRL₁ methods have also influenced advances in $\ell_1$-based regression (with provable sublinear $1/\epsilon$ iteration complexity independent of the data conditioning (Ene et al., 2019)) and motivated extensions to nonseparable penalties, multivariate analysis operators, and Bayesian model selection (Ahmad et al., 2015, Wang et al., 2019).

6. Extensions and Theoretical Significance

Iteratively reweighted ℓ₁ methods have inspired a variety of theoretical and practical extensions:

• Majorization-minimization: IRL₁ can be viewed as a specific MM scheme, where each subproblem tangentially majorizes the nonconvex penalty (e.g., log-sum, ℓ₀-approximations) (Ahmad et al., 2015).
• Biconvex frameworks: For concave penalties $g(|x|)$, IRL₁ is equivalent to alternating convex search on a biconvex functional $F(x,w)$, yielding numerical convergence of iterates even when strict functional convergence is unavailable (Fosson, 2018).
• Bayesian MAP interpretation: The limiting weighted ℓ₁ problem corresponds to MAP estimation with nonidentical Laplace priors, where IRL₁ dynamically estimates the scale parameters (Wang et al., 2019, Ahmad et al., 2015).
• Robustness to parameter choices: Empirical studies indicate IRL₁ is robust provided smoothing parameters are neither too small nor too large; the performance depends significantly on exponent $p$ (optimal for small $p\in(0.04,0.08)$), and on merits of the chosen concave approximation (Abdi, 2013).
• Algorithmic flexibility: Fixed $\epsilon$ IRL₁ variants with Lipschitz surrogate functions admit stationary-point convergence guarantees without requiring smoothing decay to zero, with coordinate-wise closed-form solutions (Lu, 2012).

7. Limitations, Challenges, and Open Problems

Open points remain in adaptive parameter tuning (e.g., decay factors $\mu$, inertia $\alpha^k$), and in rigorous complexity bounds for all variants, particularly for more general nonseparable, structured, or nonconvex penalties (Wang et al., 2021). While the KL property and semi-algebraicity cover most practical objectives, extensions to non-semi-algebraic programs, or with less regular loss components, remain challenging.

Potential limitations include sensitivity to initial smoothing, nonuniqueness of global minimizers, and, in some cases, the need to detect support stabilization for optimal performance (Wang et al., 2019). For certain rational exponents, closed-form thresholding is essential to achieve the best acceleration (e.g., for the IJT method) (Wang et al., 2021).

---

References:

• "An Extrapolated Iteratively Reweighted l1 Method with Complexity Analysis" (Wang et al., 2021)
• "Convergence Rate Analysis of Proximal Iteratively Reweighted $\ell_1$ Methods for $\ell_p$ Regularization Problems" (Wang et al., 2020)
• "Relating lp regularization and reweighted l1 regularization" (Wang et al., 2019)
• "Anderson acceleration for iteratively reweighted $\ell_1$ algorithm" (Li, 12 Mar 2024)
• "Iteratively Reweighted $\ell_1$ Approaches to Sparse Composite Regularization" (Ahmad et al., 2015)
• "Fast and Accurate Algorithms for Re-Weighted L1-Norm Minimization" (Asif et al., 2012)
• "Iterative Reweighted Minimization Methods for $l_p$ Regularized Unconstrained Nonlinear Programming" (Lu, 2012)
• "Comparison of several reweighted l1-algorithms for solving cardinality minimization problems" (Abdi, 2013)
• "Noisy Signal Recovery via Iterative Reweighted L1-Minimization" (0904.3780)
• "Iterative Refinement and Flexible Iteratively Reweighed Solvers for Linear Inverse Problems with Sparse Solutions" (Onisk et al., 4 Feb 2025)
• "Iteratively Reweighted Least Squares for Phase Unwrapping" (Dubois-Taine et al., 18 Jan 2024)
• "Improved Convergence for $\ell_\infty$ and $\ell_1$ Regression via Iteratively Reweighted Least Squares" (Ene et al., 2019)
• "A biconvex analysis for Lasso l1 reweighting" (Fosson, 2018)
• "Conjugate gradient acceleration of iteratively re-weighted least squares methods" (Fornasier et al., 2015)