---
title: Iteratively Reweighted Least Squares (IRLS)
url: https://www.emergentmind.com/topics/iteratively-reweighted-least-squares-irls
type: topic
---

# Iteratively Reweighted Least Squares (IRLS)

The Iteratively Reweighted Least Squares (IRLS) method is a majorization-minimization framework for solving non-smooth, non-convex regularized regression and matrix recovery problems. Rooted in robust statistics and signal processing, IRLS alternates between solving sequence of quadratic (weighted least-squares) subproblems and updating the weights based on the current iterate, allowing efficient surrogation of non-quadratic objectives such as ℓ₁/sparsity, ℓₚ quasi-norms, group sparsity, rank/Schatten quasi-norms, and composite low-rank plus sparse penalizations. IRLS generalizes from classical single-norm regularizations to heterogeneous combinations and delivers both practical speed and, in many cases, provable global or local convergence rates [1401.7413][1509.04063][1703.05038][2012.12250][2506.20533][2306.04961][1811.07472][2510.01729].

## 1. Mathematical Formulation and General Mechanism

The IRLS methodology addresses minimization problems of the generic form
\[
\min_{x\in\mathbb{R}^n}\;F(x),
\]
where F is typically a combination of a smooth loss (e.g., squared error) with one or more non-smooth regularization terms enforcing sparsity, low-rank, or structured properties. The core challenge is that such terms are non-differentiable or non-convex (e.g., ℓ₁-norm, Schatten-p quasi-norm for p<1, mixed norm surrogates).

The principal IRLS steps are:
- **Smoothing:** Non-smooth terms are approximated using a smoothing parameter (e.g. $\mu$ or $\varepsilon$).
- **Weighted Quadratic Majorization:** Quadratic upper bounds are constructed, yielding surrogates involving weighted least-squares in x (or Z for matrix problems).
- **Alternating Update:** On each iteration, (a) solve the weighted least-squares subproblem; (b) update weights as functions of the current iterate (e.g., $w_i^{(k)} = (|x_i^{(k)}|^2+\varepsilon^2)^{\alpha}$).
- **Continuation:** Smoothing parameters are reduced gradually, majorizing the original objective and yielding convergence to stationary points of the non-smooth problem in the limit.

Specific problem types include:
- **Sparse regression:** $\min_x \|A x - b\|_2^2 + \lambda \|x\|_1$ or generalized $\sum_k \lambda_k |x_k|^{q_k}$ [1511.08970]
- **Low-rank matrix recovery:** $\min_X \|\Phi(X)-Y\|_2^2 + \lambda \|X\|_{S_p}^p$ [1703.05038]
- **Mixed norm/structured objectives:** e.g., joint minimization of low-rank and column-sparsity via Schatten-p and $\ell_{2,q}$-norms [1401.7413][2306.04961]
- **Robust regression:** IRLS is widely used for $\ell_1$- or truncated-loss minimization under arbitrary corruptions [2006.14211][1902.06391]

## 2. Smoothing and Majorization-Minimization Principles

Non-smooth penalty terms are smoothed via padding techniques such as:
- For vector ℓₚ, replace $|x_i|^p$ with $(x_i^2+\varepsilon^2)^{p/2}$ [1511.08970][2304.10536]
- For matrices, use $\|\left[Z; \mu I \right]\|_{S_p}^p$ for Schatten-p, or log-det approximations for the rank [1401.7413][1811.07472].

The majorization-minimization (MM) approach uses quadratic surrogates that upper-bound the penalization and touch the objective at current iterate:
- For $\ell_p^p$, tight majorizers are constructed via
\[
|x|^p \leq \frac{p}{2} w(x_0) x^2 + \text{const}
\]
where $w(x_0) = (x_0^2+\varepsilon^2)^{(p-2)/2}$ [2304.10536][2308.05745].

Updating iterates via the minimizer of the quadratic surrogate ensures monotonic decrease in the objective. The smoothing/majorization proofs rely on the concavity of the penalization term (e.g., $x^{p/2}$ for $0<p<2$) and trace inequalities for matrix norms [1401.7413][1703.05038].

## 3. Algorithmic Structure and Pseudocode

The IRLS iteration typically takes the following form:

**Generic IRLS Algorithm (Editor’s term)**
1. Given measurements $y$, operators $A$, regularization strengths, initial $x^{(0)}$, and smoothing parameter $\varepsilon_0$.
2. Until convergence:
   - Form weights $w_i^{(k)}$ from $x^{(k)}$.
   - Solve weighted least-squares for $x^{(k+1)}$ (possibly with constraints on $A x = y$), e.g.:
      \[
      x^{(k+1)} = \arg\min_{A x = y} \sum_i w_i^{(k)} x_i^2
      \]
   - Update smoothing parameter (e.g., $\varepsilon_{k+1} = \min(\varepsilon_k, \sigma_s(x^{(k+1)})/N)$, for best-$s$-term error).
   - Continue until change in $x$ is below desired tolerance.

A matrix IRLS step solving joint Schatten-$p$ and column $\ell_{2,q}$ penalty reads [1401.7413]:
\[
p Z M + \lambda q X^T (X Z - X) N = 0,
\]
where
\[
M = (Z^T Z + \mu^2 I)^{p/2-1},\quad N_{jj} = (\| (X Z - X)_j \|_2^2 + \mu^2)^{q/2-1}
\]

Algorithmic acceleration is often achieved via preconditioned conjugate gradient solvers [1509.04063][1411.5057], or embedding in bilevel or quasi-Newton frameworks for large-scale/ill-conditioned settings [2106.01429][2308.05745].

## 4. Convergence Theory

Convergence properties depend on the convexity (or strong convexity) of the smoothed objective and the underlying structure (e.g., null space property — NSP — of the measurement operator):

- **Global Convergence:** For convex smoothed objective ($p, q \geq 1$), IRLS globally converges to the unique minimizer [1401.7413][2012.12250]. For singleton minimizers in compressed sensing, global linear rate is established for certain IRLS variants [2012.12250][1902.06391][2510.01729].

- **Local Superlinear Convergence:** For nonconvex Schatten-$p$ minimization, HM-IRLS yields local superlinear convergence rate of order $2-p$ (Theorem: $\|X^{k+1} - X_0\|_{S_\infty} \leq C \|X^k - X_0\|_{S_\infty}^{2-p}$) in the presence of strong Schatten-$p$ null-space property [1703.05038].

- **Global recovery in robust regression:** Stagewise-truncated IRLS variants (STIR, STIR-GD) guarantee convergence from arbitrary initialization under sub-Gaussian covariates, for up to a breakdown fraction of adversarial corruptions [2006.14211]. This is achieved via stage-wise truncation and weighted strong convexity arguments.

- **Composite structures:** IRLS can handle joint row-sparsity and low-rank via quadratic surrogates for both structures, and local quadratic convergence is established under (r,s)-RIP [2306.04961].

- **Manifold settings:** Deterministic global convergence of IRLS (FMS-DS) for robust subspace recovery in Grassmann manifolds is proved under geometric data conditions [2506.20533].

## 5. Extensions and Applications

IRLS has broad impact across domains:

- **Compressed Sensing and Signal Recovery:** ℓ₁ and ℓₚ minimization for sparse vectors under linear measurements — with phase transitions favoring IRLS over first-order methods in high-dimensional regimes [1509.04063][2012.12250].

- **Low-Rank and Structured Matrix Completion:** Recovery of matrices with low-rank or structured (e.g., Hankel, Toeplitz) constraints, with IRLS outperforming alternatives for minimal sample complexity and difficult frequency separation [1811.07472][1703.05038].

- **Mixed/Composite Regularization:** Simultaneous recovery under multiple structure-inducing penalties, with IRLS offering empirical quadratic convergence and reduced need for manual regularization balancing [1401.7413][2306.04961].

- **Robust Regression/Subspace Recovery:** IRLS variants provably achieve global recovery and resistance to corruption, outperforming standard least squares and robust alternatives [2006.14211][2506.20533].

- **Statistical Estimation:** Fast estimation of polyserial and polychoric correlations via iteratively reweighted regression on conditional expectations [2210.11115].

- **Inverse Imaging Problems:** Learned IRLS networks with analysis-based ℓₚ or Schatten-$p$ regularizers yield state-of-the-art restoration in deblurring, super-resolution, and demosaicking at low parameter counts, with memory-efficient implicit backpropagation via bilevel optimization [2304.10536][2308.05745].

## 6. Computational Aspects and Practical Implementation

Per-iteration IRLS cost is dominated by weighted least-squares solves, which can be efficiently handled via conjugate gradient methods, especially when measurement/operator matrices have fast multiplication (FFT, DCT) or sparsity. Preconditioning schemes and generator-space computations further reduce dimensionality [1411.5057][1811.07472].

Algorithmic scalability:
- **Large-scale regression:** CG-IRLS and PCG-accelerated variants maintain feasibility up to millions of variables [1509.04063].
- **Structured matrices:** Generator-space IRLS for Hankel/block-Hankel structures achieves complexity $O(n R \log n)$ per randomized SVD and $O(n R^2)$ for mat-vecs [1811.07472].
- **Composite optimization:** Multi-structure IRLS updates handle weights and surrogates in tandem with automatic parameter selection from current estimate, without needing explicit trade-off parameters [2306.04961].
- **Neural network unrolling:** IRLS steps can be recast as layers in recurrent networks, with convergence guarantees enabling implicit differentiation and memory-efficient training [2308.05745][2304.10536].

## 7. Limitations, Remedies, and Theoretical Developments

- **Failure modes:** Classical IRLS can fail to converge at the critical sparsity order $k=K$ under NSP, as explicitly constructed in [1910.07095]. Parameter updates involving best-(K+1)-term error can remedy this and restore global and local linear convergence.
- **Nonconvexity:** PL-IRLS extends IRLS to nonconvex nonsmooth settings (e.g., $|x_j|^\nu$ for $\nu<1$), and global convergence is ensured via the Kurdyka-Łojasiewicz property and sufficiently regularized surrogates [1404.6026].
- **Smooth variable projection:** Bilevel reformulations yield smooth objectives without spurious minima, admitting superlinear convergence via quasi-Newton methods and efficient inner linear solves [2106.01429].

## References

- [1401.7413]: Smoothed Low Rank and Sparse Matrix Recovery by Iteratively Reweighted Least Squares Minimization
- [1509.04063]: Conjugate gradient acceleration of iteratively re-weighted least squares methods
- [1703.05038]: Harmonic Mean Iteratively Reweighted Least Squares for Low-Rank Matrix Recovery
- [2012.12250]: Iteratively Reweighted Least Squares for Basis Pursuit with Global Linear Convergence Rate
- [2510.01729]: Improved ℓ_{p} Regression via Iteratively Reweighted Least Squares
- [2506.20533]: Global Convergence of Iteratively Reweighted Least Squares for Robust Subspace Recovery
- [2308.05745]: Iterative Reweighted Least Squares Networks With Convergence Guarantees for Solving Inverse Imaging Problems
- [2306.04961]: Recovering Simultaneously Structured Data via Non-Convex Iteratively Reweighted Least Squares
- [1811.07472]: Denoising and Completion of Structured Low-Rank Matrices via Iteratively Reweighted Least Squares
- [2210.11115]: Iteratively Reweighte Least Squares Method for Estimating Polyserial and Polychoric Correlation Coefficients
- [1411.5057]: Fast Iteratively Reweighted Least Squares Algorithms for Analysis-Based Sparsity Reconstruction
- [1511.08970]: An Iteratively Reweighted Least Squares Algorithm for Sparse Regularization
- [2304.10536]: Learning Sparse and Low-Rank Priors for Image Recovery via Iterative Reweighted Least Squares Minimization
- [1404.6026]: Proximal linearized iteratively reweighted least squares for a class of nonconvex and nonsmooth problems
- [1902.06391]: Improved Convergence for $\ell_\infty$ and $\ell_1$ Regression via Iteratively Reweighted Least Squares
- [1910.07095]: IRLS for Sparse Recovery Revisited: Examples of Failure and a Remedy

IRLS constitutes a foundational algorithmic paradigm for structured recovery and regularization in high-dimensional inference, offering generalizable methodologies, rigorous theoretical foundations (often matching information-theoretic limits), scalable computation, and broad adaptability across statistical, optimization, and learning domains.

Source: https://www.emergentmind.com/topics/iteratively-reweighted-least-squares-irls