Iterative Reweighted Schemes Overview

Updated 25 May 2026

Iterative reweighted schemes are iterative algorithms that replace challenging nonconvex penalties with a sequence of weighted convex surrogates, improving recovery thresholds and accuracy.
They employ a majorization–minimization framework to construct surrogate functions that guarantee monotonic decrease and convergence to a stationary point under mild conditions.
Widely applied in compressed sensing, super-resolution, and robust regression, these methods effectively exploit sparsity and low-rank structures in complex inverse problems.

Iterative reweighted schemes are a class of algorithms that address non-convex, non-smooth, or structured regularization in optimization, signal processing, and inverse problems by transforming challenging objectives into a sequence of tractable surrogate problems. These methods unify diverse sparsity-promoting and low-rank models by iteratively updating problem weights or parameters and solving efficiently reweighted subproblems. Widely studied in compressed sensing, super-resolution, robust regression, and large-scale inverse problems, they provide a flexible framework for exploiting structure beyond standard convex relaxations, frequently achieving improved recovery thresholds, higher accuracy, and robustness to model or data imperfections.

1. Fundamental Principles of Iterative Reweighted Schemes

The core principle of iterative reweighted schemes is the replacement of a challenging (often nonconvex) penalty, such as the ℓ₀ norm, the log-sum penalty, or block/structured quasi-norms, with a sequence of weighted convex surrogates. At each iteration, the surrogate or majorizing function is constructed to tightly upper bound the non-convex penalty at the current estimate, leading to an update rule with desirable mathematical and algorithmic properties.

Given a generic optimization problem

$\min_{x} \; f(x) + \Phi(x),$

where $\Phi$ is a non-convex, typically sparsity- or low-rank-promoting penalty, iterative reweighted schemes majorize $\Phi$ at the current point $x^{(t)}$ via a convex upper bound $Q(x \mid x^{(t)})$ , then solve the weighted subproblem

$x^{(t+1)} = \arg\min_{x} \; f(x) + Q(x \mid x^{(t)}).$

The form of $Q$ depends on $\Phi$ but is commonly quadratic or linearly weighted, allowing for efficient subproblem solutions.

Notable examples include:

Reweighted ℓ₁ minimization for sparse recovery, where weights are inversely proportional to current coefficient magnitudes, promoting sparsity more aggressively than standard ℓ₁ penalties (0904.0994).
Reweighted mixed-norms (e.g., group-Lasso, block-separable, Frobenius/block norms) to recover structured sparsity or low-rankness (Strohmeier et al., 2016).
Log-sum and nonconvex penalties majorized by quadratic surrogates, enabling joint parameter and sparse coefficient updates (Fang et al., 2014, Fang et al., 2014).

2. Majorization-Minimization and Surrogate Construction

A distinguishing feature is the use of the majorization–minimization (MM) framework, which constructs, at each iteration, a surrogate function $Q$ satisfying

$Q(x \mid x^{(t)}) \geq \Phi(x), \quad Q(x^{(t)} \mid x^{(t)}) = \Phi(x^{(t)}).$

For example, the log-sum penalty $\Phi$ 0 is majorized at $\Phi$ 1 by

$\Phi$ 2

(Fang et al., 2014, Fang et al., 2014). This surrogate is quadratic in $\Phi$ 3, admits closed-form or efficient numerical updates, and guarantees monotonic decrease of the true objective.

Similarly, surrogate construction for $\Phi$ 4-quasi-norm penalties (for $\Phi$ 5) exploits concavity: $\Phi$ 6 (Lu, 2012, Strohmeier et al., 2016, Lefkimmiatis et al., 2023).

Block- or group-based surrogates are majorized using group norms or matrix functions, which can be efficiently solved via block coordinate descent or proximal operators (Strohmeier et al., 2016, Yang et al., 2015).

3. Iterative Update Rules and Algorithmic Variants

The iterative update sequence is as follows:

Weight computation: Set weights inversely proportional (or otherwise adaptively dependent) to the magnitude or structure (e.g., group norm, singular values) of the current iterate.
Surrogate minimization: Solve the weighted subproblem (ℓ₁, ℓ₂, or group norm) subject to data or structural constraints.
Auxiliary variable updates: In structured or Bayesian variants, update additional parameters (e.g., covariance matrices, support blocks, Mahalanobis metrics) to capture underlying correlations or group structure (Zhang et al., 2011).

Variants include:

Classic IRLS (Iteratively Reweighted Least Squares), where $\Phi$ 7 (or general $\Phi$ 8) penalties are replaced by weighted $\Phi$ 9 surrogates (Lu, 2012, Ene et al., 2 Oct 2025).
IRL1 and IRL2: Iteratively reweighted ℓ₁ and ℓ₂ minimization families, with convergence guarantees under both dynamic and fixed (appropriately chosen) smoothing parameters (Lu, 2012).
Majorization-minimization for log-sum via quadratic upper bounding, as in super-resolution compressed sensing (Fang et al., 2014, Fang et al., 2014).
Block-iterative and adaptive block/group reweighting for structured sparsity or atomic norm minimization (Cho et al., 2015).
Doubly reweighted methods in the presence of both nonconvex regularization and robust loss functions, yielding "doubly IRLS" subproblems (Sun et al., 2022).
Extrapolation-accelerated IRL1 algorithms, which blend Nesterov-style momentum into the iterative reweighting process, with cluster-point and, under Kurdyka–Łojasiewicz assumptions, global convergence (Yu et al., 2017).

A generic pseudocode form is:

$\Phi$ 3

4. Theoretical Properties: Convergence, Recovery, and Phase Transitions

Majorization-based reweighted schemes guarantee:

Monotonic decrease of the true objective (due to surrogate tightness).
Convergence of the sequence $\Phi$ 0 to a stationary point, under mild regularity conditions (smoothness of $\Phi$ 1, boundedness of level sets).
Under certain parameter regimes and for appropriately diminishing smoothing, global convergence to a true solution or local minimum.

In compressed sensing and sparse recovery, iterative reweighting provably increases recoverable sparsity thresholds relative to standard convex relaxations.

The Grassmann angle framework precisely quantifies the phase transition—i.e., the critical measurement-to-unknown ratio for successful recovery—and shows that support-adaptive reweighting strictly widens the recovery region for large classes of structured signals (0904.0994).
In noisy settings, reweighted methods improve error constants, with error bounds tighter than those of unweighted ℓ₁ methods, providing robustness to perturbations (0904.3780).
Stagewise or robust IRLS variants (e.g., STIR-GD) achieve global linear convergence and error contraction even under adversarial corruption, provided hyperparameters are chosen within explicit theoretical bounds (Mukhoty et al., 2020).

For non-convex, nonsmooth penalties or block/group-structured models, fixed or adaptively decaying smoothing ensures any accumulation point is stationary for the original (non-smoothed) problem (Lu, 2012, Wang et al., 2018). In block or tensor settings, group log-sum or nonconvex block-quasinorms are handled via block-wise reweighting and surrogate minimizations (Strohmeier et al., 2016, Yang et al., 2015).

5. Applications Across Signal Processing, Inverse Problems, and Machine Learning

Iterative reweighted schemes support a wide spectrum of applications, including:

Application Domain	Representative Schemes & References
Compressed Sensing, Sparse Recovery	IRL1/IRL2, MM log-sum, Block-IRL1 (0904.0994, Fang et al., 2014, Lu, 2012)
Super-Resolution & Off-grid Recovery	Joint parameter-log-sum MM (Fang et al., 2014, Fang et al., 2014, Cho et al., 2015)
Robust & Group-sparse Regression	IRLS, dual IRLS, group-aware IRLS (Ene et al., 2 Oct 2025, Mukhoty et al., 2020, Kaushik et al., 2024, Storn, 27 Mar 2026)
MEG/EEG Source Imaging	irMxNE, block-reweighted Frobenius/group penalties (Strohmeier et al., 2016)
Tensor Decomposition	Iterative reweighted log-sum for Tucker, block-penalty minimization (Yang et al., 2015)
Inverse Imaging & Learning	IRLS with bilevel/implicit learning, unrolled MM networks (Lefkimmiatis et al., 2023, Koshelev et al., 2023)
Nonconvex/Nonsmooth Optimization	AIR, IRW, robust feature selection (Wang et al., 2018, Nie et al., 2019)
Constrained Robust Compressed Sensing	Doubly iteratively reweighted (ℓ₁-ℓ₂), inexact MM (Sun et al., 2022)

These methods offer superior performance on structured problems where traditional convex relaxations are suboptimal, including cases with highly correlated sources, block structures, parameterized or continuous dictionaries, ill-conditioned or incomplete data, and high adversarial corruption.

6. Algorithmic Enhancements, Learning, and Scalability

Recent advances encompass:

Flexible Krylov solvers and iterative refinement techniques for large-scale and ill-posed inverse problems, decoupling the memory and computational cost from classic IRLS bottlenecks (Onisk et al., 4 Feb 2025).
Unrolled and recurrent IRLS networks, with convergence guarantees, implemented via modular least-squares blocks and weight modules that can be learned directly from data, using bilevel optimization and implicit differentiation for scalable end-to-end training (Koshelev et al., 2023, Lefkimmiatis et al., 2023).
Dual IRLS and primal-dual frameworks, which are provably more stable at high p-exponents or for graph p-Laplacians and provide linear convergence in both ℓₚ-regression and variational graph learning (Ene et al., 2 Oct 2025, Storn, 27 Mar 2026).
Block or group-structured reweighting for improved sparsity model exploitation and statistically amplified recovery in group-sparse, block-sparse, and multi-task contexts (Cho et al., 2015, Strohmeier et al., 2016, Kaushik et al., 2024).

7. Open Challenges and Future Perspectives

While iterative reweighted schemes yield strict improvements over fixed-penalty models and convex relaxations in many settings, several challenges remain:

Theoretical global convergence for non-convex penalties is established largely at the stationarity level, with no universal guarantees of optimality, except in certain noise-free or idealized regimes (e.g., exact parameter recovery when M ≥ 2K atoms (Fang et al., 2014)).
Selection and adaptation of the smoothing parameters (e.g., $\Phi$ 2) can be critical; fixed-ε schemes with Lipschitz approximations yield enhanced numerical stability (Lu, 2012).
Empirical complexity and scalability have improved with modern MM and Krylov/CG backbones, though some applications (e.g., high-order tensors, hypergraphs) demand further numerical innovation (Yang et al., 2015, Storn, 27 Mar 2026).
The application of iterative reweighting in learned, distributed, or streaming settings is active, with ongoing research in scalable learning of regularization structures, robust performance under model misspecification, and integration with modern deep architectures (Koshelev et al., 2023, Lefkimmiatis et al., 2023, Kaushik et al., 2024).

Overall, iterative reweighted schemes represent a mature yet dynamically evolving toolkit in high-dimensional estimation, sparse signal processing, robust learning, and computational imaging, combining algorithmic tractability, theoretical rigor, and broad applicability across modern data analysis challenges.