The Noise-Sensitivity Phase Transition in Compressed Sensing (1004.1218v1)

Abstract: Consider the noisy underdetermined system of linear equations: y=Ax0 + z0, with n x N measurement matrix A, n < N, and Gaussian white noise z0 ~ N(0,\sigma^2 I). Both y and A are known, both x0 and z0 are unknown, and we seek an approximation to x0. When x0 has few nonzeros, useful approximations are obtained by l1-penalized l2 minimization, in which the reconstruction \hxl solves min || y - Ax||^2/2 + \lambda ||x||_1. Evaluate performance by mean-squared error (MSE = E ||\hxl - x0||_2^2/N). Consider matrices A with iid Gaussian entries and a large-system limit in which n,N\to\infty with n/N \to \delta and k/n \to \rho. Call the ratio MSE/\sigma^2 the noise sensitivity. We develop formal expressions for the MSE of \hxl, and evaluate its worst-case formal noise sensitivity over all types of k-sparse signals. The phase space 0 < \delta, \rho < 1 is partitioned by curve \rho = \rhoMSE(\delta) into two regions. Formal noise sensitivity is bounded throughout the region \rho < \rhoMSE(\delta) and is unbounded throughout the region \rho > \rhoMSE(\delta). The phase boundary \rho = \rhoMSE(\delta) is identical to the previously-known phase transition curve for equivalence of l1 - l0 minimization in the k-sparse noiseless case. Hence a single phase boundary describes the fundamental phase transitions both for the noiseless and noisy cases. Extensive computational experiments validate the predictions of this formalism, including the existence of game theoretical structures underlying it. Underlying our formalism is the AMP algorithm introduced earlier by the authors. Other papers by the authors detail expressions for the formal MSE of AMP and its close connection to l1-penalized reconstruction. Here we derive the minimax formal MSE of AMP and then read out results for l1-penalized reconstruction.

• The paper establishes a phase transition boundary in compressed sensing by linking undersampling rate (δ) and sparsity ratio (ρ) through noise sensitivity analysis.
• It employs an analytical framework using Approximate Message Passing to derive explicit formulas for minimax noise sensitivity and optimal LASSO tuning.
• Computational experiments validate the theoretical predictions, confirming robust thresholds for reliable signal recovery even in noisy conditions.

The Noise-Sensitivity Phase Transition in Compressed Sensing: An Analytical Perspective

The paper "The Noise-Sensitivity Phase Transition in Compressed Sensing" by Donoho, Maleki, and Montanari presents a thorough investigation of phase transitions in the performance of the popular LASSO technique when applied to compressed sensing problems. The focus is on an underdetermined system of linear equations disturbed by Gaussian white noise, and the work aims to understand the trade-offs between measurement sparsity and noise sensitivity.

Context and Problem Formulation

In compressed sensing, the central problem is to recover a sparse signal from an underdetermined set of linear measurements. This paper considers the model $y = Ax^0 + z^0$, where $A$ is a known $n \times N$ measurement matrix with $n < N$, $x^0$ is the unknown sparse signal, and $z^0$ is Gaussian noise with variance $\sigma^2$. It utilizes $\ell_1$-penalized $\ell_2$ minimization or LASSO to approximate $x^0$.

Key Contributions

1. Phase Transition Analysis: The paper delineates a phase boundary in the space defined by undersampling rate $\delta = n/N$ and sparsity ratio $\rho = k/n$, based on noise sensitivity considerations. It presents a formal division into a stable region where noise sensitivity is bounded ($\rho < MSE(\delta)$) and an unstable region where it becomes unbounded ($\rho \geq MSE(\delta)$). The boundary curve is derived from the mean-squared error (MSE) analysis and is shown to coincide with the boundary for $\ell_1-\ell_0$ equivalence in the noiseless case.
2. Analytical Framework via Approximate Message Passing (AMP): The paper extends the use of AMP, a message-passing algorithm known for its computational efficiency, to evaluate the formal MSE. The authors derive explicit formulas for minimax noise sensitivity using AMP and propose optimal tuning of the LASSO parameter $\lambda$ in terms of minimizing this worst-case MSE.
3. Game-Theoretic Interpretation: The problem is framed as a minimax game involving interactions between estimator strategies and nature’s worst-case signals. The optimal estimator penalization parameter and least-favorable signal distributions are identified using this framework.
4. Computational Validation: Theoretical predictions of phase boundaries and sensitivity metrics are corroborated with extensive computational experiments. This includes testing the derived least-favorable signals and game-theoretic structures such as saddlepoints by employing the LASSO on synthetic data matrices with Gaussian entries.

Implications and Future Directions

The paper's findings have significant implications for designing signal recovery methods in noisy settings. The demonstrated equivalence of phase transitions for both noisy and noiseless cases aids in understanding the thresholds for reliable signal recovery using LASSO. By bridging areas between information theory, robust statistics, and statistical physics, this paper also opens avenues for further exploration into universality across different types of matrices and noise models, which remains an open mathematical challenge.

Moreover, verifying these results in non-Gaussian settings and developing rigorous proofs for the observed phase transitions across a wider class of problems would broaden the applicability of these methods. The framework could also influence future developments in adaptive algorithm designs that incorporate learned priors from real-world data, potentially improving performance in practical scenarios.

In conclusion, this work not only provides a detailed theoretical underpinning for noise sensitivity in compressed sensing but also sets a precedent for the role of game-theoretic approaches in understanding estimation error bounds under adversarial conditions.