Expectation-Maximization Gaussian-Mixture Approximate Message Passing (1207.3107v3)

Abstract: When recovering a sparse signal from noisy compressive linear measurements, the distribution of the signal's non-zero coefficients can have a profound effect on recovery mean-squared error (MSE). If this distribution was apriori known, then one could use computationally efficient approximate message passing (AMP) techniques for nearly minimum MSE (MMSE) recovery. In practice, though, the distribution is unknown, motivating the use of robust algorithms like LASSO---which is nearly minimax optimal---at the cost of significantly larger MSE for non-least-favorable distributions. As an alternative, we propose an empirical-Bayesian technique that simultaneously learns the signal distribution while MMSE-recovering the signal---according to the learned distribution---using AMP. In particular, we model the non-zero distribution as a Gaussian mixture, and learn its parameters through expectation maximization, using AMP to implement the expectation step. Numerical experiments on a wide range of signal classes confirm the state-of-the-art performance of our approach, in both reconstruction error and runtime, in the high-dimensional regime, for most (but not all) sensing operators.

• The paper introduces the EM-GM-AMP algorithm, which combines EM and AMP techniques to achieve near-optimal MMSE recovery using Gaussian mixture models.
• It improves sparse recovery over traditional methods like LASSO by adaptively learning unknown signal distributions from noisy measurements.
• Empirical evaluations demonstrate significant gains in recovery accuracy and computational efficiency, broadening its applicability in compressive sensing.

Overview of Expectation-Maximization Gaussian-Mixture Approximate Message Passing

The paper by Vila and Schniter introduces a novel empirical-Bayesian approach to compressive sensing, leveraging Expectation-Maximization (EM) and Approximate Message Passing (AMP) to achieve improved sparse signal recovery. The primary goal is to learn and utilize unknown signal distributions, even when acquiring noisy compressive linear measurements. By modeling the non-zero coefficients of sparse signals using a Gaussian Mixture Model (GMM), the paper aims to perform Minimum Mean Squared Error (MMSE) recovery through adaptive parameter estimation.

The authors address a significant limitation in contemporary sparse recovery methods, like LASSO, which are robust but often suboptimal for non-least-favorable signal distributions. The proposed EM-GM-AMP algorithm efficiently learns the signal's statistical properties and demonstrates enhanced reconstruction of sparse signals, evidencing stronger performance in mean-squared error reduction and computational runtime.

Key Insights

• Sparse Signal Modeling: The paper constructs the signal model as a K-sparse vector within an additive white Gaussian noise (AWGN) framework, examining cases where accurate signal recovery is possible given specific matrix isometry properties.
• Improved Algorithmic Framework: The EM-GM-AMP approach addresses the unknown coefficient distributions by using Gaussian mixtures, further conditioning their estimation through expectation maximization combined with GAMP updates. This design approximates MMSE recovery in scenarios lacking prior statistical information on signal and noise.
• Empirical Evaluation: Through comprehensive numerical experiments across various signal classes, the authors highlight EM-GM-AMP’s superior capability to provide state-of-the-art performance in both recovery error and computational efficiency. Notably, the performance enhancements are assessed under empirical phase transitions for different signal distributions including Bernoulli-Gaussian, Bernoulli, and Bernoulli-Rademacher signals.
• Model Order Selection: The procedure also entails iterative model-order selection based on the Bayesian Information Criterion (BIC), providing a systematic approach for determining the number of mixture components. This refinement is shown to further enhance performance for specific signal structures.

Implications and Future Directions

The theoretical and practical implications of this work are extensive. Practically, by incorporating a flexible signal model and iterative parameter learning, practitioners can achieve near-optimal sparse recovery without exhaustive model tuning. Theoretically, this combination of EM and AMP signifies a significant advancement in accommodating complex signal distributions in high-dimensional settings.

Future research could extend the robustness of AMP frameworks by addressing non-zero-mean and super-Gaussian sensing matrices—a limitation outlined in this paper. Moreover, enhancing convergence guarantees and adapting the framework to suit broader noise distributions or non-additive distortions could open new avenues for development, especially given the algorithm's adaptability to various real and complex Gaussian settings.

In summary, the EM-GM-AMP paradigm offers a compelling alternative to standard sparse recovery techniques, combining statistical learning with iterative message passing to effectively recover sparse signals from noisy measurements. This methodology not only broadens the applicability of compressive sensing but also promises practical utility in fields requiring efficient, high-fidelity signal reconstruction.