AMP-Inspired Deep Networks for Sparse Linear Inverse Problems (1612.01183v2)

Abstract: Deep learning has gained great popularity due to its widespread success on many inference problems. We consider the application of deep learning to the sparse linear inverse problem, where one seeks to recover a sparse signal from a few noisy linear measurements. In this paper, we propose two novel neural-network architectures that decouple prediction errors across layers in the same way that the approximate message passing (AMP) algorithms decouple them across iterations: through Onsager correction. First, we propose a "learned AMP" network that significantly improves upon Gregor and LeCun's "learned ISTA." Second, inspired by the recently proposed "vector AMP" (VAMP) algorithm, we propose a "learned VAMP" network that offers increased robustness to deviations in the measurement matrix from i.i.d. Gaussian. In both cases, we jointly learn the linear transforms and scalar nonlinearities of the network. Interestingly, with i.i.d. signals, the linear transforms and scalar nonlinearities prescribed by the VAMP algorithm coincide with the values learned through back-propagation, leading to an intuitive interpretation of learned VAMP. Finally, we apply our methods to two problems from 5G wireless communications: compressive random access and massive-MIMO channel estimation.

• The paper introduces a novel LAMP network that unfolds AMP iterations into deep layers to significantly enhance sparse signal recovery.
• LVAMP extends the approach to non-i.i.d. settings, achieving robust convergence and precision for matrices with diverse singular values.
• Optimized shrinkage functions are jointly learned with linear transforms, effectively minimizing MSE and improving performance in 5G communications.

Overview of "AMP-Inspired Deep Networks for Sparse Linear Inverse Problems"

The paper "AMP-Inspired Deep Networks for Sparse Linear Inverse Problems," authored by Borgerding, Schniter, and Rangan, presents a compelling exploration into deep learning methodologies for solving the sparse linear inverse problem, specifically addressing the recovery of sparse signals from limited noisy measurements. The core proposition involves the development of two neural network architectures inspired by Approximate Message Passing (AMP) algorithms: "learned AMP" (LAMP) and "learned VAMP" (LVAMP).

Key Contributions

1. Introduction of Learned AMP (LAMP):
   • The paper introduces the LAMP network, building on the AMP algorithm known for its efficacy in sparse signal recovery. By unfolding the iteration process of AMP into deep network layers and learning the optimal network parameters such as linear transforms and thresholds, LAMP effectively achieves signal reconstruction.
   • Unlike AMP, LAMP incorporates learnable parameters, offering empirical improvements over traditional architectures like LISTA, primarily through enhanced network topology informed by Onsager correction.
2. Learned VAMP (LVAMP) Network:
   • Inspired by the VAMP algorithm, the LVAMP network extends the applicability of AMP-like strategies to matrices beyond the i.i.d. Gaussian assumption, performing robustly with right-rotationally invariant matrices.
   • The architecture benefits from interpretable parameterization reflective of MMSE estimation principles, achieving superiority in handling matrices with diverse singular value distributions.
   • LVAMP also demonstrates substantial convergence speed and precision gains by aligning learned parameters with theoretically optimal matched VAMP ones.
3. Enhanced Network Performance through Shrinkage Functions:
   • The paper explores several families of shrinkage functions, such as piecewise linear and exponential, for tuning LAMP and LVAMP networks more finely to problem specifics, leading to significant MSE minimization.
   • Joint learning of these shrinkage functions alongside linear transformations signifies a focused approach on optimizing deep network capability for signal recovery tasks.
4. Applications to 5G Communications:
   • The research contributes to practical domains including compressive random access and massive MIMO channel estimation, both pivotal in 5G communication systems.
   • By framing these problems as sparse linear inverse challenges, LAMP and LVAMP demonstrate competitive advantages over traditional methods, offering promising alternatives for efficient network access and channel state estimation.

Implications and Future Directions

The implications of this research span the theoretical enhancement of neural networks inspired by algorithmic principles, bringing about more robust and generalizable signal recovery tools. The fusion of unsupervised learning methodologies like AMP and deep learning architectures paves the way for high-performance computational solutions in sparse signal processing and beyond.

Future developments could extend these networks to broader signal and data types, including those involving complex and non-linear transformations. Furthermore, the potential expansion of LVAMP's understood models to mixed matrix types reflects a fertile ground for further research, likely benefitting fields like image recovery and generalized linear models extensively.

The paper's numerical findings affirm that AMP-inspired networks can achieve near-oracle performance levels, bridging the theoretical-experimental divide, and underscoring the potential for adapting classical algorithmic strategies within modern machine learning frameworks. This fusion could catalyze new lines of inquiry within the signal processing and AI community, particularly in contexts necessitating sparse recovery under atypical measurement conditions.