Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds

Abstract: In recent years, unfolding iterative algorithms as neural networks has become an empirical success in solving sparse recovery problems. However, its theoretical understanding is still immature, which prevents us from fully utilizing the power of neural networks. In this work, we study unfolded ISTA (Iterative Shrinkage Thresholding Algorithm) for sparse signal recovery. We introduce a weight structure that is necessary for asymptotic convergence to the true sparse signal. With this structure, unfolded ISTA can attain a linear convergence, which is better than the sublinear convergence of ISTA/FISTA in general cases. Furthermore, we propose to incorporate thresholding in the network to perform support selection, which is easy to implement and able to boost the convergence rate both theoretically and empirically. Extensive simulations, including sparse vector recovery and a compressive sensing experiment on real image data, corroborate our theoretical results and demonstrate their practical usefulness. We have made our codes publicly available: https://github.com/xchen-tamu/linear-lista-cpss.

• The paper establishes a theoretical guarantee for the linear convergence of LISTA under ideal conditions.
• It introduces asymptotic coupling of weights and an adaptive thresholding scheme to improve sparse recovery performance.
• Numerical experiments show that LISTA outperforms ISTA and FISTA in speed and accuracy for compressive sensing tasks.

Theoretical Linear Convergence of Unfolded ISTA and Its Practical Weights and Thresholds

The paper "Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds" provides a comprehensive study of the unfolding of the Iterative Shrinkage Thresholding Algorithm (ISTA) into neural network structures, namely Learned ISTA (LISTA). The authors focus on theoretical guarantees to address the underpinnings of this learning-based model, which has shown empirical success in solving sparse recovery problems.

Key Contributions

The authors establish theoretical convergence rates and parameter dependencies for LISTA, a neural network approximation of ISTA. The primary contributions include:

1. Asymptotic Coupling in Weights: The paper introduces a structure for weight coupling across layers, which is crucial to achieve convergence. This result simplifies the network by reducing the number of parameters required while preserving performance in sparse recovery tasks.
2. Linear Convergence Proof: For the first time, the paper presents a proof for the linear convergence rate of LISTA under ideal conditions. Conventional ISTA and its variant, FISTA, typically exhibit sublinear convergence, making this a significant theoretical improvement.
3. Support Selection Incorporation: The authors propose introducing a thresholding scheme to enhance support selection, thereby boosting convergence both in theory and practice.
4. Numerical Validation: Through extensive simulations including sparse vector recovery and compressive sensing experiments on real image datasets, the authors validate the theoretical claims. The experiments demonstrate that coupled weight structures and adaptive thresholds significantly enhance LISTA's convergence speed and accuracy compared to ISTA and FISTA.

Implications and Future Directions

The theoretical advancements provided in this paper have several implications for the field of machine learning, particularly in the domain of signal processing and sparse coding:

• Practical Accelerations: By reducing trainable parameters and increasing convergence rates, LISTA becomes even more appealing for practical applications where computational efficiency and performance are crucial.
• Theoretical Insights: Establishing a linear convergence rate provides valuable insights into the behavior of unfolded iterative algorithms, potentially influencing the design of future neural network-based optimization algorithms.
• Generalizability: The approaches discussed could be extended to other iterative procedures beyond ISTA, suggesting a broader avenue for research in adapting traditional optimization techniques into learnable structures.
• Robustness and Adaptability: The practical application to compressive sensing in natural images indicates that LISTA, with the proposed weight coupling and support selection strategies, remains robust and adaptable even under non-ideal conditions where assumptions are violated.

Overall, this study enhances the understanding of LISTA's convergence properties, laying the groundwork for future explorations into efficiently combining traditional algorithms with modern deep learning approaches to tackle high-dimensional sparse recovery tasks. The results not only bridge a critical gap in theoretical understanding but also serve as a guide for optimizing the performance of learning-based iterative methods in practical applications.