A General Iterative Shrinkage and Thresholding Algorithm for Non-convex Regularized Optimization Problems (1303.4434v1)

Abstract: Non-convex sparsity-inducing penalties have recently received considerable attentions in sparse learning. Recent theoretical investigations have demonstrated their superiority over the convex counterparts in several sparse learning settings. However, solving the non-convex optimization problems associated with non-convex penalties remains a big challenge. A commonly used approach is the Multi-Stage (MS) convex relaxation (or DC programming), which relaxes the original non-convex problem to a sequence of convex problems. This approach is usually not very practical for large-scale problems because its computational cost is a multiple of solving a single convex problem. In this paper, we propose a General Iterative Shrinkage and Thresholding (GIST) algorithm to solve the nonconvex optimization problem for a large class of non-convex penalties. The GIST algorithm iteratively solves a proximal operator problem, which in turn has a closed-form solution for many commonly used penalties. At each outer iteration of the algorithm, we use a line search initialized by the Barzilai-Borwein (BB) rule that allows finding an appropriate step size quickly. The paper also presents a detailed convergence analysis of the GIST algorithm. The efficiency of the proposed algorithm is demonstrated by extensive experiments on large-scale data sets.

• The paper introduces the GIST algorithm that efficiently solves non-convex optimization problems by computing proximal steps for various sparsity-inducing penalties.
• It leverages the Barzilai-Borwein rule to significantly accelerate convergence, as validated on high-dimensional sparse datasets.
• Comprehensive theoretical analysis and empirical experiments confirm GIST’s convergence properties and practical superiority over convex relaxation methods.

Overview of the General Iterative Shrinkage and Thresholding Algorithm for Non-convex Regularized Optimization Problems

The paper "A General Iterative Shrinkage and Thresholding Algorithm for Non-convex Regularized Optimization Problems" presents a novel algorithm, GIST, designed to efficiently address non-convex optimization problems characterized by non-convex sparsity-inducing penalties. Non-convex regularizers have emerged as powerful tools in sparse learning because they overcome some limitations of their convex counterparts, particularly in approximations of the $\ell_0$-norm. Still, solving the associated non-convex optimization problems presents significant challenges due to their inherent computational complexity.

Contributions

The major contribution of this research is the development of the General Iterative Shrinkage and Thresholding (GIST) algorithm. This algorithm effectively iterates solutions for a large class of non-convex penalties by solving a proximal operator problem, which admits a closed-form solution in several cases. By utilizing the Barzilai-Borwein (BB) rule for line search initialization, the GIST algorithm improves convergence speed significantly, offering a pragmatic solution for large-scale datasets.

The paper offers a comprehensive analysis of the convergence properties of the GIST algorithm, ensuring that it reaches critical points under both monotone and non-monotone line search criteria. Extensive experimental validation demonstrates GIST's efficiency compared to existing methods such as Multi-Stage convex relaxation (MS).

Technical Approach

The manuscript systematically constructs the theoretical framework underpinning the GIST algorithm. It addresses the optimization problem of the form $f(\mathbf{w}) = l(\mathbf{w}) + r(\mathbf{w})$, with $l(\mathbf{w})$ having Lipschitz continuous gradients and $r(\mathbf{w})$ expressed as the difference of two convex functions.

One noteworthy feature is the algorithm's competence in handling a broad range of non-convex penalties, including the $\ell_q$-norm, Smoothly Clipped Absolute Deviation (SCAD), Log-Sum Penalty (LSP), and Minimax Concave Penalty (MCP). For each of these, the efficient computation of proximal operators is facilitated by solutions that address potential non-smoothness and non-convexity.

The paper provides numerical results that showcase GIST's performance across various high-dimensional sparse datasets, demonstrating acceleration in convergence compared to other methods like Sequential Convex Programming (SCP).

Implications

The development of GIST represents a significant advancement for non-convex optimization in the domain of sparse learning. By providing both a theoretical foundation and empirical results that validate its effectiveness, the paper contributes significantly to the understanding and further development of non-convex optimization techniques.

This work is particularly noteworthy for researchers who are focused on advancing machine learning applications that require efficient and effective optimization algorithms. With its solid theoretical grounding and observed empirical efficiency, GIST holds potential for application across disciplines where large-scale non-convex optimization is essential, including signal processing, statistical learning, and beyond.

Future Directions

Future research avenues include theoretical investigations into the performance bounds of the GIST algorithm, such as parameter estimation and prediction error bounds. Moreover, enhancements to address multi-task learning scenarios or similar complex frameworks could extend the applicability of GIST further. Expanding this line of inquiry may involve integrating the algorithm with emerging computational paradigms to accommodate even larger datasets or more complex regularization structures.

In summary, the GIST algorithm represents a significant stride toward practical and efficient non-convex optimization, equipped to handle the complexities of modern sparse learning problems. This research lays a robust foundation for subsequent advancements and applications in various scientific and engineering domains.