Majorization-Minimization Dual Stagewise Algorithm for Generalized Lasso (2501.02197v1)

Abstract: The generalized lasso is a natural generalization of the celebrated lasso approach to handle structural regularization problems. Many important methods and applications fall into this framework, including fused lasso, clustered lasso, and constrained lasso. To elevate its effectiveness in large-scale problems, extensive research has been conducted on the computational strategies of generalized lasso. However, to our knowledge, most studies are under the linear setup, with limited advances in non-Gaussian and non-linear models. We propose a majorization-minimization dual stagewise (MM-DUST) algorithm to efficiently trace out the full solution paths of the generalized lasso problem. The majorization technique is incorporated to handle different convex loss functions through their quadratic majorizers. Utilizing the connection between primal and dual problems and the idea of ``slow-brewing'' from stagewise learning, the minimization step is carried out in the dual space through a sequence of simple coordinate-wise updates on the dual coefficients with a small step size. Consequently, selecting an appropriate step size enables a trade-off between statistical accuracy and computational efficiency. We analyze the computational complexity of MM-DUST and establish the uniform convergence of the approximated solution paths. Extensive simulation studies and applications with regularized logistic regression and Cox model demonstrate the effectiveness of the proposed approach.

• The paper presents the MM-DUST algorithm achieving efficient optimization of the generalized lasso problem through majorization-minimization and dual stagewise updates.
• It formulates a dual space approach that converts the problem into a box-constrained convex model, thereby reducing computational complexity in high-dimensional settings.
• The algorithm demonstrates uniform convergence and accuracy in tracing solution paths, making it suitable for complex models like logistic regression and Cox proportional hazards.

Overview of the Majorization-Minimization Dual Stagewise Algorithm for Generalized Lasso

The paper introduces a novel optimization algorithm called the Majorization-Minimization Dual Stagewise (MM-DUST) algorithm, which is designed to address computational challenges associated with the generalized lasso problem, particularly in non-Gaussian and non-linear models. Generalized lasso is a powerful extension of the traditional lasso method that incorporates structural regularization by applying an $\ell_1$ penalty to a linear transformation of parameters. This allows for broad applications such as fused lasso, clustered lasso, and constrained lasso.

Algorithmic Framework

The MM-DUST algorithm departs from previous methods, which primarily focus on linear models, by efficiently handling the generalized lasso in non-linear settings. The algorithm employs a three-pronged strategy:

1. Majorization-Minimization (MM): The algorithm applies majorization-minimization techniques to address a diverse array of convex loss functions by utilizing quadratic majorizers. By iteratively alternating between majorization and minimization steps, a surrogate function simplifies the optimization, ultimately facilitating computational tractability.
2. Dual Space Optimization: Instead of tackling the primal problem directly, MM-DUST solves the problem in the dual space. This leads to a box-constrained convex problem, which is computationally efficient to handle since it reduces dimensionality concerns associated with large datasets.
3. Stagewise Learning: Incorporating concepts from stagewise learning, the algorithm updates dual parameters incrementally, trading off step size to achieve a balance between statistical accuracy and computational efficiency. The gradual complexity increment via learning rate adjustments enhances the algorithm's flexibility and performance in tracing full solution paths.

Theoretical Insights

The paper establishes theoretical guarantees for the MM-DUST algorithm. It demonstrates uniform convergence of the solution paths and a computational complexity that scales efficiently with data size. Specifically, it shows that the MM-DUST paths converge to the exact generalized lasso solution paths as the step size approaches zero. This ensures that the balance between path-fitting accuracy and computational cost can be effectively managed.

Practical Implications

The MM-DUST algorithm exhibits significant promise for real-world applications, especially in scenarios with complex loss functions such as logistic regression and Cox proportional hazards models. The simulation studies confirm the potential of MM-DUST in reducing computational time while maintaining accuracy, making it well-suited for high-dimensional datasets where traditional methods may struggle.

Future Directions

The research invites further exploration of adaptive step sizes, which could potentially enhance computational savings even further. Additionally, extending the framework to accommodate other regularization structures or non-convex penalties presents an enticing avenue for future investigation. The integration of more sophisticated tuning strategies could also bolster the applicability of MM-DUST in diverse machine learning contexts.

In summary, this paper contributes a robust algorithmic solution to the generalized lasso problem, broadening the toolkit available for handling large-scale, complex statistical models. The MM-DUST algorithm not only advances theoretical understanding but also provides a practical, scalable approach for addressing real-world data challenges.