Online Alternating Direction Method

Abstract: Online optimization has emerged as powerful tool in large scale optimization. In this paper, we introduce efficient online algorithms based on the alternating directions method (ADM). We introduce a new proof technique for ADM in the batch setting, which yields the O(1/T) convergence rate of ADM and forms the basis of regret analysis in the online setting. We consider two scenarios in the online setting, based on whether the solution needs to lie in the feasible set or not. In both settings, we establish regret bounds for both the objective function as well as constraint violation for general and strongly convex functions. Preliminary results are presented to illustrate the performance of the proposed algorithms.

Efficient Online Algorithms Based On the Alternating Direction Method

The paper "Online Alternating Direction Method" by Huahua Wang and Arindam Banerjee addresses the adaptation of the Alternating Direction Method (ADM) to the online learning framework, providing both theoretical and empirical insights. This research contributes to the field of online optimization, with a specific focus on enhancing the applicability of ADM, a method already noted for its adaptability and empirical success in various optimization challenges, particularly those involving composite objectives.

In the context of batch optimization, ADM has been useful for problems with linear equality constraints, and the paper extends its utility to online settings where constraints may similarly exist. By introducing a novel proof technique for ADM in batch settings, the authors establish a convergence rate of $O(1/T)$, an important result that supports regret analysis in online settings. The consideration of both the presence and absence of constraints in solution spaces highlights the practicality of the research. The studies focus on regret bounds concerning the objective function and constraint violations, both for general and strongly convex functions, underscoring the robustness of the proposed online ADM (OADM) algorithms.

The paper effectively demonstrates that OADM can be considered on par with existing methods like COMID and RDA in handling composite online optimization problems. However, OADM's additional capability to manage linear equality constraints brings a noteworthy enhancement compared to previous models. Through both theoretical and experimental explorations, the authors provide evidence that the proposed algorithms achieve sublinear regret bounds, a central criterion for online learning performance. Specifically, they prove that OADM can achieve an $O(\sqrt{T})$ bound on both objective and constraint violation regrets, a significant finding, as traditional online algorithms typically do not accommodate such linear constraints explicitly.

The experimental results serve as an illustrative validation of the theoretical claims, showing the algorithm's efficiency in solving practical problems such as generalized lasso and total variation. When compared to established methods like FOBOS and RDA, OADM not only demonstrates competitive performance in selecting sparse dimensions and recovering data but also showcases its superiority in handling distributed optimization scenarios where distributed variables must reach global consensus.

The implications of this research are both practical and theoretical. Theoretically, it provides a comprehensive analysis framework for online constrained optimization that could stimulate the development of further analytic tools and methodologies. Practically, it promises more efficient, distributed online optimization solutions which can be vital for large-scale data scenarios prevalent in machine learning applications today.

Looking ahead, given the continuous evolution and application of large-scale distributed systems, the theories and methods presented in this paper could be foundational for further innovations in related areas. Future research could explore extending these methods to more diverse constraint types and different architectures, including varied network structures and data distributions. Such directions could further solidify the role of ADM and its variants as a cornerstone in machine learning and optimization disciplines.