Coordinate Descent Converges Faster with the Gauss-Southwell Rule Than Random Selection (1506.00552v2)

Abstract: There has been significant recent work on the theory and application of randomized coordinate descent algorithms, beginning with the work of Nesterov [SIAM J. Optim., 22(2), 2012], who showed that a random-coordinate selection rule achieves the same convergence rate as the Gauss-Southwell selection rule. This result suggests that we should never use the Gauss-Southwell rule, as it is typically much more expensive than random selection. However, the empirical behaviours of these algorithms contradict this theoretical result: in applications where the computational costs of the selection rules are comparable, the Gauss-Southwell selection rule tends to perform substantially better than random coordinate selection. We give a simple analysis of the Gauss-Southwell rule showing that---except in extreme cases---its convergence rate is faster than choosing random coordinates. Further, in this work we (i) show that exact coordinate optimization improves the convergence rate for certain sparse problems, (ii) propose a Gauss-Southwell-Lipschitz rule that gives an even faster convergence rate given knowledge of the Lipschitz constants of the partial derivatives, (iii) analyze the effect of approximate Gauss-Southwell rules, and (iv) analyze proximal-gradient variants of the Gauss-Southwell rule.

• The paper provides a tighter convergence analysis showing the Gauss-Southwell rule converges faster than random selection for coordinate descent under strong-convexity.
• It refutes previous analyses suggesting equal rates, arguing practical GS superiority stems from non-tightness of prior theoretical bounds.
• The research introduces the Gauss-Southwell-Lipschitz (GSL) variant, supports findings with numerical experiments, and discusses practical implementation strategies for GS methods.

Analysis of Coordinate Descent Convergence with Gauss-Southwell Rule

The paper "Coordinate Descent Converges Faster with the Gauss-Southwell Rule Than Random Selection," by Nutini et al., revisits the performance differential between randomized and Gauss-Southwell (GS) coordinate selection strategies in coordinate descent methods for convex optimization. These methods are especially relevant for large-scale problems due to their iterative approach of optimizing one coordinate at a time. While randomized selection has been conventionally preferred for its computational efficiency, empirical observations show that the GS selection can, in practical terms, outperform the random strategy when its computational cost is comparable.

The key argument lies in re-evaluating the convergence rates previously established, notably by Nesterov’s widely acknowledged work. By providing a tighter analysis under strong-convexity conditions, Nutini et al. demonstrate that the GS rule, except in degenerate cases, offers faster convergence. This refines the understanding of the theoretical benefits of the GS rule, particularly highlighting that the practical superiority of GS observed in some empirical studies likely stems from the non-tightness of prior theoretical analyses rather than innate algorithmic characteristics.

The paper establishes that the GS rule achieves superior rates for specific problem classes outside the stringent bounds where both methods converge at similar speeds. Moreover, exact coordinate optimization is shown to enhance convergence for problems with particular sparsity structures. The introduction of the Gauss-Southwell-Lipschitz (GSL) rule as a novel variant, leveraging Lipschitz constants of partial derivatives, underscores this discussion, offering improved convergence rates and demonstrating efficacy in a broader array of scenarios compared to the classical GS rule.

Key empirical results presented in the paper substantiate the theoretical findings. Numerical experiments on both synthetic and real-world datasets reveal substantial performance gains in terms of iterations required for convergence. These computational experiments cover typical machine learning problems such as least squares and logistic regression, demonstrating the practical utility of the GS and GSL rules across diverse contexts.

The implications of this research extend into practical algorithm design. Developing efficient methods for implementing the GS rule in sparse graph-based problems is crucial. With improvements in computational strategies, as demonstrated with the use of max-heaps and nearest-neighbour search approximations, the implementation overhead for GS rules can be reduced, making them viable for real-time applications.

The paper also speculates on the potential for embedding these findings within more complex optimization scenarios, including non-smooth and composite optimization problems via proximal-gradient adaptations. It further points toward possible expansions incorporating parallel computing strategies or hybrid coordinates selection methods, potentially affording even greater speed-ups.

Conclusively, while random selection benefits from a simpler and often less costly computational complexity, this paper substantiates the mathematical and practical conditions under which GS-based methods not only match but surpass them. This nuanced understanding of GS and its variants could inform future trends in adaptive optimization strategies, especially important as problem scales and complexity continue to grow in modern applications. Future work should focus on efficiently scaling these methods and exploring their applicability in non-convex and more structured problem domains.