Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes
(1212.1824v2)
Published 8 Dec 2012 in cs.LG, math.OC, and stat.ML
Abstract: Stochastic Gradient Descent (SGD) is one of the simplest and most popular stochastic optimization methods. While it has already been theoretically studied for decades, the classical analysis usually required non-trivial smoothness assumptions, which do not apply to many modern applications of SGD with non-smooth objective functions such as support vector machines. In this paper, we investigate the performance of SGD without such smoothness assumptions, as well as a running average scheme to convert the SGD iterates to a solution with optimal optimization accuracy. In this framework, we prove that after T rounds, the suboptimality of the last SGD iterate scales as O(log(T)/\sqrt{T}) for non-smooth convex objective functions, and O(log(T)/T) in the non-smooth strongly convex case. To the best of our knowledge, these are the first bounds of this kind, and almost match the minimax-optimal rates obtainable by appropriate averaging schemes. We also propose a new and simple averaging scheme, which not only attains optimal rates, but can also be easily computed on-the-fly (in contrast, the suffix averaging scheme proposed in Rakhlin et al. (2011) is not as simple to implement). Finally, we provide some experimental illustrations.
Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes
This paper by Ohad Shamir and Tong Zhang provides an in-depth analysis of Stochastic Gradient Descent (SGD) without relying on the traditional smoothness assumptions, which are often inapplicable in modern machine learning scenarios involving non-smooth objective functions. The authors focus on convex and strongly-convex non-smooth optimization problems, frequently encountered in applications like support vector machines.
Key Contributions
Convergence of Individual Iterates:
The paper establishes O(log(T)/T) suboptimality for the last SGD iterate in non-smooth convex cases, and O(log(T)/T) in non-smooth strongly convex cases. These results advance our understanding by being among the first finite-sample bounds applicable to individual iterates in non-smooth settings.
Averaging Schemes:
The authors introduce and analyze a novel running averaging scheme called polynomial-decay averaging, which not only meets minimax-optimal convergence rates but also can be computed on-the-fly. This surpasses the complexity of suffix averaging as proposed in previous works, which is not easily implementable without pre-determining stopping time T.
Improved Suffix Averaging Analysis:
The paper provides tighter bounds for suffix averaging, offering a clearer understanding of its performance spectrum concerning different parameters.
Implications and Future Directions
The absence of smoothness in the paper's core framework aligns with contemporary machine learning problems, demanding exploration beyond classical assumptions. The convergence results for individual iterates and new averaging schemes are particularly relevant for practitioners, suggesting more efficient ways to utilize SGD in real-world, large-scale applications without knowing stopping times in advance.
Despite these advances, several questions remain open:
Tightness of Existing Bounds: For both convex and strongly convex cases, questions about the tightness and potential improvements of these bounds encourage further analytical investigations.
High-Probability Variants: Transitioning these results into high-probability bounds could provide more robust assurances for practitioners, especially concerning the variability of the last iterate.
This paper’s contributions also underscore the versatility and vigor of SGD, especially in scenarios involving non-smooth objectives where traditional analytic tools fall short. As machine learning continues to evolve with increasingly complex models and large datasets, these insights could influence both theoretical and practical aspects in the optimization domain.
Practical Applications
The insights provided in this work are pivotal for implementing SGD in machine learning, particularly in convex optimization settings that do not meet the classical smoothness criteria. Given the high scalability and simplicity of SGD, these results can lead to more efficient algorithms capable of tackling a broad range of non-smooth optimization problems.
In closing, Shamir and Zhang's exploration of SGD contributes significantly to academic theory while simultaneously addressing pragmatic needs in machine learning, proposing solutions that bridge existing gaps in non-smooth optimization. This research paves the way for further investigations that could potentially enhance our understanding and application of SGD in diverse contexts.