Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithm (1310.5715v5)

Abstract: We obtain an improved finite-sample guarantee on the linear convergence of stochastic gradient descent for smooth and strongly convex objectives, improving from a quadratic dependence on the conditioning $(L/\mu)^2$ (where $L$ is a bound on the smoothness and $\mu$ on the strong convexity) to a linear dependence on $L/\mu$. Furthermore, we show how reweighting the sampling distribution (i.e. importance sampling) is necessary in order to further improve convergence, and obtain a linear dependence in the average smoothness, dominating previous results. We also discuss importance sampling for SGD more broadly and show how it can improve convergence also in other scenarios. Our results are based on a connection we make between SGD and the randomized Kaczmarz algorithm, which allows us to transfer ideas between the separate bodies of literature studying each of the two methods. In particular, we recast the randomized Kaczmarz algorithm as an instance of SGD, and apply our results to prove its exponential convergence, but to the solution of a weighted least squares problem rather than the original least squares problem. We then present a modified Kaczmarz algorithm with partially biased sampling which does converge to the original least squares solution with the same exponential convergence rate.

• The paper presents an improved finite-sample guarantee for the linear convergence of SGD by reducing dependency on the conditioning parameter from quadratic to linear.
• It demonstrates that importance sampling enhances convergence by shifting reliance from worst-case to average smoothness, streamlining optimization.
• The work reveals a novel connection between SGD and the randomized Kaczmarz algorithm, offering insights for hybrid sampling techniques with enhanced noise tolerance.

Stochastic Gradient Descent and the Randomized Kaczmarz Algorithm

This paper investigates the convergence properties of Stochastic Gradient Descent (SGD) and establishes connections with the Randomized Kaczmarz algorithm, introducing the role of importance sampling within this context. The authors focus on smooth and strongly convex objective functions, offering substantial improvements in the theoretical guarantees associated with the linear convergence of SGD.

Key Contributions

1. Improved Convergence Rates: The paper presents an improved finite-sample guarantee for the linear convergence of SGD. Specifically, it reduces the quadratic dependence on the conditioning parameter $(L/\mu)^2$ to a linear dependence on $L/\mu$, where $L$ is the smoothness bound and $\mu$ is the strong convexity parameter. This adjustment significantly enhances the convergence rate under certain conditions.
2. Importance Sampling: By reweighting the sampling distribution, the authors demonstrate that importance sampling can further enhance convergence. This technique results in a linear dependence on the average smoothness, rather than the worst-case scenario, offering a more efficient convergence strategy across a broader range of cases.
3. Connection with Kaczmarz Method: The paper draws a novel connection between SGD and the Randomized Kaczmarz algorithm, viewing the latter as an instance of SGD. This relationship allows the transfer of insights between these methodologies, culminating in an exponential convergence proof for the Randomized Kaczmarz method, specifically towards a weighted least squares solution.
4. Partial Bias Sampling: A new family of algorithms using partially biased sampling is introduced. These algorithms maintain the convergence rate improvements from importance sampling while also ensuring higher noise tolerance. This hybrid strategy offers a practical solution when the exact variance of contributions is unknown or when samples have varying importance.

Implications and Future Directions

The results have meaningful implications for both practical applications and theoretical developments. Practically, the improved convergence rates can significantly decrease computational requirements in large-scale optimization problems common in machine learning. The emphasis on importance sampling provides a method to refine and enhance existing SGD-based methodologies by integrating new sampling strategies that exploit structural insights of the data.

Theoretical implications include the potential for extending these results to broader classes of optimization problems and further bridging connections with other iterative methods. The recasting of the Randomized Kaczmarz method as an instance of SGD may inspire new algorithms and analytical techniques linking other optimization and numerical linear algebra methods.

Future research could explore dynamic importance sampling strategies that evolve with the algorithm’s progress, offering even greater flexibility and efficiency. Moreover, the application of these insights in non-convex settings, where SGD is commonly applied in practice, could present opportunities for groundbreaking advancements.

In conclusion, this paper's findings contribute significantly to the understanding and applicability of SGD and related optimization techniques, offering both immediate practical benefits and many avenues for future theoretical exploration.