SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradient (1703.00102v2)

Abstract: In this paper, we propose a StochAstic Recursive grAdient algoritHm (SARAH), as well as its practical variant SARAH+, as a novel approach to the finite-sum minimization problems. Different from the vanilla SGD and other modern stochastic methods such as SVRG, S2GD, SAG and SAGA, SARAH admits a simple recursive framework for updating stochastic gradient estimates; when comparing to SAG/SAGA, SARAH does not require a storage of past gradients. The linear convergence rate of SARAH is proven under strong convexity assumption. We also prove a linear convergence rate (in the strongly convex case) for an inner loop of SARAH, the property that SVRG does not possess. Numerical experiments demonstrate the efficiency of our algorithm.

• The paper introduces a new stochastic recursive gradient framework that avoids storing past gradients while enhancing computational efficiency.
• It demonstrates linear convergence under strong convexity and outperforms methods like SVRG and SAG in empirical evaluations.
• SARAH’s low-memory, recursive approach makes it a practical and scalable choice for optimizing large-scale machine learning problems.

An Overview of SARAH: A StochAstic Recursive Gradient Algorithm

The paper under review introduces the StochAstic Recursive Gradient algoritHm (SARAH), alongside its practical variant SARAH+, as a novel approach to solving finite-sum minimization problems frequently encountered in machine learning contexts. This work proposes SARAH as an alternative to prevailing stochastic optimization methods such as Stochastic Gradient Descent (SGD), Stochastic Variance Reduced Gradient (SVRG), and others like SAG and SAGA, which operate under the premise of leveraging specific properties of the optimization problem for improved convergence and computational efficiency.

Problem Formulation

The specific problem considered is the minimization of a finite sum of convex functions, where the objective function $P(w) = \frac{1}{n} \sum_{i=1}^n f_i(w)$ is smooth and exhibits a Lipschitz continuous gradient. This formulation is common in supervised learning tasks such as least squares and logistic regression.

SARAH's Approach and Theoretical Foundation

SARAH introduces a recursive framework for updating the stochastic gradient estimates, diverging from methods like SAG/SAGA that require storing past gradients. SARAH's key innovation lies in its update equations, which involve recursive computation of stochastic gradient estimates without retaining an extensive memory of past gradient information. This aspect distinguishes SARAH from gradient tracking and direct storage methods.

Convergence and Complexity Analysis

SARAH is theoretically supported by a proof that confirms its linear convergence rate under strong convexity assumptions. Notably, its convergence characteristics for the inner loop are shown to be more favorable compared to SVRG, which lacks linear convergence properties in this aspect. The complexity analysis indicates that SARAH achieves similar computational efficiency to leading variance-reducing stochastic gradient methods, yet often surpasses them in scenarios where lower memory usage is critical. Specifically, for strongly convex functions, SARAH attains a complexity of $\mathcal{O}((n + \kappa) \log(1/\epsilon))$, where $\kappa$ is the condition number, while maintaining practical merits without requiring storage of past gradients.

Numerical Experiments and Practical Implications

The empirical results validate the robustness and efficiency of SARAH, particularly in large-scale machine learning tasks. The experiments encompass standard datasets and logistic regression problems, confirming SARAH's superior or competitive performance in achieving lower loss residuals and test errors compared to other contemporary methods like SVRG and SAG. The findings suggest that SARAH can be leveraged effectively in real-world applications demanding optimization over large datasets with high-dimensional feature spaces.

Future Directions and Implications

SARAH's development opens avenues for further exploration into recursive gradient methods, especially those that can exploit sparsity or other intrinsic structures of the optimization problem to reduce computational overhead. The prospect of extending SARAH for non-convex optimization problems or integrating it within more complex machine learning pipelines remains an exciting area for research.

Overall, SARAH's contribution to the landscape of stochastic optimization is notable for its innovative approach to gradient computation, its robustness in handling large-scale problems, and its theoretical guarantees that underline its practical performance benefits. Future work may also delve into hybridizing SARAH with adaptive gradient techniques to further enhance its applicability and performance.