Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization (1506.08272v5)

Abstract: Asynchronous parallel implementations of stochastic gradient (SG) have been broadly used in solving deep neural network and received many successes in practice recently. However, existing theories cannot explain their convergence and speedup properties, mainly due to the nonconvexity of most deep learning formulations and the asynchronous parallel mechanism. To fill the gaps in theory and provide theoretical supports, this paper studies two asynchronous parallel implementations of SG: one is on the computer network and the other is on the shared memory system. We establish an ergodic convergence rate $O(1/\sqrt{K})$ for both algorithms and prove that the linear speedup is achievable if the number of workers is bounded by $\sqrt{K}$ ($K$ is the total number of iterations). Our results generalize and improve existing analysis for convex minimization.

Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization

The paper "Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization," authored by Xiangru Lian, Yijun Huang, Yuncheng Li, and Ji Liu, presents a rigorous analysis of asynchronous parallel implementations of the Stochastic Gradient (SG) algorithm in the context of nonconvex optimization. This work addresses the theoretical gaps that have persisted regarding the convergence and speedup properties of these implementations, particularly in deep learning scenarios where nonconvex formulations dominate.

Key Contributions and Methodology

The authors focus on two asynchronous parallel implementations of SG: one executed over a computer network (AsySG-con) and the other deployed on shared memory systems (AsySG-incon). The investigation into these implementations is driven by:

• Ergodic Convergence Rates: The paper establishes an asymptotic convergence rate of $O(1/\sqrt{KM})$ for both AsySG-con and AsySG-incon, where $K$ is the total number of iterations and $M$ is the size of the minibatch.
• Linear Speedup Conditions: It is demonstrated that both algorithms can achieve linear speedup, provided the number of workers is bounded by $O(\sqrt{K})$. This extends and improves upon existing theories mainly applicable to convex problems.

AsySG-con: Network-Based Implementation

AsySG-con leverages the network's architecture to ensure atomicity during parameter updates, thus supporting consistent parameter states. The analysis introduces assumptions such as independence of random variables and bounded delay, which are essential for validating the theoretical convergence results.

• Convergence Bound Improved: The analysis refines the upper bound on the number of workers from previous studies, improving it by a factor of $K^{1/4}M^{-1/4}$.

AsySG-incon: Shared Memory Implementation

In contrast, AsySG-incon addresses the computational reality of shared memory systems, where consistent read operations are challenging due to lock-free environments. The authors provide a more precise description of this approach compared to the existing Hogwild! algorithm.

• Broad Applicability: Although the results do not strictly dominate the Hogwild! findings due to differences in problem settings, they offer applicability to a broader range of scenarios, including nonconvex optimization, providing theoretical guarantees on convergence and speedup.

Empirical Validation

The authors validate their theoretical findings through empirical studies on both computer clusters and multicore systems with synthetic and standard datasets (e.g., LENET, CIFAR10-FULL). They distinguish between iteration speedup and running time speedup, showcasing consistent linear scalability across varying numbers of workers.

Implications and Future Directions

This paper's contributions are significant for both theoretical exploration and practical implementations:

• Theoretical Insight: It lays a robust foundation for the understanding and further development of asynchronous methods in nonconvex optimization, particularly in deep learning.
• Practical Application: The linear speedup results can inform the design and deployment of distributed systems aimed at optimizing nonconvex functions, ensuring computational efficiency and scalability.
• Future Work: The research opens avenues for further exploration into diverse asynchronous optimization techniques that could be applied to even more complex and heterogeneous environments.

The paper exemplifies a comprehensive approach to bridging theoretical insights with real-world applicability, providing a substantial step forward in the optimization of nonconvex functions in asynchronous settings.