Online Stochastic DCA with applications to Principal Component Analysis

Abstract: Stochastic algorithms are well-known for their performance in the era of big data. In convex optimization, stochastic algorithms have been studied in depth and breadth. However, the current body of research on stochastic algorithms for nonsmooth, nonconvex optimization is relatively limited. In this paper, we propose new stochastic algorithms based on DC (Difference of Convex functions) programming and DCA (DC Algorithm) - the backbone of nonconvex, nonsmooth optimization. Since most real-world nonconvex programs fall into the framework of DC programming, our proposed methods can be employed in various situations, in which they confront stochastic nature and nonconvexity simultaneously. The convergence analysis of the proposed algorithms is studied intensively with the help of tools from modern convex analysis and martingale theory. Finally, we study several aspects of the proposed algorithms on an important problem in machine learning: the expected problem in Principal Component Analysis.