Dropping Convexity for Faster Semi-definite Optimization (1509.03917v3)

Abstract: We study the minimization of a convex function $f(X)$ over the set of $n\times n$ positive semi-definite matrices, but when the problem is recast as $\min_U g(U) := f(UU^\top)$, with $U \in \mathbb{R}^{n \times r}$ and $r \leq n$. We study the performance of gradient descent on $g$---which we refer to as Factored Gradient Descent (FGD)---under standard assumptions on the original function $f$. We provide a rule for selecting the step size and, with this choice, show that the local convergence rate of FGD mirrors that of standard gradient descent on the original $f$: i.e., after $k$ steps, the error is $O(1/k)$ for smooth $f$, and exponentially small in $k$ when $f$ is (restricted) strongly convex. In addition, we provide a procedure to initialize FGD for (restricted) strongly convex objectives and when one only has access to $f$ via a first-order oracle; for several problem instances, such proper initialization leads to global convergence guarantees. FGD and similar procedures are widely used in practice for problems that can be posed as matrix factorization. To the best of our knowledge, this is the first paper to provide precise convergence rate guarantees for general convex functions under standard convex assumptions.

• The paper proposes converting convex semi-definite optimization problems into non-convex unconstrained problems using matrix factorization (X = UU").
• The authors introduce Factored Gradient Descent (Fgd), a tailored method for the non-convex problem that achieves convergence rates comparable to classic gradient descent.
• The new approach yields significant computational savings for large-scale semi-definite optimization problems by eliminating costly steps like eigenvalue decompositions.

Dropping Convexity for Faster Semi-definite Optimization

In the article "Dropping Convexity for Faster Semi-definite Optimization," the authors investigate an innovative recasting of convex optimization problems by leveraging matrix factorizations. The paper primarily concerns optimization problems in which the objective function, $f$, is convex and differentiable, and the search space is restricted to positive semi-definite matrices. Standard solutions to these problems are computationally intensive due to the requirement of maintaining positive semi-definiteness. This paper presents a promising approach to reformulate such problems for improved efficiency.

Overview of the Approach

The paper proposes converting the convex semi-definite optimization problem into a non-convex unconstrained optimization problem through matrix factorization. By expressing semi-definite matrices as products of a smaller matrix $U$ with its transpose, the optimization over these matrices is transformed into an unconstrained optimization problem over $U$. This approach eliminates the need for costly eigenvalue decompositions during iterative optimization, which is required to keep the iterates within the cone of positive semi-definite matrices.

Factored Gradient Descent (Fgd)

The authors introduce Factored Gradient Descent (Fgd), a first-order optimization method specifically tailored for the recast problem. Fgd applies gradient descent directly to the factorized space, leveraging a novel step size rule $\eta$ designed to achieve rapid convergence. The step size is a function not only of the smoothness constant $M$ but also of the spectral properties of the current estimate and its gradient, ensuring stability and convergence even under non-convexity.

Theoretical Results

The authors demonstrate that upon proper initialization, Fgd converges efficiently to solutions close to the optimum. They establish proofs showing that the convergence behavior of Fgd matches that of classic gradient descent when the original function is smooth ($O(1/k)$ convergence) or restricted strongly convex (exponential convergence). The theoretical guarantees are anchored in standard convex notions but adapted to a transformed space, thereby exemplifying the potential of matrix factorization in generalizing convergence results to non-convex settings.

Robust Initialization

The initialization plays a crucial role in the performance of Fgd, notably as it avoids adverse effects from saddle points that may arise in the non-convex space. The paper describes how standard techniques or first-order oracle models can be employed to ascertain a starting point proximal to the intended optimum, ensuring the described convergence rates are attainable.

Implications and Future Directions

This formulation holds significant implications for technologies reliant on semi-definite optimization, such as machine learning tasks involving matrix completion or covariance selection. Practically, this recasting could lead to major computational savings in large-scale problems. However, further exploration is needed to extend these strategies to stochastic or constrained settings, which may further broaden the scope of the proposed methods.

The authors suggest potential paths for future research, including adapting acceleration methods known from convex optimization to the non-convex factored gradient descent framework and exploring proximal methods that can handle non-smooth constraints in the factored space. Additionally, further refinement in initialization schemes and direct handling of higher-dimensional factor spaces are promising avenues.

Conclusion

In summary, the paper "Dropping Convexity for Faster Semi-definite Optimization" presents a compelling approach to convert complex convex optimization into efficient non-convex formulations by matrix factorization. While maintaining rigorous convergence guarantees, the authors demonstrate substantial computational benefits that can extend to sophisticated applications in optimization and machine learning. As the findings challenge existing paradigms on optimization boundaries, they open a field of possibilities for enhanced optimization techniques and inspire future developments in the field.