Douglas-Rachford splitting for nonconvex optimization with application to nonconvex feasibility problems (1409.8444v5)

Abstract: We adapt the Douglas-Rachford (DR) splitting method to solve nonconvex feasibility problems by studying this method for a class of nonconvex optimization problem. While the convergence properties of the method for convex problems have been well studied, far less is known in the nonconvex setting. In this paper, for the direct adaptation of the method to minimize the sum of a proper closed function $g$ and a smooth function $f$ with a Lipschitz continuous gradient, we show that if the step-size parameter is smaller than a computable threshold and the sequence generated has a cluster point, then it gives a stationary point of the optimization problem. Convergence of the whole sequence and a local convergence rate are also established under the additional assumption that $f$ and $g$ are semi-algebraic. We also give simple sufficient conditions guaranteeing the boundedness of the sequence generated. We then apply our nonconvex DR splitting method to finding a point in the intersection of a closed convex set $C$ and a general closed set $D$ by minimizing the squared distance to $C$ subject to $D$. We show that if either set is bounded and the step-size parameter is smaller than a computable threshold, then the sequence generated from the DR splitting method is actually bounded. Consequently, the sequence generated will have cluster points that are stationary for an optimization problem, and the whole sequence is convergent under an additional assumption that $C$ and $D$ are semi-algebraic. We achieve these results based on a new merit function constructed particularly for the DR splitting method. Our preliminary numerical results indicate that our DR splitting method usually outperforms the alternating projection method in finding a sparse solution of a linear system, in terms of both the solution quality and the number of iterations taken.

• The paper demonstrates that the nonconvex DR splitting method can yield stationary points when the step-size is below a specific threshold and a cluster point exists.
• It introduces a novel merit function to prove global convergence and local convergence rates under semi-algebraic assumptions.
• The method outperforms alternating projection in sparse solution finding, making it promising for applications in signal processing and machine learning.

An Expert Overview of Nonconvex Douglas-Rachford Splitting

The paper "Douglas-Rachford splitting for nonconvex optimization with application to nonconvex feasibility problems" by Guoyin Li and Ting Kei Pong presents an adaptation of the Douglas-Rachford (DR) splitting method for nonconvex optimization problems. While the DR splitting method has been effectively utilized for convex optimization, its application to nonconvex scenarios has not been thoroughly investigated. This paper provides new insights into the behavior of DR splitting when applied to nonconvex problems, particularly focusing on feasibility problems.

Problem Scope

The authors address nonconvex optimization problems characterized by the minimization of a function comprising the sum of a proper closed function $g$ and a smooth function $f$ with a Lipschitz continuous gradient. They highlight that the DR splitting method can yield a stationary point for such problems given that the step-size parameter is below a certain threshold and the sequence generated has at least one cluster point. This extends the DR methodology beyond the scope of convex problems, providing convergence insights and local convergence rates under specific algebraic function conditions.

Mathematical Contributions

1. Convergence Results: The paper proves global convergence and a local convergence rate using a novel merit function introduced specifically for the DR splitting method in the nonconvex context. The sequence convergence is established under additional conditions that require $f$ and $g$ to be semi-algebraic, providing new theoretical guarantees for nonconvex optimization.

2. Boundedness Criteria: The authors present simple sufficient conditions ensuring sequence boundedness, which are fundamental for establishing convergence. They demonstrate that the sequence remains constrained if either the convex set $C$ or the general closed set $D$ is bounded and the step-size adheres to the threshold requirement.

3. Practical Application: The DR splitting method is applied to a feasibility problem involving minimizing the squared distance to a convex set $C$ subject to another closed set $D$. The method shows promising results in outperforming the alternating projection method in sparse solution finding for linear systems, both in iteration count and solution quality.

Results and Comparisons

Preliminary numerical experiments indicate that the adapted DR splitting method exceeds the performance of the alternating projection method in computed solution quality and iteration efficiency. Furthermore, the DR approach also demonstrates superior behavior compared to the classical DR splitting when minimizing the sum of indicator functions of the sets $C$ and $D$, especially in nonconvex settings.

Implications and Future Work

The paper provides a pivotal extension of the DR splitting method to nonconvex problems, paving the way for its application in more complex optimization scenarios including signal processing and machine learning tasks that naturally feature nonconvex structures. Future research could explore enhancements to the threshold determination for step-size parameters to further refine the algorithm's practical applicability and robustness.

The paper offers substantial potential for the development of improved algorithms tackling nonconvex optimization challenges. It suggests that nonconvex DR splitting can be beneficial in other areas of research, such as manifold optimization or computational mathematical problems involving non-smooth structures. As semi-algebraic functions are commonly found in real-world applications, expanding the understanding of DR splitting's behavior in these contexts is a promising research direction.