Convex-Concave Procedure

• Convex-Concave Procedure (CCP) is a method to solve difference-of-convex (DC) programming problems by decomposing objectives and constraints into convex and concave components.
• It iteratively builds convex surrogate subproblems using first-order affine approximations to ensure monotonic progress toward stationarity.
• CCP is applied in areas like combinatorial optimization, sparse recovery, and neural network training, and is implemented through frameworks like DCCP/CVXPY.

The convex-concave procedure (CCP) is an iterative majorization-minimization algorithm for solving general nonconvex optimization problems where the objective and constraint functions have the form of a sum of convex and concave terms, i.e., difference-of-convex (DC) programming. CCP provides a practical heuristic solution framework for wide classes of nonconvex problems in continuous optimization, combinatorial programming, and neural network training where convex relaxation, combinatorial continuation, or alternative gradient-based approaches are inapplicable or inefficient. The algorithm's practical impact derives from its generality, simplicity, and amenability to automatic implementation and disciplined modeling environments such as DCCP/CVXPY.

1. Mathematical Framework

CCP is formulated for optimization problems over $x \in \mathbb{R}^n$:

$$\begin{array}{ll} \underset{x \in \mathcal{D}}{\mathrm{minimize}} & F(x) = f_0^{\mathrm{cvx}}(x) + f_0^{\mathrm{ccv}}(x) \ \mathrm{subject\ to} & G_i(x) = f_i^{\mathrm{cvx}}(x) + f_i^{\mathrm{ccv}}(x) \leq 0,\quad i = 1,\dots, m, \end{array}$$

where each $f_i^{\mathrm{cvx}}$ is convex and each $f_i^{\mathrm{ccv}}$ is concave. The domain $\mathcal{D}$ is the intersection of domains where these terms are finite. In this standard DC form, the nonconvexity arises from the concave terms in both objective and constraints—these can encode combinatorial, geometrical, or algebraic nonconvexities such as sparsity, Boolean quadratic constraints, rank, or max-min operators.

2. Core Algorithmic Steps

At each iteration $k$, CCP replaces every concave component by its first-order affine upper approximation at $x^{(k)}$, yielding a convex surrogate. Let $x^{(k)}$ be the current iterate, and $g_i^{(k)} \in \partial f_i^{\mathrm{ccv}}(x^{(k)})$ denote a subgradient. For each such concave term:

$$\hat{f}_i^{\mathrm{ccv}}(x; x^{(k)}) = f_i^{\mathrm{ccv}}(x^{(k)}) + (g_i^{(k)})^\top (x - x^{(k)}),$$

which satisfies $$f_i^{\mathrm{ccv}}(x) \leq \hat{f}_i^{\mathrm{ccv}}(x; x^{(k)})$$ for all $x$.

The convex subproblem at iteration $k$ is

$$\begin{array}{ll} \underset{x \in \mathcal{D},\, s \ge 0}{\mathrm{minimize}} & f_0^{\mathrm{cvx}}(x) + \hat{f}_0^{\mathrm{ccv}}(x; x^{(k)}) + \tau_k \sum_{i=1}^m s_i \ \mathrm{subject\ to} & f_i^{\mathrm{cvx}}(x) + \hat{f}_i^{\mathrm{ccv}}(x; x^{(k)}) \leq s_i, \quad i=1,\dots,m \end{array}$$

where $s_i$ are slack variables and $\tau_k$ is a penalty on constraint violation, progressively increased. The problems at each iteration are convex and solvable by standard cone or quadratic programming techniques.

The loop is:

1. Initialize $x^{(0)} \in \operatorname{int}\mathcal{D}$, $\tau_0 > 0$, $\mu > 1$, and $\tau_{\max}$.
2. For $k = 0, 1, 2, \dots$:
   • Compute all subgradients $g_i^{(k)}$.
   • Form and solve the convex subproblem as above to obtain $x^{(k+1)}$.
   • Update $\tau_{k+1} = \min\{\mu \tau_k, \tau_{\max}\}$.
   • Apply convergence criteria such as $\|x^{(k+1)} - x^{(k)}\|$ or objective decrease.

3. Theoretical Properties and Convergence

CCP can be shown, under mild regularity conditions (bounded closed feasible level sets for all DC terms and solvability of the subproblem), to have every limit point $\bar x$ of the iterates satisfy stationarity (KKT) conditions for the original DC program:

$$0 \in \partial f_0^{\mathrm{cvx}}(\bar x) + \partial f_0^{\mathrm{ccv}}(\bar x) + \sum_{i} \lambda_i\left[ \partial f_i^{\mathrm{cvx}}(\bar x) + \partial f_i^{\mathrm{ccv}}(\bar x) \right]$$

with $\lambda_i \ge 0$, $\lambda_i G_i(\bar x) = 0$ (complementary slackness). CCP is fundamentally a sequential majorization-minimization (MM) algorithm: at each step the convex surrogate majorizes the penalized original objective, guaranteeing monotonicity and convergence to stationary points (not globally optimal solutions in general). These properties are standard in established references (Shen et al., 2016).

4. Domain Constraints and Nondifferentiability

A critical practical aspect is proper handling of concave functions with restricted domains and potential nondifferentiability at domain boundaries. CCP as implemented in DCCP (Shen et al., 2016) extends linearization to handle domain indicators: given a concave function $g$ only defined on a convex set $\mathcal{D}_g$, the affine majorant becomes

$$\hat{g}(x; z) = g(z) + \nabla g(z)^{\top}(x-z) - \mathbb{I}_{\mathcal{D}_g}(x)$$

where $\mathbb{I}_{\mathcal{D}_g}$ is $0$ in-domain and $+\infty$ otherwise (enforcing feasibility). To avoid failures in gradient computation when iterates reach the boundary, damping is applied:

$$x^{(k+1)} \gets \alpha x^{(k+1)} + (1-\alpha) x^{(k)},\quad 0 < \alpha < 1$$

This steers iterates to remain in the (relative) interior of the domain, preserving the ability to compute valid affine upper bounds.

5. Implementation: Disciplined Convex-Concave Programming (DCCP) and Ecosystem

DCCP (Shen et al., 2016) is a Python extension of CVXPY that enables disciplined modeling of DC programs for human-developed convex-concave models:

• Express objectives and constraints directly using functions classified as affine, convex, or concave, leveraging CVXPY atoms.
• Each DCCP iteration employs expression-level auto-differentiation to compute affine majorants and reconstitutes the convex subproblem for calling generic cone solvers (e.g., SCS, ECOS).
• Domain constraints are enforced via expression.domain information, ensuring feasibility.
• DCCP introduces slack variables for “relaxed” DC constraints and enforces feasibility and progress through penalty schedules.
• Damping and projection initialization routines are incorporated for robustness.

Minimal code example for nonconvex $\sqrt{x}$ minimization with $x \geq -1$:

import dccp, cvxpy as cp x = cp.Variable() obj = cp.sqrt(x) constr = [x >= -1] prob = cp.Problem(cp.Minimize(obj), constr) res = prob.solve(method='dccp') print("x* =", x.value)

Applications in DCCP range from combinatorial geometry (circle-packing with nonoverlap), Boolean least-squares, and sparse recovery ($\sum \sqrt{x_i}$) to phase retrieval and semidefinite relaxations.

6. Application Domains and Examples

CCP/DCCP enables solution of nonconvex programs in domains including, but not limited to:

• Geometric packing: Nonconvex pairwise-distance constraints for maximal packing subject to convex geometric operators.
• Combinatorial optimization: Boolean least squares, binary quadratic programming, and assignment with nonconvex combinatorial constraints.
• Sparse estimation: Recovery problems with nonconvex, possibly concave sparsity-inducing penalties.
• Neural networks: Training of morphological and dilation-erosion neural architectures that lack subdifferentiable operator structure amenable to gradient descent.
• Statistical arbitrage: Portfolio optimization for statistical arbitrages under nonconvex band constraints (Johansson et al., 12 Feb 2024).

The following table summarizes typical DC decompositions and the resulting subproblem type:

Problem Type	DC Decomposition	Convex Subproblem
Sparse recovery	$\sum \sqrt{x_i} = \sum f_i(x_i)$ (concave)	QP or SOCP
Boolean least squares	$\|Ax-y\|_2$, $x_i^2=1$ (nonconvex constraint)	QP with relaxation
Circle or sphere packing	Pairwise $\|c_i-c_j\|_2 \geq r_i+r_j$ (nonconvex)	SOCP with linearization
Morphological neural nets	Max-min compositions, pointwise max-diff (nonconvex)	Convex QP via DCCP

7. Methodological Considerations and Best Practices

Practical usage of CCP and DCCP requires careful attention to several methodological points:

• Initialization: A feasible $x^{(0)} \in \mathrm{int}(\mathcal{D})$ is essential for well-behaved convergence; random-projection heuristics are applied if unavailable.
• Penalty scheduling: Penalty $\tau_k$ is increased geometrically ($\mu \sim 2$–$10$, $\tau_{\max} \sim 10^6$) to enforce constraints.
• Damping factor $\alpha$: Typical $\alpha \sim 0.5$ precludes boundary subgradient failures.
• Multiple restarts: Given only stationarity guarantees and nonconvexity, multiple random initializations help avoid poor local optima.
• Solver selection: Choice of cone solver backend (ECOS, SCS) and warm-starting can have significant impact on iteration time.
• Convergence diagnostics: Monitor for small step changes or objective improvements for termination; in highly nonconvex or degenerate cases, empirical judgment is needed.

CCP remains a “heuristic”—no global optimality or approximation guarantees—yet provides a rigorous and general stationarity framework for DC programming, with practical tractability for a range of nonconvex applications leveraging state-of-the-art convex solvers and modeling software (Shen et al., 2016).