Alternate Convex Search (ACS)

• Alternate Convex Search (ACS) is an iterative heuristic that decomposes biconvex optimization problems by alternately solving convex subproblems, making it foundational in fields like machine learning and signal processing.
• ACS employs a block-relaxation procedure with explicit x-update and y-update phases, ensuring convergence to a stationary point under mild conditions such as Slater’s criterion.
• Enhanced through proximal regularization, slack-based initialization, and infeasible-start approaches, ACS is effectively implemented in Python via the dbcp package for disciplined biconvex programming.

Alternate Convex Search (ACS) is an iterative heuristic widely used for solving biconvex optimization problems, where the optimization variables are partitioned into two blocks and the problem is convex in each block when the other is fixed. ACS is the foundational approach for practical biconvex optimization in fields such as machine learning, signal processing, computational science, and control. Each iteration consists of alternately optimizing over the two variable blocks via convex subproblems that can be efficiently handled by off-the-shelf convex solvers. ACS forms the computational core of automated biconvex optimization frameworks such as disciplined biconvex programming (DBCP) and its open-source Python implementation, dbcp, which extends the convex optimization modeling language CVXPY (Zhu et al., 3 Nov 2025).

1. Problem Formulation in Biconvex Optimization

A biconvex optimization problem is given in variables $x \in \mathcal{X} \subseteq \mathbb{R}^n$ and $y \in \mathcal{Y} \subseteq \mathbb{R}^k$ as follows: $\begin{array}{ll} \displaystyle \min_{x, y} & f_0(x, y) \[6pt] \text{s.t.} & f_i(x, y) \leq 0,\quad i = 1, \ldots, m,\ & h_j(x, y) = 0,\quad j = 1, \ldots, p, \end{array} \tag{P}$ where each $f_i(x, y)$ is biconvex (convex in $x$ for fixed $y$ and convex in $y$ for fixed $x$) and each $h_j(x, y)$ is biaffine. The feasible set is defined as

$$\Omega = \{(x, y) \mid f_i(x, y) \leq 0,\; h_j(x, y) = 0\},$$

which itself is biconvex.

Biconvex formulations arise extensively in applications such as matrix factorization, bilinear regression, $k$-means clustering, dictionary learning, blind deconvolution, and IO-HMM fitting. For these problems, the decoupling structure permits iterative focus on one block of variables at a time.

2. The ACS Algorithm and Convex Subproblems

ACS employs a block-relaxation procedure by alternately fixing one block of variables and optimizing over the other. Let $(x^{(k)}, y^{(k)})$ denote the current iterate. Each ACS iteration consists of two principal steps:

• $x$-update:

$\begin{array}{ll} x^{(k+1)} = \arg\min_{x \in \mathcal{X}}\; f_0(x, y^{(k)}) \[4pt] \quad\text{s.t. } f_i(x, y^{(k)}) \leq 0,\; h_j(x, y^{(k)}) = 0. \end{array} \tag{%%%%11%%%%}$

• $y$-update:

$\begin{array}{ll} y^{(k+1)} = \arg\min_{y \in \mathcal{Y}}\; f_0(x^{(k+1)}, y) \[4pt] \quad\text{s.t. } f_i(x^{(k+1)}, y) \leq 0,\; h_j(x^{(k+1)}, y) = 0. \end{array} \tag{%%%%13%%%%}$

Each subproblem, $(P_x)$ and $(P_y)$, is a convex program and can be solved via standard conic or quadratic programming solvers.

Algorithmic Skeleton

The algorithmic steps are formalized in the following pseudocode (Algorithm 3.1):

Algorithm 3.1 Alternate Convex Search (ACS) Input: a feasible starting point (x^(0), y^(0)) ∈ Ω k ← 0 repeat 1. Solve x^(k+1) ← argmin_x { f0(x, y^(k)) | fi(x, y^(k)) ≤ 0, hj(x, y^(k)) = 0 } If infeasible, terminate. 2. Solve y^(k+1) ← argmin_y { f0(x^(k+1), y) | fi(x^(k+1), y) ≤ 0, hj(x^(k+1), y) = 0 } If infeasible, terminate. 3. k ← k + 1 until stopping criterion is met Output: (x^(k), y^(k))

This loop is repeated until a stopping criterion, such as a negligible objective or variable gap, is met.

3. Convergence Properties

Under mild conditions (e.g., Slater’s condition for each subproblem), ACS guarantees that:

• The generated sequence of objective values, $\{f_0(x^{(k)}, y^{(k)})\}$, is monotonically nonincreasing and converges to a finite limit.
• Any limit point $(x^*, y^*)$ of the iterates $\{(x^{(k)}, y^{(k)})\}$ is a stationary point of $f_0$ over $\Omega$ and is thus “partially optimal” for the biconvex problem.

These properties are consistent with established results in the literature, such as those of Gorski et al. (2007). Stationarity does not generally imply global optimality due to the inherent nonconvexity of biconvex problems.

4. Enhancements: Proximal Regularization and Initialization

Proximal Augmentation

To address numerical instability or promote convergence in ill-conditioned regimes, a proximal term may be introduced: $$\min_{x}\; f_0(x, y^{(k)}) + \frac{\lambda}{2} \|x - x^{(k)}\|_2^2$$ for some $\lambda > 0$, introducing strong convexity in the subproblem. The same procedure applies for the $y$-update. Empirically, proximal ACS yields more robust convergence in “flat” regions of the objective landscape.

Feasibility and Slack-Based Initialization

ACS requires an initial feasible point $(x^{(0)}, y^{(0)}) \in \Omega$. To obtain this, a slack-variable relaxation is used: $$\min_{x, y, s, t}\; \mathbf{1}^T s + \mathbf{1}^T t\ \text{s.t. } f_i(x, y) \leq s_i,\, h_j(x, y) = t_j,\, s \geq 0,$$ which itself can be solved using ACS, driving slack variables to zero and recovering feasibility.

Infeasible-Start ACS

Alternatively, infeasible-start ACS augments the original objective with a weighted slack penalty $\nu$, ensuring iterates remain feasible in the relaxed sense. For sufficiently large $\nu$, the ultimate solution tends to respect the original constraints with vanishing slacks.

5. Implementation in the dbcp Python Package

The dbcp Python package operationalizes ACS as an extension of CVXPY, supporting biconvex modeling and automated solver orchestration. The user formulates biconvex optimization problems analogously to disciplined convex programming. A representative example is:

X = cp.Variable((m,k), nonneg=True) Y = cp.Variable((k,n), nonneg=True) obj = cp.Minimize(cp.sum_squares(X@Y - A)) prob = BiconvexProblem(obj, [[X],[Y]]) prob.solve(lbd=0.1, gap_tolerance=1e-6)

Key implementation features include:

• Abstract Syntax Tree (AST) Parsing and Verification: dbcp’s AST walker enforces DBCP rules, extending curvature and monotonicity checks of DCP with a “product-rule” to verify biconvexity.
• Automatic Subproblem Construction: Given the variable partition, dbcp constructs two CVXPY Problem instances for $(P_x)$ and $(P_y)$, managing variable fixation and updates.
• Solver Dispatch: Each subproblem is dispatched to a user-specified conic solver (ECOS, SCS, OSQP, Clarabel, etc.).
• ACS Orchestration: The package maintains the ACS loop, tracks convergence metrics, and applies proximal or slack penalties via parameters such as lbd and nu.
• Initialization: Initial values may be taken from user-supplied seeds, random initialization, or the ACS-based feasibility routine, depending on feasibility detection at startup.

dbcp natively handles generalized inequalities (second-order cone, positive semidefinite constraints) by automatically selecting slack directions and adapting feasibility routines.

6. Stopping Criteria and Parameterization

Commonly adopted stopping criteria in ACS implementations include:

• Objective-gap Rule: $$|f_0(x^{(k+1)}, y^{(k+1)}) - f_0(x^{(k)}, y^{(k)})| < \varepsilon$$.
• Variable-gap Rule: $$\max\{\|x^{(k+1)} - x^{(k)}\|,\; \|y^{(k+1)} - y^{(k)}\|\} < \varepsilon$$.
• Maximum Iteration Cap: Enforces an upper limit on the number of ACS steps.

The dbcp package recommends a default tolerance of $\varepsilon = 10^{-6}$.

7. Applications, Extensions, and Observations

ACS, as formalized and automated in dbcp, efficiently solves a wide variety of biconvex problems—including matrix factorization, bilinear regression, $k$-means clustering, dictionary learning, blind deconvolution, and IO-HMM fitting—typically converging in a few dozen iterations. The inclusion of slack-based initialization and infeasible-start augmentation with penalty parameter $\nu$ allows robust handling of initially infeasible specifications, and the package’s abstraction layer enables rapid prototyping with minimal code.

Empirical findings indicate that proximal regularization is particularly effective in slow-converging or numerically “flat” objective regions, while the infeasible-start approach liberates users from feasible initialization requirements (with the caveat that sufficiently large $\nu$ may be needed for full constraint satisfaction).

Summary Table: Principal ACS Features

Aspect	Approach/Default	Implementation in dbcp
Subproblem Solvers	Any convex solver (ECOS, SCS...)	CVXPY solver interface
Initialization	Feasible point / ACS feasibility	User seed, random, or ACS init
Proximal Regularization	Optional, $\lambda > 0$	lbd argument
Stopping Criteria	Objective / variable gaps	gap_tolerance argument
Handling Infeasibility	Slack relaxation / penalty	nu argument / automatic slack

ACS is the workhorse methodology for biconvex programming, alternating block-wise convex subproblem solutions until stationarity is reached. The disciplined biconvex programming framework, as embodied in dbcp, systematizes verification, subproblem construction, solver calls, stopping checks, and initialization, enabling rapid and reliable application across a spectrum of biconvex models (Zhu et al., 3 Nov 2025).