Momentum-Conic Descent (MOCO) Optimization

• Momentum-Conic Descent (MOCO) is an advanced optimization method for convex conic programs enhanced by heavy-ball momentum to boost convergence in both primal and dual formulations.
• It employs a geometric ray-search strategy that alternates between ray minimization and a Frank–Wolfe-type subproblem for efficient descent over closed convex cones.
• MOCO integrates preconditioning and memory-efficient sketching techniques, making it highly effective for large-scale semidefinite programming in signal processing and machine learning.

Momentum Conic Descent (MOCO) is an advanced first-order optimization method designed for convex conic programs where the objective is minimized over a closed convex cone. MOCO generalizes the original Conic Descent (CD) algorithm by incorporating a heavy-ball momentum term, yielding enhanced convergence rates and efficiency in both primal and dual formulations. This algorithm is particularly relevant for large-scale semidefinite programming (SDP) problems in signal processing and machine learning, and introduces innovations in stopping criteria, preconditioning, and memory-efficient computation for low-rank solutions (Li et al., 2023).

1. Primal and Dual Formulation of Conic Programs

Consider the convex conic program:

$$\min_{x} \; f(x) \quad \text{subject to} \;\; x \in \mathcal{K},$$

where $\mathcal{K} \subseteq \mathbb{R}^d$ is a closed convex cone, and $f:\mathcal{K} \rightarrow \mathbb{R}$ is convex and differentiable. The equivalent unconstrained formulation leverages the indicator function:

$$\min_{x} \; F(x) := f(x) + \mathbb{I}_{\mathcal{K}}(x).$$

The Fenchel dual is expressed as:

$$L(x, y) = f(x) + \mathbb{I}_{\mathcal{K}}(x) - \langle y, x \rangle, \quad y \in \mathbb{R}^d,$$

resulting in the dual problem:

$\sup_{y \in \mathcal{K}^*} \; [-f^*(y)],$

where $f^*$ is the convex conjugate of $f$, and $\mathcal{K}^*$ is the dual cone. Strong duality holds under mild regularity conditions such as Slater's condition.

2. Geometric Ray-Search Intuition and Algorithmic Structure

Every $x \in \mathcal{K}$ admits the representation $\mathcal{K} \subseteq \mathbb{R}^d$0, with $\mathcal{K} \subseteq \mathbb{R}^d$1, $\mathcal{K} \subseteq \mathbb{R}^d$2, and scalar $\mathcal{K} \subseteq \mathbb{R}^d$3. The algorithm first solves a univariate problem along each ray:

$\mathcal{K} \subseteq \mathbb{R}^d$4

Finding the optimal $\mathcal{K} \subseteq \mathbb{R}^d$5 reduces to a compact search over directions on the cone. Conic Descent alternates between ray minimization and a Frank–Wolfe-type subproblem for ray search:

• Ray minimization: $\mathcal{K} \subseteq \mathbb{R}^d$6.
• Ray search: $\mathcal{K} \subseteq \mathbb{R}^d$7 with $\mathcal{K} \subseteq \mathbb{R}^d$8 as the descent direction.

MOCO extends this by incorporating a momentum term via heavy-ball averaging for $\mathcal{K} \subseteq \mathbb{R}^d$9, enhancing descent speed.

3. Momentum-Conic Descent (MOCO) Algorithm

MOCO iteratively updates both the search direction and scaling using momentum-augmented gradients. The principal steps per iteration $f:\mathcal{K} \rightarrow \mathbb{R}$0 are:

1. Ray Minimization: $f:\mathcal{K} \rightarrow \mathbb{R}$1.
2. Momentum Update: $f:\mathcal{K} \rightarrow \mathbb{R}$2, with $f:\mathcal{K} \rightarrow \mathbb{R}$3.
3. Frank–Wolfe Subproblem (Ray Search): $f:\mathcal{K} \rightarrow \mathbb{R}$4.
4. Step-Size Line Search: $f:\mathcal{K} \rightarrow \mathbb{R}$5.
5. Primal Update: $f:\mathcal{K} \rightarrow \mathbb{R}$6.

At termination, the solution is given by $f:\mathcal{K} \rightarrow \mathbb{R}$7.

Key MOCO equations:

• Conic dual: $f:\mathcal{K} \rightarrow \mathbb{R}$8
• Descent direction: $f:\mathcal{K} \rightarrow \mathbb{R}$9
• Heavy-ball momentum: $$\min_{x} \; F(x) := f(x) + \mathbb{I}_{\mathcal{K}}(x).$$0
• Primal update: $$\min_{x} \; F(x) := f(x) + \mathbb{I}_{\mathcal{K}}(x).$$1

4. Convergence Rates and Proof Sketches

Convergence analysis for MOCO under strict convexity and Lipschitz gradient conditions shows:

• Primal Rate:

$$\min_{x} \; F(x) := f(x) + \mathbb{I}_{\mathcal{K}}(x).$$2

where $$\min_{x} \; F(x) := f(x) + \mathbb{I}_{\mathcal{K}}(x).$$3 quantifies additional reduction from momentum.

• Dual Rate:

$$\min_{x} \; F(x) := f(x) + \mathbb{I}_{\mathcal{K}}(x).$$4

Thus, an $$\min_{x} \; F(x) := f(x) + \mathbb{I}_{\mathcal{K}}(x).$$5-approximate KKT point is obtained in $$\min_{x} \; F(x) := f(x) + \mathbb{I}_{\mathcal{K}}(x).$$6 iterations.

The proof leverages Bregman-type lower bounds built from linearizations of $$\min_{x} \; F(x) := f(x) + \mathbb{I}_{\mathcal{K}}(x).$$7, and invokes a generalization of Nesterov’s lemma to relate primal and dual gaps.

5. Stopping Criterion and Preconditioning Techniques

Direct computation of the dual residual requires a projection onto $$\min_{x} \; F(x) := f(x) + \mathbb{I}_{\mathcal{K}}(x).$$8, often computationally expensive. Instead, MOCO uses the subproblem multiplier:

$$\min_{x} \; F(x) := f(x) + \mathbb{I}_{\mathcal{K}}(x).$$9

with guaranteed rate:

$$L(x, y) = f(x) + \mathbb{I}_{\mathcal{K}}(x) - \langle y, x \rangle, \quad y \in \mathbb{R}^d,$$0

Termination is certified when $$L(x, y) = f(x) + \mathbb{I}_{\mathcal{K}}(x) - \langle y, x \rangle, \quad y \in \mathbb{R}^d,$$1, yielding $$L(x, y) = f(x) + \mathbb{I}_{\mathcal{K}}(x) - \langle y, x \rangle, \quad y \in \mathbb{R}^d,$$2.

Preconditioning by linear change-of-variables $$L(x, y) = f(x) + \mathbb{I}_{\mathcal{K}}(x) - \langle y, x \rangle, \quad y \in \mathbb{R}^d,$$3 can sharply reduce the dual error constant. An appropriately chosen positive-definite $$L(x, y) = f(x) + \mathbb{I}_{\mathcal{K}}(x) - \langle y, x \rangle, \quad y \in \mathbb{R}^d,$$4 that balances the Hessian and cone geometry minimizes $$L(x, y) = f(x) + \mathbb{I}_{\mathcal{K}}(x) - \langle y, x \rangle, \quad y \in \mathbb{R}^d,$$5, thereby accelerating convergence.

6. Memory-Efficient MOCO for SDP with Low-Rank Structure

MOCO adapts for large-scale semidefinite programs (SDP):

$$L(x, y) = f(x) + \mathbb{I}_{\mathcal{K}}(x) - \langle y, x \rangle, \quad y \in \mathbb{R}^d,$$6

with $$L(x, y) = f(x) + \mathbb{I}_{\mathcal{K}}(x) - \langle y, x \rangle, \quad y \in \mathbb{R}^d,$$7 linear and $$L(x, y) = f(x) + \mathbb{I}_{\mathcal{K}}(x) - \langle y, x \rangle, \quad y \in \mathbb{R}^d,$$8 with $$L(x, y) = f(x) + \mathbb{I}_{\mathcal{K}}(x) - \langle y, x \rangle, \quad y \in \mathbb{R}^d,$$9-Lipschitz gradient. To circumvent storing $\sup_{y \in \mathcal{K}^*} \; [-f^*(y)],$0, MOCO maintains:

• $\sup_{y \in \mathcal{K}^*} \; [-f^*(y)],$1
• A random sketch $\sup_{y \in \mathcal{K}^*} \; [-f^*(y)],$2, using fixed Gaussian $\sup_{y \in \mathcal{K}^*} \; [-f^*(y)],$3 with $\sup_{y \in \mathcal{K}^*} \; [-f^*(y)],$4

The affine update for the sketch:

$\sup_{y \in \mathcal{K}^*} \; [-f^*(y)],$5

where $\sup_{y \in \mathcal{K}^*} \; [-f^*(y)],$6 is the minimal-eigenvector of $\sup_{y \in \mathcal{K}^*} \; [-f^*(y)],$7. Each Frank–Wolfe iteration costs $\sup_{y \in \mathcal{K}^*} \; [-f^*(y)],$8 via Lanczos, with total memory $\sup_{y \in \mathcal{K}^*} \; [-f^*(y)],$9. Recovery of an $f^*$0-accurate $f^*$1 from the sketch is controlled by the true rank $f^*$2 and the excess singular values, provided $f^*$3.

7. Empirical Performance and Practical Guidelines

Numerical experiments demonstrate MOCO's effectiveness on raised-up SDP problems such as matrix completion and phase-retrieval:

• For matrix completion ($f^*$4 recovery from noisy, partial entries), MOCO and CD have comparable runtime–primal error profiles, but greedy-accelerated MOCOg outperforms all methods at large $f^*$5.
• For phase-retrieval (rank-1 SDP lifted from quadratic measurements), MOCOg and a heuristic step-size variant (MOCOh) match or surpass CDg in visual quality and runtime-loss performance, significantly outperforming standard Frank–Wolfe approaches.

Noteworthy practical observations include:

• Momentum-augmented Frank–Wolfe within the conic framework (MOCO) yields tighter convergence by the positive momentum term $f^*$6.
• The stopping criterion $f^*$7 is efficiently computed and directly certifies dual feasibility.
• Preconditioners significantly reduce dual residual constants, allowing earlier termination.
• The memory-efficient variant using sketching is effective for large-scale SDP with rigorous low-rank recovery.
• Greedy acceleration (Burer–Monteiro step) and heuristic step-size selection (e.g., $f^*$8 where $f^*$9) expedite convergence without substantial additional memory cost (Li et al., 2023).