Convex Relaxation Regression (CoRR)

• CoRR is an algorithmic framework that uses convex surrogates to transform nonconvex regression and optimization problems into tractable convex ones.
• It employs methods such as convex envelope estimation, semidefinite programs, and sequential relaxation to ensure global convergence and enhance variable selection.
• Empirical evaluations demonstrate CoRR’s effectiveness in black-box optimization, sparse regression, and compressed sensing, yielding near-optimal solutions with robust statistical guarantees.

Convex Relaxation Regression (CoRR) encompasses a class of algorithmic and theoretical frameworks that systematically leverage convex relaxations to approximate or solve non-convex problems in regression and optimization. The unifying principle is to substitute an intractable or hard objective with a convex surrogate, often constructed via optimization over convex envelopes, semidefinite programs, or sequential subproblem strategies. Recent instantiations span black-box global optimization, sparse variable selection, and high-dimensional statistical modeling.

1. Foundational Principles of Convex Relaxation Regression

The core idea of CoRR is to recast the non-convex optimization of a target function $f:\mathcal{X}\to\mathbb{R}$ as a tractable convex problem by approximating $f$ with its convex envelope or a related surrogate. For global black-box function minimization, the convex envelope $f_c$ is defined as the tightest convex lower bound of $f$ over $\mathcal{X}$. Minimizing $f_c$ yields global optima coincident with those of $f$. For structured regression (e.g., sparse regression), convex relaxations transform discrete support constraints or non-convex penalties (e.g., $\ell_0$) into convex programs, making use of advanced constructs such as perspective functions, semidefinite relaxations, or block-wise convexifications.

This general methodology enables the use of convex optimization algorithms with guarantees of global convergence, superior conditioning, and improved computational tractability.

2. CoRR for Black-Box Global Optimization

The CoRR algorithm for black-box smooth function minimization, as developed by Hazan and Koren (Azar et al., 2016), approximates the convex envelope $f_c$ through empirical convex function fitting:

• Sampling phase: $2T$ i.i.d. points are drawn from a distribution $\rho$ over $\mathcal{X}$, responses $f(x)$ are evaluated, and the data partitioned into fitting and constraint batches.
• Convex surrogate fitting: A parametric family $\mathcal{H}$ of convex functions (e.g., nonnegative quadratics) is fixed. The surrogate $h(x;\theta)$ is fit by minimizing absolute error on the fitting data under an empirical mean constraint on the constraint batch. This constrained regression ensures that, when the surrogate mean matches the mean of the convex envelope, the best fit approximates $f_c$.
• Optimization of surrogate: The optimized surrogate $h(\cdot;\theta_\mu)$ is minimized using convex optimization, and a search over mean values $\mu$ selects the parameter achieving the lowest evaluation of $f$ at the surrogate minimizer.

Key theoretical results establish that, under convexity and regularity assumptions, CoRR converges to the true minimizer of $f$ at a rate $$\mathcal{O}\left(\left[\log(1/\delta)/T\right]^\alpha\right)$$ for some $\alpha>0$. The algorithm is efficient for moderate dimensions, requiring polynomial time in sample size and surrogate parameterization. This framework provides a general-purpose, data-driven approach to constructing convex relaxations for a broad class of non-convex black-box optimizations (Azar et al., 2016).

3. Sequential Convex Relaxation for Sparse Regression

Extensions of CoRR to high-dimensional sparse linear regression introduce sequential convex relaxation frameworks, such as iSCRA-TL1 (Bi et al., 2024). The process is as follows:

• Iterative truncated $\ell_1$ minimizations: The algorithm successively solves quadratic programs where the $\ell_1$ penalty is restricted (truncated) to a dynamically shrinking set of candidate coordinates, adaptively identifying portions of the true support in each iteration.
• Adaptive working sets: On each iteration, indices corresponding to large-magnitude components in the current solution are “accepted” and excluded from subsequent regularization, reducing false positives.
• Restricted null-space conditions: The theoretical analysis is based on robust restricted null space properties (rRNSP, rSRNSP), which are strictly weaker than the classical restricted isometry or eigenvalue conditions. These conditions guarantee that, after at most $r$ steps (for $r$-sparse signals), the true support is fully identified, and the estimator is $\ell_1$-close to the oracle least-squares solution.
• Complexity and accuracy: Each QP is efficiently solved via semismooth Newton AL methods, with total complexity $O(r\, \text{QP cost})$. The approach does not require strong initializations or restrictive conditions, and achieves variable selection accuracy beyond traditional Lasso for ill-conditioned or low-sample regimes (Bi et al., 2024).

4. Convexification Techniques: Perspective, SDP, and Rank-One Relaxations

Several convex relaxation techniques have been proposed to address the fundamental computational barrier of $\ell_0$-constrained problems:

• Perspective relaxations: These lift mixed-integer quadratic programs to continuous convex programs by introducing auxiliary variables and convexifying quadratic terms via the perspective function, yielding penalties such as the Minimax Concave Penalty (MCP) and reverse Huber penalty (Dong et al., 2015, Atamturk et al., 2019).
• Semidefinite relaxations: SDP-based approaches lift the problem to matrix space; probabilistic interpretations tie these relaxations to the Boolean Quadric Polytope, and connections to combinatorial optimization (e.g., Max-Cut) enable the use of randomized rounding for approximate solutions (Dong et al., 2015).
• Rank-one convexification: Recent advances derive the convex hull of mixed-integer quadratic sets for rank-one blocks, yielding non-separable, unbiased sparsity-inducing regularizers that improve upon perspective relaxations, particularly when the design matrix $X$ has non-diagonally dominant covariance. The resultant SDPs exhibit near-tightness (optimality gaps below 0.5% on both real and synthetic data) and excellent statistical properties in support and prediction accuracy, outperforming classic penalized approaches such as Elastic Net under suitable signal-to-noise regimes (Atamturk et al., 2019).

A comparative summary is given below:

Method	Convexification Type	Statistical/Computational Properties
Perspective/MCP	SOCP	Separable, tight when $X^TX$ is diagonal
SDP (Boolean/poly)	Semidefinite	Tightness depends on rank, scalable for $p \lesssim 400$
Rank-one SDP	SDP (extended+cuts)	Non-separable, unbiased, strong for arbitrary $X^TX$
Sequential CoRR	QP sequence	Adaptive, certified under weak null-space conditions

5. Theoretical Guarantees and Rates

Theoretical analysis in CoRR frameworks characterizes convergence, regularization accuracy, and computational complexity:

• Black-box CoRR: Polynomial-rate convergence to the global minimum under Hölder-type error bounds and generalized Lipschitz conditions; error $f(x) - f^* = O([ \log(1/\delta)/T ]^{\alpha})$ with high probability (Azar et al., 2016).
• Sequential sparse CoRR: Support recovery guaranteed within $r$ steps for $r$-sparse signals under robust sequential RNSP, with precise $\ell_1$ error bounds at each iteration, achieving the oracle estimator under milder conditions than classical Lasso or REC (Bi et al., 2024).
• SDP relaxations: Gaps below 0.5% for rank-one-based relaxations, uniform improvement over perspective relaxations especially in poorly-conditioned regimes; certified rank-1 solutions recover the exact sparse optimum in favorable cases (Atamturk et al., 2019, Dong et al., 2015).

6. Applications and Empirical Evaluations

Convex Relaxation Regression and its variants have been empirically validated across a spectrum of tasks:

• Black-box optimization: Recovery of global minimizers of multi-modal functions in 1–30 dimensions, outperforming pattern search, quasi-Newton, and simulated annealing in large $n$ regimes (Azar et al., 2016).
• Linear and sparse regression: Variable selection with provable support recovery under high-dimensional noise, oracle estimator attainment in $O(r)$ steps, and low $\ell_1$-errors even when classical methods fail (Bi et al., 2024).
• Compressed sensing, model selection: Significant gains in accuracy, false positive rate, and speed observed in LASSO and compressed sensing problems by relaxing via auxiliary-variable schemes or non-convex regularizers, with robustness to noise and ill-conditioning (Zheng et al., 2018, Atamturk et al., 2019).
• Benchmark datasets: On UCI data, rank-one SDP formulates near-optimal solutions with low runtime, recovering sparser supports and achieving lower statistical risk compared to Elastic Net and classical convex formulations (Atamturk et al., 2019).

7. Context, Open Problems, and Outlook

Convex Relaxation Regression unifies a spectrum of algorithmic strategies for non-convex learning via surrogates with explicit statistical and computational guarantees. Techniques span convex envelope estimation, sequential trimming of candidate supports, perspective-based convexifications, and tight SDP relaxations leveraging advanced combinatorial and geometric insights.

Current open challenges include scaling SDP-based relaxations to much larger dimensions (e.g., via low-rank or first-order methods), extending robust null-space-based sequential approaches to broader structured sparsity models, and further bridging the gap between global and local optima in settings where convex envelopes are only loosely approximated.

The versatility of CoRR frameworks suggests broad applicability, including but not limited to statistical model selection, signal recovery, machine learning model discovery, and black-box engineering optimization. Ongoing research continues to elucidate the trade-offs among statistical risk, computational tractability, and relaxation tightness in high-dimensional and nonconvex regimes (Azar et al., 2016, Bi et al., 2024, Dong et al., 2015, Atamturk et al., 2019).