Linear Time Convex Relaxation

• Linear time convex relaxation is a method that converts nonconvex problems into tractable convex surrogates, enabling high-precision solutions with O(1/ε) iteration complexity.
• It leverages gradient and mirror descent techniques, along with entropy or Bregman regularization, to maintain linear scaling per iteration relative to input size.
• Applications include robust subspace estimation, neural network verification, and optimal transport, with theoretical guarantees ensuring efficient convergence and tight relaxations.

Linear time convex relaxation refers to a class of algorithmic strategies in convex optimization, variational inference, and combinatorial optimization where the runtime to construct an ε-approximate solution scales linearly in 1/ε or, for large-scale problems, exhibits linear scaling in input size or discretization. These methods employ convex relaxations—substituting nonconvex feasible sets by convex superset surrogates—while leveraging algorithmic structures (such as gradient steps, entropy or Bregman regularization, tight relaxation design, and online game-playing frameworks) that achieve high-precision results in theoretically optimal time regimes. This paradigm underpins advances across robust subspace estimation, optimal transport, convexified learning, combinatorial tomography, and neural network verification.

1. Principles of Linear Time Convex Relaxation

The central goal of convex relaxation is to convert an original nonconvex, often NP-hard or highly-structured, optimization problem into a convex surrogate. A method is classified as "linear time" if:

• The number of main algorithmic iterations needed to reach an ε-approximate optimum is O(1/ε), or,
• The cost per iteration scales linearly with the ambient problem size, or,
• Discretized convex relaxations involve O(n) constraints or variables, where n is the grid or sample size.

Critical technical mechanisms enabling this efficiency include:

• First-order (gradient or mirror descent) iterations without expensive subroutines (e.g., quadratic solves) [0610119].
• Barrier or penalty terms leveraging strong or strict convexity in the constraints, improving convergence rates.
• Structural problem representations (e.g., conic, semidefinite, or second-order cones) that allow fast projection and update steps.

For convex programs with strictly convex constraints, as in maximum entropy estimation and portfolio risk bounds, only elementary gradient steps and projections are needed per iteration; combined with strong convexity, this yields O(1/ε) iteration complexity [0610119]. In large-scale variational or combinatorial problems, carefully designed convex relaxations often restrict the relaxed feasible set so that each constraint is verifiable or separable in linear time (Bandegi et al., 2015, Tjandraatmadja et al., 2020).

2. Online Game-Playing and Regret Minimization Connections

An influential framework connects convex relaxation algorithms with regret minimization in online game playing [0610119]:

• The convex optimization process is viewed analogously to a two-player repeated game.
• At each round, an adversarial "cost function" is chosen and the primal player selects variables (e.g., a candidate solution x).
• Gradient or mirror descent techniques with regret minimization properties update the solution, guaranteeing average regret (approximation error relative to the best fixed decision in hindsight) vanishes as O(1/ε).
• The equilibrium of these repeated plays approximates the solution to the original convex program within O(ε).

This connection justifies the linear-in-1/ε convergence of appropriate gradient-based algorithms and their avoidance of quadratic subproblems or full-blown variational inequalities, with strict convexity of constraints ensuring sufficient curvature for fast convergence.

3. Applications and Problem Structures

Linear time convex relaxation techniques are used in a diverse set of problems:

Application Domain	Representative Problem/Constraint Structure	Linear Time Mechanism
Maximum entropy frequency estimation	Entropy maximization under moment constraints (strictly convex)	Gradient-based updates; strong convexity [0610119]
Portfolio optimization under convex risk	Convex/risk constrainted quadratic programs	Only gradient/projection steps per iteration [0610119]
Nonlocal pairwise interaction minimization	LP over convex cone of autocorrelations	O(n) constraint discretization; linear LP solvers (Bandegi et al., 2015)
Large-scale robust subspace estimation (REAPER)	Convex hull of projectors; ℓ1 norm residuals	Iterated reweighted least squares; linear passes on data (Lerman et al., 2012)
Computer vision multilabel and stereo	Piecewise convex envelope lifting	Primal-dual methods; linear time per pixel (Möllenhoff et al., 2015, Laude et al., 2016)
Neural network verification (ReLU relaxations)	Multivariate ReLU graph convex hulls over box domains	Linear time separation of exponential constraints (Tjandraatmadja et al., 2020, Mao et al., 9 Oct 2024)
Optimal transport with entropy/Bregman regularization	Bregman-KL regularized LP over couplings	Entropic Sinkhorn; linear-time scaling/gradient steps (Takatsu, 2021)

In optimal transport, strictly convex regularization (KL or general Bregman divergence) both ensures uniqueness/interiority of minimizers and admits efficient iterative algorithms (Sinkhorn, projected gradient, or Riemannian descent), with each iteration operable in linear time relative to the problem size (Takatsu, 2021).

In neural network verification, the tightest convex relaxation of a single ReLU neuron over a box, while involving exponentially many inequalities, can be separated (i.e., the most violated constraint found) in O(n) time using greedy or sorting algorithms (Tjandraatmadja et al., 2020). Multi-neuron relaxations can leverage polyhedral combinatorics to compute exact output bounds for networks of bounded width in layered linear time (Mao et al., 9 Oct 2024).

4. Algorithmic and Mathematical Structures

Linear time convex relaxation algorithms leverage several common structures:

• Strong/strict convexity of constraint functions, resulting in error contraction at each iteration.
• Decomposition of the problem into iteratively solvable subproblems—each per iteration or per layer/element, such as per-pixel in vision or per-neuron in neural verification.
• Use of variable splitting, barrier, or penalty formulations to encode nonconvex constraints as convex equivalents with tight duality gaps.
• Explicit construction of convex hulls or envelopes: for example, autocorrelation sets in pairwise interaction energies (Bandegi et al., 2015), or lifted convex envelopes over discretized simplices for multilabel vision (Laude et al., 2016).

Related regularity conditions (such as log-homogeneity, affine invariance, or full-rank embedding in relaxations) are often exploited to ensure theoretical guarantees, error bounds, and reduce runtime scaling constants.

Regret bounds in online learning-inspired analysis typically take the form:

$$\operatorname{Regret}_T = \sum_{t=1}^T \langle \nabla f(x^t), x^t - x^* \rangle \leq O(\log(d) / \eta) + \eta \sum_{t=1}^T \|\nabla f(x^t)\|^2$$

Optimizing step size η leads to O(1/ε) iteration requirements for ε-accuracy.

5. Performance Guarantees and Linear Time Scaling

Theoretical efficiency is demonstrated under two axes:

1. Iteration Complexity: For strictly convex constraint sets, O(1/ε) iterations suffice for ε-approximation; each iteration only requires elementary arithmetic operations (gradient evaluation, additions, scalar multiplications, simple projections).
2. Scaling with Input Dimension: In high-dimensional or discretized instances, the number of constraints or variables in the convex relaxation is O(n), and each iteration or projection is completed in linear time in n (input/sample size, label/pixel count, or variable dimension).

This translates to total computational complexity proportional to O(1/ε * n), under mild further assumptions.

Notably, such algorithms stand in contrast to earlier approaches—such as those relying on iterative quadratic programming or full SDP solves in each step—which can require Θ(1/ε²) iterations or superlinear per-iteration work [0610119].

6. Trade-Offs and Limitations

While the linear time convex relaxation framework offers significant computational gains, certain trade-offs or structural limitations may arise:

• For general nonconvex programs without strictly convex structure or additional regularity, no comparable scaling laws are typically attainable.
• The convexification step can introduce a relaxation gap; guaranteeing tightness (small or zero optimality gap) depends on problem-specific structural properties, such as infeasibility of fractional solutions or the existence of combinatorial certificates.
• In combinatorial problems (e.g., bilinear network optimization), the relaxed polytope may require concise facet construction or separation oracles to remain tractable, but the best possible relaxation may still be of exponential description length (Khademnia et al., 2023).
• For deep neural networks, exact, linear-time convex hulls for whole-layer or whole-network structures are generally not available; strong relaxations exist for small-width or single/multi-neuron settings (Mao et al., 9 Oct 2024, Tjandraatmadja et al., 2020), but scalability beyond certain network sizes may be limited by combinatorial explosion unless further structure is leveraged.

A plausible implication is that advances in polyhedral combinatorics, efficient separation routines, or problem-specific geometric analysis may further expand the conditions under which linear time convex relaxation is practically realized.

7. Impact and Research Directions

Linear time convex relaxation strategies fundamentally alter the computational landscape for a variety of convex and nonconvex optimization problems:

• They enable robust estimation and inference in high-dimensional statistics and machine learning, as well as timely and certifiable decision-making in signal processing, control, and computational imaging.
• In neural network verification, they close longstanding gaps in robustness certification by circumventing convex barriers inherent in naive relaxation schemes (Tjandraatmadja et al., 2020, Mao et al., 9 Oct 2024).
• Theoretical underpinnings from online learning and regret minimization frameworks prompt new algorithmic architectures, blending optimization and game theory.
• Applications in convexification of network flows, optimal transport with Bregman regularization, and sublabel-accurate variational vision methods demonstrate the broad versatility and practical efficiency of this paradigm (Bandegi et al., 2015, Möllenhoff et al., 2015, Takatsu, 2021, Lerman et al., 2012, Laude et al., 2016, Ardizzoni et al., 12 Mar 2025).

The continued development of convex relaxation techniques that maintain both computational scalability and approximation tightness remains a prominent direction. Specific open questions include sharpening error bounds under weaker convexity, generalizing fast separation routines for higher-order relaxations, and unifying convex relaxation theory across diverse application domains.