Orthant-Based Proximal SG (OBProx-SG)

• The paper presents OBProx-SG, a method that alternates proximal stochastic gradient and orthant projection steps to aggressively promote sparsity in l1-regularized optimization.
• It establishes global convergence under nonconvexity and linear convergence in strongly convex settings through a carefully designed modulo switching schedule.
• Empirical evaluations demonstrate that OBProx-SG significantly reduces model density while maintaining predictive accuracy, outperforming traditional methods.

Orthant-Based Proximal Stochastic Gradient methods (OBProx-SG) address the efficient solution of $\ell_1$-regularized optimization problems, which arise in domains such as feature selection and model compression. The canonical form is $\min_x F(x) = f(x) + \lambda\|x\|_1$, where $f(x) = \frac{1}{N} \sum_{i=1}^N f_i(x)$ is an average over smooth—possibly nonconvex—component losses, and the $\ell_1$-regularization promotes sparsity. OBProx-SG unifies the global convergence properties of proximal stochastic gradient approaches with aggressive sparsity promotion via orthant projection, yielding solutions with substantially reduced support while maintaining per-iteration cost comparable to standard stochastic proximal methods (Chen et al., 2020).

1. Problem Setting and Motivation

The primary objective is to solve

$$\min_x F(x) = f(x) + \lambda\|x\|_1, \quad f(x) = \frac{1}{N}\sum_{i=1}^N f_i(x), \quad \lambda > 0,$$

where $f_i : \mathbb{R}^n \to \mathbb{R}$ are smooth and $N, n \gg 1$. In the convex scenario, each $f_i$ is convex and $L$-smooth. If $f$ is nonconvex, it is assumed to be continuously differentiable with Lipschitz gradients within a compact level set, and stochastic gradients possess bounded variance. The $\min_x F(x) = f(x) + \lambda\|x\|_1$0-term induces sparsity by shrinking coefficients toward zero, which is crucial in high-dimensional settings for interpretability and computational efficiency.

2. OBProx-SG Algorithmic Structure

OBProx-SG alternates between two subroutines—a Proximal Stochastic Gradient (Prox-SG) Step and an Orthant Step—using a simple modulo switching schedule determined by user-specified integers $\min_x F(x) = f(x) + \lambda\|x\|_1$1 and $\min_x F(x) = f(x) + \lambda\|x\|_1$2. The control flow is as follows:

• Input: Initial point $\min_x F(x) = f(x) + \lambda\|x\|_1$3, step size $\min_x F(x) = f(x) + \lambda\|x\|_1$4, $\min_x F(x) = f(x) + \lambda\|x\|_1$5, $\min_x F(x) = f(x) + \lambda\|x\|_1$6.
• For $\min_x F(x) = f(x) + \lambda\|x\|_1$7:
  • If $\min_x F(x) = f(x) + \lambda\|x\|_1$8, perform a Prox-SG Step;
  • Else, perform an Orthant Step;
  • Update step size $\min_x F(x) = f(x) + \lambda\|x\|_1$9.

2.1 Prox-SG Step

Given $f(x) = \frac{1}{N} \sum_{i=1}^N f_i(x)$0 and step size $f(x) = \frac{1}{N} \sum_{i=1}^N f_i(x)$1, sample a mini-batch $f(x) = \frac{1}{N} \sum_{i=1}^N f_i(x)$2 and compute the stochastic gradient $f(x) = \frac{1}{N} \sum_{i=1}^N f_i(x)$3. The update is

$f(x) = \frac{1}{N} \sum_{i=1}^N f_i(x)$4

This amounts to a coordinate-wise soft-thresholding operation promoting sparsity, with

$f(x) = \frac{1}{N} \sum_{i=1}^N f_i(x)$5

2.2 Orthant Step

Define sets $f(x) = \frac{1}{N} \sum_{i=1}^N f_i(x)$6, $f(x) = \frac{1}{N} \sum_{i=1}^N f_i(x)$7, $f(x) = \frac{1}{N} \sum_{i=1}^N f_i(x)$8, with $f(x) = \frac{1}{N} \sum_{i=1}^N f_i(x)$9 restricted to the spanning orthant face

$\ell_1$0

where coordinates in $\ell_1$1 are fixed to zero and those in $\ell_1$2 keep their sign. On $\ell_1$3, the objective simplifies to a smooth function $\ell_1$4.

The step comprises:

• Stochastic gradient computation for $\ell_1$5:

$\ell_1$6

• Gradient descent update: $\ell_1$7.
• Orthant projection: for each coordinate,

$\ell_1$8

The switching schedule is governed by $\ell_1$9, $$\min_x F(x) = f(x) + \lambda\|x\|_1, \quad f(x) = \frac{1}{N}\sum_{i=1}^N f_i(x), \quad \lambda > 0,$$0; OBProx-SG$$\min_x F(x) = f(x) + \lambda\|x\|_1, \quad f(x) = \frac{1}{N}\sum_{i=1}^N f_i(x), \quad \lambda > 0,$$1 is a variant with $$\min_x F(x) = f(x) + \lambda\|x\|_1, \quad f(x) = \frac{1}{N}\sum_{i=1}^N f_i(x), \quad \lambda > 0,$$2, $$\min_x F(x) = f(x) + \lambda\|x\|_1, \quad f(x) = \frac{1}{N}\sum_{i=1}^N f_i(x), \quad \lambda > 0,$$3, i.e., finitely many Prox-SG Steps followed by only Orthant Steps.

3. Convergence Guarantees

Analysis proceeds under the condition that $$\min_x F(x) = f(x) + \lambda\|x\|_1, \quad f(x) = \frac{1}{N}\sum_{i=1}^N f_i(x), \quad \lambda > 0,$$4 has $$\min_x F(x) = f(x) + \lambda\|x\|_1, \quad f(x) = \frac{1}{N}\sum_{i=1}^N f_i(x), \quad \lambda > 0,$$5-Lipschitz stochastic gradients, is bounded below, and that $$\min_x F(x) = f(x) + \lambda\|x\|_1, \quad f(x) = \frac{1}{N}\sum_{i=1}^N f_i(x), \quad \lambda > 0,$$6 is uniformly bounded and unbiased with variance $$\min_x F(x) = f(x) + \lambda\|x\|_1, \quad f(x) = \frac{1}{N}\sum_{i=1}^N f_i(x), \quad \lambda > 0,$$7.

Key Theoretical Results:

• Global Convergence: If $$\min_x F(x) = f(x) + \lambda\|x\|_1, \quad f(x) = \frac{1}{N}\sum_{i=1}^N f_i(x), \quad \lambda > 0,$$8, then $$\min_x F(x) = f(x) + \lambda\|x\|_1, \quad f(x) = \frac{1}{N}\sum_{i=1}^N f_i(x), \quad \lambda > 0,$$9 in general (possibly nonconvex) cases, where $f_i : \mathbb{R}^n \to \mathbb{R}$0 is the proximal-gradient mapping.
• Linear Rate for Strong Convexity: If $f_i : \mathbb{R}^n \to \mathbb{R}$1 is $f_i : \mathbb{R}^n \to \mathbb{R}$2-strongly convex and $f_i : \mathbb{R}^n \to \mathbb{R}$3, the following holds:

$f_i : \mathbb{R}^n \to \mathbb{R}$4

where $f_i : \mathbb{R}^n \to \mathbb{R}$5 counts the cumulative Prox-SG Steps and $f_i : \mathbb{R}^n \to \mathbb{R}$6 is determined by the starting level set.

• Support Identification and OBProx-SG$f_i : \mathbb{R}^n \to \mathbb{R}$7: When $f_i : \mathbb{R}^n \to \mathbb{R}$8, under mild local convexity and proper initialization, Orthant Steps alone can drive the proximal mapping norm to zero once the correct orthant is identified.
• High-Probability Support Identification: In the $f_i : \mathbb{R}^n \to \mathbb{R}$9-strongly convex case, after $N, n \gg 1$0 Prox-SG steps, the iterate is in a neighborhood of the correct support with high probability.

4. Comparative Analysis with Related Methods

OBProx-SG is contrasted with prevalent stochastic schemes for $N, n \gg 1$1-regularized problems:

Method	Sparsity Promotion	Convergence Rate/Cost
Prox-SG	Shrinkage region $N, n \gg 1$2 (moderate)	$N, n \gg 1$3 or linear for strong convexity; slow
RDA	Averaged gradient enlarges truncation region (aggressive)	More aggressive sparsity, but slower convergence
Prox-SVRG	Moderate (like Prox-SG)	Linear in convex case; needs full-gradient per epoch (costly)
OBProx-SG	Orthant face projection, much larger “zero region” (aggressive)	Linear in Prox-SG steps; one mini-batch per iteration

The Orthant Step in OBProx-SG features a substantially larger zero region for each positive coordinate, leading to aggressive fabrication of sparsity at a low computational burden per iteration (Chen et al., 2020).

5. Empirical Evaluation

Empirical studies evaluate OBProx-SG and OBProx-SG$N, n \gg 1$4 on convex and nonconvex $N, n \gg 1$5-regularized problems:

5.1 Convex Case

• Datasets: a9a, higgs, kdda, news20, real-sim, rcv1, url_combined, w8a; $N, n \gg 1$6.
• OBProx-SG and OBProx-SG$N, n \gg 1$7 achieve objective values on par with Prox-SG/Prox-SVRG and outperform RDA.
• Solution density (fraction of nonzeros): Prox-SG $N, n \gg 1$8–$N, n \gg 1$9, RDA $f_i$0–$f_i$1, Prox-SVRG $f_i$2–$f_i$3, OBProx-SG $f_i$4–$f_i$5, OBProx-SG$f_i$6 up to $f_i$7 lower than OBProx-SG.
• Runtime: OBProx-SG, Prox-SG, and RDA are comparable; Prox-SVRG incurs $f_i$8–$f_i$9 higher computational cost.

5.2 Nonconvex Case

• Tasks: $L$0-regularized MobileNetV1 and ResNet18 on CIFAR-10 and Fashion-MNIST.
• OBProx-SG and Prox-SG/Prox-SVRG obtain statistically equivalent objective values and test accuracy (within $L$1–$L$2), with RDA showing degraded performance.
• Density: Prox-SG $L$3–$L$4 nonzeros, Prox-SVRG/RDA $L$5, OBProx-SG $L$6–$L$7, OBProx-SG$L$8 $L$9–$f$0—leading to up to $f$1 sparser networks without loss in accuracy.
• Sparsity evolution reveals a rapid density decrease post transition to Orthant Steps.

6. Implementation Considerations and Theoretical Rationale

• Prox-SG analysis ensures expected decrease in $f$2, yielding convergence to vanishing mapping norm or to a neighborhood under strong convexity.
• Once $f$3 covers the true solution support, the problem is smooth using the Orthant Step, which converges (with decaying step sizes) to the global or stationary point with the same support.
• High probability of correct orthant identification is established via concentration inequalities under strong convexity.
• Efficient implementation is facilitated: track only the current sign vector $f$4 and nonzero index set $f$5. In the Orthant Step, only the relevant gradient needs computation; coordinates outside the orthant are projected to zero.
• The modulo switching schedule is lightweight, and both subroutines leverage the same mini-batch sampling, gradient, and projection logic.

7. Synthesis and Impact

OBProx-SG demonstrates unified globalization of stochastic proximal gradient methods with aggressive support identification via orthant projection. This design enables provable convergence to global optima under convexity (or stationary points for nonconvex $f$6), linear rates under strong convexity, and empirical reduction in iterate density, all with low per-iteration complexity. In both convex and deep learning applications (e.g., MobileNetV1, ResNet18), OBProx-SG achieves markedly improved sparsity over established methods without degrading objective value or predictive performance (Chen et al., 2020).