Stochastic Split LBI (S²-LBI)

• S²-LBI is an algorithm that integrates stochastic gradients with Linearized Bregman Iterations to trace a continuous structural sparsity path in one training run.
• It enables dynamic network growth and pruning by selectively adding or removing filters based on statistical criteria and selection consistency.
• Empirical studies on datasets like MNIST and CIFAR-10 show that S²-LBI achieves competitive accuracy with significantly reduced parameter counts.

Stochastic Split Linearized Bregman Iteration (S²-LBI) is an algorithm specifically designed for efficient, single-pass training of deep neural networks with structured sparsity via an iterative regularization path. By integrating Linearized Bregman Iterations (LBI) with scalable mini-batch stochastic gradients, S²-LBI produces a continuum of models—from very sparse to fully dense—in a single training run. This structural-sparsity solution path intrinsically supports dynamic network enlargement or simplification in a computationally consistent and theoretically principled manner, delivering selection consistency alongside computational efficiency (Fu et al., 2019).

1. Motivation and Background

Conventional deep neural networks achieve expressive power with millions of parameters, but in resource-constrained applications, over-parameterization poses a liability. Standard optimizers such as SGD and Adam lack intrinsic support for parameter selection or pruning; typically, ℓ₁ or group-ℓ₁ penalties are added, necessitating repetitive optimization for each fixed regularization weight λ. This approach is computationally expensive and algorithmically ad hoc. S²-LBI bridges this gap by generating the entire structural sparsity path in a single pass, supporting provable recovery of underlying low-dimensional structure and yielding model-selection consistency guarantees that have been established for its deterministic antecedents (Fu et al., 2019).

2. Mathematical Formulation and Algorithm

For a standard $L$-layer deep network $f_{(\Theta)}: \mathcal{X} \to \mathcal{Y}$ with parameters $\Theta = \{W^1, \ldots, W^L\}$, S²-LBI introduces a splitting mechanism by associating to each weight $W^\ell$ an auxiliary variable $\Gamma^\ell$, coupled via a quadratic penalty. The penalized objective is: $$\bar{L}(\Theta, \Gamma) = L(\Theta) + \frac{1}{2\nu} \|W^\ell - \Gamma^\ell\|_2^2$$ where $L(\Theta)$ denotes the empirical loss, and $\nu > 0$ controls the strength of the split. Structured sparsity is enforced on $\Gamma^\ell$ by a convex penalty $\Omega$ (group-lasso for filters: $f_{(\Theta)}: \mathcal{X} \to \mathcal{Y}$0; ℓ₁ for FC layers: $f_{(\Theta)}: \mathcal{X} \to \mathcal{Y}$1).

The update rules in discrete time (omitting mini-batch indices) are:

• $f_{(\Theta)}: \mathcal{X} \to \mathcal{Y}$2
• $f_{(\Theta)}: \mathcal{X} \to \mathcal{Y}$3
• $f_{(\Theta)}: \mathcal{X} \to \mathcal{Y}$4
• $f_{(\Theta)}: \mathcal{X} \to \mathcal{Y}$5
• $f_{(\Theta)}: \mathcal{X} \to \mathcal{Y}$6

Here, $f_{(\Theta)}: \mathcal{X} \to \mathcal{Y}$7 is the proximal operator (e.g., soft/group-thresholding), $f_{(\Theta)}: \mathcal{X} \to \mathcal{Y}$8 is a “dual” variable, and $f_{(\Theta)}: \mathcal{X} \to \mathcal{Y}$9 is the sparse approximation. Each S²-LBI run produces the full path $\Theta = \{W^1, \ldots, W^L\}$0, supporting downstream structural selection (Fu et al., 2019).

3. Continuous-Time Path and Theoretical Guarantees

S²-LBI can be viewed as a discretization of the continuous-time Stochastic Split Linearized Bregman Inverse-Scale-Space (S²-LBISS) dynamics. With step size $\Theta = \{W^1, \ldots, W^L\}$1: $\Theta = \{W^1, \ldots, W^L\}$2

$\Theta = \{W^1, \ldots, W^L\}$3

Here, the artificial “time” $\Theta = \{W^1, \ldots, W^L\}$4 is analogous to a regularization parameter: as $\Theta = \{W^1, \ldots, W^L\}$5 increases, new entries in $\Theta = \{W^1, \ldots, W^L\}$6 arise in order of importance, tracing a path from sparse to dense. Early stopping at $\Theta = \{W^1, \ldots, W^L\}$7 is used for parsimony to prevent overfitting. For generalized linear models, under an irrepresentable-type condition and Bregman divergence bounds, path-consistency is established: the true support is exactly recovered at finite $\Theta = \{W^1, \ldots, W^L\}$8 (Fu et al., 2019).

4. Model Selection, Pruning, and Network Growth

The structural-sparsity path output by S²-LBI directly facilitates on-the-fly network pruning or growth:

• Forward (Growth) Criterion: At layer $\Theta = \{W^1, \ldots, W^L\}$9, if $W^\ell$0 exceeds a preset threshold $W^\ell$1 (e.g., $W^\ell$2), $W^\ell$3 new randomly initialized filters are added to $W^\ell$4 and S²-LBI continues in the expanded space.
• Backward (Pruning) Criterion: After running S²-LBI, each filter $W^\ell$5 receives a score $W^\ell$6, where $W^\ell$7 is filter magnitude at termination, $W^\ell$8 is the first iteration when $W^\ell$9 became nonzero. The lowest-scoring filters are pruned, yielding negligible accuracy loss.
• Optimal Stopping Time: $\Gamma^\ell$0 is chosen via monitoring held-out validation error along the path, or by fixing a desired sparsity.

This mechanism supports highly flexible, one-pass procedures for both network enlargement and simplification (Fu et al., 2019).

5. Computational Complexity and Practical Implementation

Each S²-LBI iteration introduces only two inexpensive operations over standard SGD: a gradient step on $\Gamma^\ell$1 and a proximal map. For a convolutional layer with $\Gamma^\ell$2 parameters, $\Gamma^\ell$3 (group-thresholding) is $\Gamma^\ell$4—the cost is essentially identical to vanilla SGD. Unlike traditional ℓ₁-SGD, which requires $\Gamma^\ell$5 runs for $\Gamma^\ell$6 different $\Gamma^\ell$7 values, S²-LBI delivers the entire sparsity path with a single execution.

Key practical guidelines include: step-size $\Gamma^\ell$8, damping $\Gamma^\ell$9, split weight $$\bar{L}(\Theta, \Gamma) = L(\Theta) + \frac{1}{2\nu} \|W^\ell - \Gamma^\ell\|_2^2$$0–$$\bar{L}(\Theta, \Gamma) = L(\Theta) + \frac{1}{2\nu} \|W^\ell - \Gamma^\ell\|_2^2$$1, and mini-batch sizes $$\bar{L}(\Theta, \Gamma) = L(\Theta) + \frac{1}{2\nu} \|W^\ell - \Gamma^\ell\|_2^2$$2–$$\bar{L}(\Theta, \Gamma) = L(\Theta) + \frac{1}{2\nu} \|W^\ell - \Gamma^\ell\|_2^2$$3. Early stopping is best managed by monitoring the validation loss or accuracy along the sparse solution path (Fu et al., 2019).

6. Empirical Results and Validation

S²-LBI achieves competitive and highly compressed models on standard datasets:

• MNIST Growth: Starting from a single $$\bar{L}(\Theta, \Gamma) = L(\Theta) + \frac{1}{2\nu} \|W^\ell - \Gamma^\ell\|_2^2$$4 filter plus one FC layer, S²-LBI dynamically adds filters. After $$\bar{L}(\Theta, \Gamma) = L(\Theta) + \frac{1}{2\nu} \|W^\ell - \Gamma^\ell\|_2^2$$5 epochs, the evolved architecture contains $$\bar{L}(\Theta, \Gamma) = L(\Theta) + \frac{1}{2\nu} \|W^\ell - \Gamma^\ell\|_2^2$$6 filters and $$\bar{L}(\Theta, \Gamma) = L(\Theta) + \frac{1}{2\nu} \|W^\ell - \Gamma^\ell\|_2^2$$7 hidden units ($$\bar{L}(\Theta, \Gamma) = L(\Theta) + \frac{1}{2\nu} \|W^\ell - \Gamma^\ell\|_2^2$$8K parameters), reaching $$\bar{L}(\Theta, \Gamma) = L(\Theta) + \frac{1}{2\nu} \|W^\ell - \Gamma^\ell\|_2^2$$9 test accuracy—comparable to LeNet-5’s $L(\Theta)$0 with about $L(\Theta)$1 the parameters.
• MNIST Pruning: Starting with full LeNet-5 ($L(\Theta)$2K parameters), backward S²-LBI prunes $L(\Theta)$3 of the weights (notably in conv.c5), yielding a $L(\Theta)$4K-parameter network with $L(\Theta)$5 accuracy and no further fine-tuning.
• CIFAR-10 Growth: Toy networks (with $L(\Theta)$6 or $L(\Theta)$7 conv layers) grown by S²-LBI reach $L(\Theta)$8 and $L(\Theta)$9 accuracy with $\nu > 0$0 and $\nu > 0$1 filters, matching SGD/Adam baselines for size.
• CIFAR-10 Pruning: On ResNet-20, S²-LBI pruning outperforms ridge, ℓ₁, group-ℓ₁, and random strategies up to $\nu > 0$2 parameter reduction. Combining magnitude $\nu > 0$3 and entry-time $\nu > 0$4 into $\nu > 0$5 produces superior trade-offs empirically (Fu et al., 2019).

7. Connections and Significance

S²-LBI generalizes Split LBI from quadratic loss to arbitrary deep networks, extending path-consistency and sparsity-model recovery to stochastic optimization settings relevant for modern large-scale learning. Its unification of model selection and efficient single-pass training marks a distinct departure from conventional regularization workflows, providing a theoretically grounded, structurally aware alternative for dynamic neural network selection. Its design offers concrete benefits for embedded systems, mobile applications, and situations where on-the-fly architecture adaptation is paramount (Fu et al., 2019).