--- title: Linearized Bregman Iterations url: https://www.emergentmind.com/topics/linearized-bregman-iterations type: topic --- # Linearized Bregman Iterations Linearized Bregman Iterations Linearized Bregman iterations (LinBreg) are an optimization method for producing sparse solutions to empirical risk minimization in deep learning, grounded in mirror descent and Bregman divergence concepts. When combined with dynamically updated support—alternating between periods of fixed “static” sparsity and phases where new support (non-zeros) can be activated—LinBreg enables very efficient, theoretically principled exploration of sparse parameter spaces. Recent work has proposed a multilevel extension that adapts support freezing, providing both strong computational savings and robust convergence in deep neural network training under high global sparsity constraints [2602.03535]. ## 1. Mathematical Framework The core LinBreg approach addresses composite optimization of the form: \[ \min_{\theta \in \mathbb{R}^{d}}~ \mathcal{L}(\theta) + J(\theta) \] where $\mathcal{L}$ is a differentiable non-convex loss (empirical risk) and $J$ is a proper, convex, lower semi-continuous regularizer, typically promoting sparsity, e.g., $J(\theta) = \lambda \|\theta\|_1$. To facilitate mirror descent, LinBreg uses a $\delta$-elasticized regularizer: \[ J_\delta(\theta) = \frac{1}{2\delta} \|\theta\|^2 + J(\theta) \] and introduces dual variables $v \in \partial J_\delta(\theta)$. Each iteration comprises: \[ \begin{aligned} v^{(k+1)} & = v^{(k)} - \tau \nabla \mathcal{L}(\theta^{(k)}) \\ \theta^{(k+1)} & = \mathrm{prox}_{\delta J} (\delta v^{(k+1)}) \end{aligned} \] For $\ell_1$-regularization, this reduces to iterative soft-thresholding, with $\theta_i$ set to zero when $|v_i| < \lambda$. Bregman divergence associated to $J$ is \[ D_J^p(\tilde\theta, \theta) = J(\tilde\theta) - J(\theta) - \langle p, \tilde\theta - \theta \rangle, \quad p \in \partial J(\theta) \] ## 2. Multilevel LinBreg: Alternating Static and Dynamic Sparsity In practice, LinBreg is enhanced by freezing the sparsity pattern (“support”) periodically: - The parameter vector $\theta$ is split by active groups (i.e., groups for which $\theta_{(g)} \neq 0$). - During $m$ consecutive “coarse” steps, updates are restricted to the support—no new nonzeros are activated, reducing the dimensionality and cost of both forward/backward passes and prox computations. - A full (“fine”) LinBreg step is then performed—computing the gradient in the full space and updating both the support and the active values, potentially activating new coordinates (restoring “dynamic” sparsity). Transitions between phases may be scheduled adaptively based on the relative magnitude of the projected gradient, e.g., performing coarse-only updates while \[ \|R^{(k)} \nabla \mathcal{L}(\theta^{(k)})\| \geq \kappa \|\nabla \mathcal{L}(\theta^{(k)})\|, \quad \|R^{(k)} \nabla \mathcal{L}(\theta^{(k)})\| > \varepsilon \] where $R^{(k)}$ is the restriction to supports. ## 3. Algorithmic Pseudocode \[ \begin{algorithm}[htb] \caption{Dynamic Sparse Multilevel LinBreg} \begin{algorithmic}[1] \Require initial $(\theta^{(0)}, v^{(0)}) \in \partial J_\delta(\theta^{(0)})$, step-sizes $\tau, \{\hat\tau^{i,k}\}$, support-freeze period $m$ \For{$k=0,1,2,\dots$} \State Define active support mask $R^{(k)} = \{i: \theta^{(k)}_i \neq 0\}$ \State $(\hat\theta^0, \hat v^0) \gets (R^{(k)} \theta^{(k)}, R^{(k)} v^{(k)})$ \For{$i=0,\dots,m-1$} \Comment{coarse (support-frozen) steps} \State $\hat g \gets \nabla \hat{\mathcal{L}}^{(k)}(\hat\theta^i)$ \State $\hat v^{i+1} \gets \hat v^{i} - \hat\tau^{i,k} \hat g$ \State $\hat \theta^{i+1} \gets \mathrm{prox}_{\delta \hat J^{(k)}}(\delta \hat v^{i+1})$ \EndFor \State Prolongate: $\tilde\theta \gets \theta^{(k)} + P^{(k)}(\hat\theta^{m} - \hat\theta^{0})$ \State $\tilde v \gets v^{(k)} + P^{(k)}(\hat v^{m} - \hat v^{0})$ \State Full (fine) LinBreg step: $g \gets \nabla \mathcal{L}(\tilde\theta)$ \State $v^{(k+1)} \gets \tilde v - \tau g$, $\theta^{(k+1)} \gets \mathrm{prox}_{\delta J} (\delta v^{(k+1)})$ \EndFor \end{algorithmic} \end{algorithm} \] Hyperparameters include the sparsity control $\lambda$, freezing interval $m$, and step-sizes $\tau < 1/L$ and $\hat\tau^{i,k} \leq 1/(4L)$ for theoretical guarantees. ## 4. Convergence Properties Under the following conditions: - $\mathcal{L}$ is relatively smooth with respect to $J_\delta$ - A Polyak–Łojasiewicz-type Bregman inequality holds - The stochastic (“coarse”) gradient estimator is unbiased with bounded variance one establishes linear convergence in expectation up to a small error from stochasticity: \[ \mathbb{E}[ \mathcal{L}(\theta^{(k)}) - \mathcal{L}^* ] \leq (1-r)^k [ \mathcal{L}(\theta^{(0)}) - \mathcal{L}^* ] + \sum_{i=0}^{k-1} (1-r)^{k-i} \hat\rho_i \] where $r$ is a geometric contraction rate. As $\max_j \hat{\tau}^{j,i} \to 0$, $\hat\rho_i \to 0$, so $\mathcal{L}(\theta^{(k)}) \to \mathcal{L}^*$. The argument leverages alternating full (fine) LinBreg steps, which generate robust descent by relative smoothness and PL inequality, and coarse blocks, which descend in expectation but may incur additional variance. Telescoping across outer iterations yields convergence. ## 5. Computational Complexity and Efficiency Let $f_D$ denote the FLOP count for a dense forward pass, $f_S$ for a sparse pass (proportional to remaining non-zeros), and $s\%$ the achieved sparsity. - Standard LinBreg: $\approx 3f_S + f_D$ FLOPs/iteration (one gradient and two prox in a sparse regime). - Multilevel (ML) LinBreg: one dense (full) step every $(m+1)$-th iteration; otherwise, all operations restricted to the active support. \[ \text{FLOPs}_\text{ML} \approx \frac{m \times (3f_S) + (2f_S + f_D)}{m+1} \] In the regime $m \gg 1$ and high sparsity ($s \approx 96$–$97\%$), the relative computational overhead falls to $\sim 6\%$ of dense SGD training, compared to $38\%$ for standard LinBreg [2602.03535]. ## 6. Empirical Results On standard benchmarks (CIFAR-10, TinyImageNet) and models (ResNet18, VGG16, WideResNet28-10), multilevel LinBreg converges as follows: | Model/Method | Sparsity [%] | Test Acc. [%] | |-----------------------------|--------------|--------------------| | ResNet18, SGD Dense | 0.4 | 92.93 | | ResNet18, Prune+FT @95% | 95.0 | 90.84 | | ResNet18, LinBreg $\lambda$=0.2 | 96.0 | 90.35 | | ResNet18, ML LinBreg $\lambda$=0.007 | 96.1 | 90.24 | | VGG16, Prune+FT @92% | 92.0 | 91.39 | | VGG16, ML LinBreg $\lambda$=0.003 | 91.3 | 90.71 | | WRN28-10, Prune+FT @96% | 96.0 | 90.55 | | WRN28-10, ML LinBreg $\lambda$=0.005 | 96.6 | 91.69 | ML LinBreg consistently matches or exceeds accuracy of standard LinBreg, pruning+fine-tune, and state-of-the-art dynamic sparse training baselines, with the advantage of never requiring a full dense epoch. Timing experiments with SparseProp layers show an observed $49\%$ reduction in end-to-end training time versus dense baselines for CPU training, with forward and backward times reduced by $32\%$ and $58\%$, respectively [2602.03535]. ## 7. Context, Significance, and Implications The LinBreg/multilevel approach provides a mathematics-driven alternative to purely heuristic dynamic sparse training methods like those based on hard-pruning or drop-regrow mask updates. Its convergence is guaranteed under standard relative smoothness and Bregman-PL assumptions and is backed by empirical results at extreme sparsities. The adaptive freezing procedure maximally exploits the induced sparsity, reducing both the dimensionality of gradient computations and the computational graph for most iterations. This directly translates to tangible FLOP and wall-clock training savings on standard architectures at minimal loss of predictive accuracy. A plausible implication is that LinBreg-based and multilevel mirror-descent frameworks can serve as "general-purpose" sparse solvers for deep learning models, particularly when ultra-high sparsity and computational efficiency are paramount, and may be extended to more complex settings such as group sparsity, block-structured regularization, or adaptive per-layer regularization. Further investigation is warranted into large-scale distributed implementations and potential extensions to non-convex, structured, or adaptive regularization penalties [2602.03535]. Source: https://www.emergentmind.com/topics/linearized-bregman-iterations