Split Bregman Iteration

• Split Bregman Iteration is a method that reformulates composite optimization problems via variable splitting and augmented Lagrangian relaxation.
• It efficiently decouples smooth and nonsmooth components, enabling alternating minimization steps with updates like soft-thresholding.
• The approach is widely applicable to image restoration, compressive sensing, and deep model sparsification with provable convergence.

Split Bregman Iteration is an operator-splitting framework for efficiently solving a wide class of nonsmooth, constrained, and structured optimization problems that feature convex (and sometimes nonconvex) composite objectives, notably those with non-differentiable regularization such as $\ell_1$ and total variation (TV) penalties. Originally developed for image restoration and compressive sensing, Split Bregman has subsequently seen broad application in high-dimensional statistical learning, inverse covariance estimation, signal processing, and deep model sparsification. Its distinctive approach proceeds by variable splitting, augmented Lagrangian relaxation, and alternating direction updates, typically with provable convergence and low per-iteration complexity.

1. General Principle and Variable Splitting

At the core of Split Bregman iteration is the transformation of a composite-regularized minimization problem into an equivalent constrained formulation by the introduction of auxiliary (split) variables. Consider prototypical convex programs: $\min_{x} \; f(x) + g(Ax)$ where $f$ is smooth and convex, $g$ is convex and possibly nonsmooth (e.g., $\ell_1$ norm), and $A$ is linear. The standard Split Bregman template rewrites: $$\min_{x,z}\; f(x) + g(z) \quad \text{s.t.} \; Ax = z.$$ This decouples the non-differentiable $g$ from the smooth $f$, enabling efficient alternating minimization steps and facilitating the use of shrinkage or proximal operators for nonsmooth terms. The constraints are handled through an augmented Lagrangian or, equivalently, via Bregman iterations. This formulation directly supports a broad array of structured regularizers including the fused lasso, group lasso, joint sparsity, and total variation (Ye et al., 2010, Ye et al., 2010, Isaac et al., 2013).

2. Augmented Lagrangian and Iterative Scheme

For a generic split problem

$$\min_{x, z} \; f(x) + g(z) \quad \text{s.t.} \; Ax - z = 0,$$

the augmented Lagrangian is

$\min_{x} \; f(x) + g(Ax)$0

The iteration, also viewed as an instance of ADMM, alternates:

1. $\min_{x} \; f(x) + g(Ax)$1
2. $\min_{x} \; f(x) + g(Ax)$2
3. $\min_{x} \; f(x) + g(Ax)$3

In many cases, particularly for $\min_{x} \; f(x) + g(Ax)$4 and TV-type regularizations, the $\min_{x} \; f(x) + g(Ax)$5-update admits element-wise soft-thresholding ("shrinkage"), while the $\min_{x} \; f(x) + g(Ax)$6-update is a smooth minimization—either closed-form or reduced to efficiently solvable linear systems (Ye et al., 2010, Hoeltgen et al., 2015, Isaac et al., 2013). This decoupling allows for explicit, computationally inexpensive updates.

3. Structural Examples and Specialized Algorithms

Sparse Inverse Covariance Estimation

The graphical lasso problem,

$\min_{x} \; f(x) + g(Ax)$7

is split via a variable $\min_{x} \; f(x) + g(Ax)$8 decoupling the log-determinant and $\min_{x} \; f(x) + g(Ax)$9 norm, subject to $f$0. The Split Bregman iteration yields a $f$1-update (matrix quadratic solve), an $f$2-update (entrywise soft-thresholding), and a dual update. The matrix square root in the $f$3-step is accelerated via Newton iteration, avoiding costly eigendecompositions and yielding significant speedup over block coordinate descent-based graphical lasso (Ye et al., 2010).

Fused Lasso, Generalized Structured Regularization

Split Bregman supports the fused lasso and its generalizations by introducing split variables for both the coefficients and their differences. For the problem

$f$4

(sparsity and smoothness via $f$5, e.g., finite differences), the iteration includes a smooth $f$6-update (solving a linear system), followed by two soft-threshold updates for auxiliary variables representing the coefficients and differences. The method is proven to scale efficiently to large dimensional “small $f$7, large $f$8” settings (Ye et al., 2010, Simone et al., 2019).

Multi-dimensional Sparse Structured Signal Approximation

For structured sparse coding of multichannel signals enforcing temporal or group smoothness,

$f$9

where $g$0 encodes temporal/structural difference operators, variable splitting and split Bregman yield alternating Sylvester solves and shrinkage steps. This enables efficient exploitation of structured sparsity and improved empirical performance in coding time-varying and group-structured signals (Isaac et al., 2013).

4. Acceleration and Advanced Schemes

Algorithmic enhancements to Split Bregman encompass inner linear-system solvers, multilevel or subspace-acceleration, and integration into accelerated outer-loop frameworks.

• Newton Iteration for Matrix Roots: For inverse covariance estimation, Newton's method evaluates the symmetric positive-definite square root in the matrix-resolvent step, reducing cubic costs relative to standard spectral decompositions (Ye et al., 2010).
• Fast Fixed-Point Linear Solvers: In weighted TV minimization, the Fast Weighted Split Bregman (FWSB) approach uses fixed-point iterations for the linear solve, exploiting the sparsity of discrete gradients and reducing compute time by constant factors (Lazzaro et al., 2018).
• Subspace Acceleration: In joint $g$1-TV or fused-lasso models, split Bregman can be accelerated by alternating standard nonsmooth-proximal substeps with smooth subspace-restricted solves, determined by optimality diagnostics with safeguard fallbacks, thus reducing the number of expensive global iterations (Simone et al., 2019).

5. Connections to Operator Splitting and Monotone Methods

Split Bregman iteration is closely related to classical operator-splitting schemes such as Douglas–Rachford and ADMM. This connection has been formalized in the monotone operator framework:

• Bregman Monotone Operator Splitting: Generalizations admit Bregman divergences generated by strictly convex functions $g$2, enabling metric adaptation, preconditioning, and enhanced convergence rates. For appropriate choice of $g$3, Split Bregman iteration coincides with Douglas–Rachford (when $g$4 is scaled-Euclidean). With alternative $g$5, problem geometry and curvature can be better matched, potentially leading to faster contraction constants (Niwa et al., 2018).
• Bregman Forward–Backward Splitting: Recent results establish convergence for Bregman-based forward-backward schemes in the sum of monotone operators, even in Banach (non-Hilbert) geometries, and recover classical Split Bregman for $g$6 and TV regularization as a special case. Rates of $g$7 in objective decrease are guaranteed under minimal assumptions, with the geometric flexibility to handle more general problem classes (Bùi et al., 2019).

6. Extensions and Contemporary Developments

The Split Bregman framework has been extended to address stochastic optimization and large-scale nonconvex problems—particularly deep model sparsification. Stochastic Split LBI ($g$8-LBI) employs split and Bregman updates in a mini-batch stochastic regime, enabling pathwise selection of sparse network structures with dynamic pruning and expansion:

• Converges empirically in nonconvex regimes at rates comparable to SGD.
• Facilitates automatic sparsity path generation for parameter-efficient models (Fu et al., 2019).

Further, Split Bregman has been successfully adapted for variational optical flow problems, with theoretical convergence and error control, and has been shown to be competitive in speed and accuracy with state-of-the-art solvers for dense correspondence (Hoeltgen et al., 2015).

7. Convergence and Stopping Criteria

Convergence of Split Bregman/ADMM iterations is established under broad regularity for any penalty parameter $g$9. In practice, termination is determined by primal feasibility ($\ell_1$0), objective relative change, or related optimality gaps, with typical tolerance levels of $\ell_1$1 (Ye et al., 2010, Simone et al., 2019). Empirically, only a modest number of outer Bregman or ADMM iterations are required for high-accuracy solutions.

---

In summary, Split Bregman iteration provides a unified and efficient strategy for a wide class of high-dimensional nonsmooth optimization problems. Its modular variable splitting, explicit update steps, and connections to operator splitting underpin its adaptability to diverse settings, including structured sparsity, imaging, and large-scale learning (Ye et al., 2010, Ye et al., 2010, Isaac et al., 2013, Niwa et al., 2018, Bùi et al., 2019, Simone et al., 2019).