Split Bregman for Matrix Recovery

• The paper introduces the Split Bregman method, decoupling nonsmooth regularizers from smooth terms via variable splitting and augmented Lagrangian techniques.
• It demonstrates successful application in sparse inverse covariance estimation and latent variable graphical model selection with notable speedups over traditional methods.
• Empirical results in low-rank matrix recovery show improved computational efficiency and accuracy, outperforming alternatives like SVT and FPC in key performance metrics.

The Split Bregman method is a first-order splitting algorithm designed to solve convex matrix recovery problems involving nonsmooth regularizers, particularly those promoting sparsity or low rank. The approach is grounded in variable splitting and augmented Lagrangian techniques, leading to simple subproblems that admit closed-form or highly efficient proximal updates. It has been successfully applied to sparse inverse covariance estimation, latent variable graphical model selection, and matrix recovery via nuclear norm minimization. The method's appeal lies in both its generality—handling a broad class of matrix regularization problems—and its computational advantage over traditional block-coordinate and semidefinite programming solvers.

1. Core Principles and Problem Formulations

Split Bregman leverages the concept of decoupling complex or nondifferentiable regularizers from smooth components in convex optimization. A canonical example is low rank matrix recovery: given linear measurements $Y = \mathcal{A}(Z) + n$ of an unknown $m \times n$ matrix $Z$, the optimal original problem,

$$\min_{Z}\ \mathrm{rank}(Z) \quad \text{s.t.} \quad \mathcal{A}(Z) = Y,$$

is replaced via convex relaxation with nuclear norm minimization,

$$\min_{Z} \ \frac{1}{2}\|\mathcal{A}(Z) - Y\|_2^2 + \lambda \|Z\|_*,$$

where $\|Z\|_*$ denotes the sum of singular values.

For sparse inverse covariance estimation, the objective is to compute a positive definite matrix $\Theta$ given covariance data $S$, via

$$\min_{\Theta \succ 0} \ -\log\det\Theta + \operatorname{tr}(S\Theta) + \lambda \sum_{i \neq j} |\Theta_{ij}|,$$

which combines the Gaussian negative log-likelihood with an $\ell_1$ penalty to induce sparsity in off-diagonal ($m \times n$0) elements (Ye et al., 2010).

In the setting of covariance estimation with latent variables, one seeks a decomposition of the inverse observed covariance into a sparse matrix minus a low-rank component, solved via a convex program penalized by both $m \times n$1 and nuclear (trace) norm terms (Ye et al., 2011).

2. Variable Splitting and Augmented Lagrangian Framework

The hallmark of the Split Bregman method is introducing auxiliary variables to decouple nondifferentiable penalties from smooth terms, transforming a single minimization into a constrained problem. For matrix completion: $m \times n$2 the augmented Lagrangian becomes

$m \times n$3

where $m \times n$4 collects the Bregman iterates and $m \times n$5 is a penalty parameter (Gogna et al., 2013, 0905.1643). This decoupling produces subproblems solvable via efficient least-squares or proximal operators (e.g., soft-thresholding, singular value thresholding).

3. Algorithmic Structure and Proximal Steps

The generic split Bregman (ADMM) iteration consists of alternating minimization:

• Smooth (quadratic or log-det) update:

Solved as a matrix least-squares or via a matrix equation (e.g., for the $m \times n$6-update in precision matrix estimation:

$m \times n$7

with an explicit matrix-analytic solution involving the symmetric matrix square root (Ye et al., 2010, Ye et al., 2011)).

• Nonsmooth proximal update:

For $m \times n$8 or nuclear norm, the step is entrywise soft-thresholding or singular value shrinkage, e.g.,

$m \times n$9

where SVT denotes singular value thresholding (Gogna et al., 2013, 0905.1643).

• Dual (Bregman) update:

Updated as a simple matrix addition:

$Z$0

The iteration continues until primal-dual residuals fall below tolerance. In large-scale settings, the matrix square roots required in the quadratic step are accelerated via Newton–Schulz iteration, reducing the per-iteration cost substantially (Ye et al., 2010).

4. Convergence Properties and Computational Complexity

Under convexity and standard regularity conditions, Split Bregman (a direct instance of ADMM) is globally convergent to the unique solution of the regularized optimization problem. Each iteration incurs:

Subproblem	Computational Complexity
Quadratic update	$Z$1 or $Z$2 (matrix sqrt/SVD/LS)
Proximal update	$Z$3 or $Z$4 (soft/SVT)
Dual update	$Z$5 or $Z$6

The $Z$7-update in graphical models and the auxiliary A-update in latent variable models involve matrix square roots, which are $Z$8 but benefit from Newton iteration acceleration (6–10 iterations typically suffice) (Ye et al., 2010, Ye et al., 2011). For matrix recovery, the Z-update is a regularized least squares problem solved via conjugate gradient or direct methods, and the W-update requires only a single SVD per iteration (Gogna et al., 2013, 0905.1643). For large-scale problems, further acceleration is achieved via approximate randomized SVD (0905.1643).

5. Empirical Performance and Practical Applications

Split Bregman has demonstrated superior speed and accuracy on both synthetic and real-world datasets:

• In sparse inverse covariance estimation (sparse precision matrices), experiments up to $Z$9 showed 2.7× speedup over graphical lasso (SBGLasso: 269s vs Graphical Lasso: 730s), and agreement in objective values within $$\min_{Z}\ \mathrm{rank}(Z) \quad \text{s.t.} \quad \mathcal{A}(Z) = Y,$$0 (Ye et al., 2010).
• Latent variable graphical model selection (SBLVGG) on yeast gene-expression data (up to 6316 genes) attained 2.5–3× faster solution than LogdetPPA and delivered sparser and lower rank estimators, with improved held-out negative log-likelihood (Ye et al., 2011).
• For low-rank matrix recovery, Split Bregman (MSB) outperformed Fixed-Point Continuation (FPC) and Singular Value Thresholding (SVT) in both CPU time and NMSE across varying ranks and sampling ratios; e.g., for $$\min_{Z}\ \mathrm{rank}(Z) \quad \text{s.t.} \quad \mathcal{A}(Z) = Y,$$1, $$\min_{Z}\ \mathrm{rank}(Z) \quad \text{s.t.} \quad \mathcal{A}(Z) = Y,$$2, MSB averaged 32.8s, compared to SVT’s 447.2s (Gogna et al., 2013). On MovieLens 100K, MSB achieved an MAE of 0.78, slightly better than FPC (0.82) and SVT (1.41).

Applications include collaborative filtering (e.g., MovieLens ratings), gene expression analysis, image inpainting, wireless sensor networks, system identification, and control systems (Gogna et al., 2013, 0905.1643).

6. Extensions and Generality

The split Bregman framework naturally accommodates convex regularizers beyond $$\min_{Z}\ \mathrm{rank}(Z) \quad \text{s.t.} \quad \mathcal{A}(Z) = Y,$$3 and nuclear norm, such as elastic-net, fused lasso, and group-lasso penalties (Ye et al., 2010). In these instances, only the proximal update for the auxiliary variable needs to be changed to the appropriate operator, leaving the rest of the algorithm—smooth updates, dual/Bregman step, matrix iterations—unaltered. The method extends to structured matrix decomposition (sparse + low-rank), and convex programs arising in latent variable graphical model selection (Ye et al., 2011).

7. Algorithmic Summary

Representative Split Bregman (MSB) algorithm for nuclear-norm minimization (Gogna et al., 2013), presented in pseudocode for clarity:

$$\min_{Z}\ \mathrm{rank}(Z) \quad \text{s.t.} \quad \mathcal{A}(Z) = Y,$$4

Similar principles apply to precision matrix estimation (SBGLasso) and SBLVGG for graphical models, with problem-specific subproblems and matrix updates (Ye et al., 2010, Ye et al., 2011). Empirical and theoretical results show robust convergence, high computational efficiency, and global optimality under convexity.

---

Key references:

• Split Bregman for sparse inverse covariance estimation (Ye et al., 2010)
• Latent variable graphical model selection (Ye et al., 2011)
• Matrix recovery using Split Bregman (Gogna et al., 2013)
• Fixed point and Bregman iterative methods for rank minimization (0905.1643)