Block-Activated Decomposition Algorithm

• The paper introduces a sparse optimization framework that integrates block coordinate updates with exact combinatorial search, yielding stronger stationarity criteria than traditional methods.
• It employs a proximal combinatorial subproblem on selected variable blocks, balancing exploration and exploitation through both random and greedy block selection strategies.
• Empirical results on sparsity-constrained and ℓ0-regularized least-squares problems demonstrate enhanced descent quality and convergence, albeit with trade-offs in computational complexity.

The Block-Activated Decomposition Algorithm (BDA) is a sparse-optimization framework introduced under the name “Block Decomposition Algorithm,” and denoted in experiments by DEC; these labels refer to the same method. It targets sparse optimization problems that are inherently NP-hard by combining two mechanisms that are usually separated in the literature: efficient block or coordinate updates, and exact combinatorial search on a small working set. At each iteration, BDA selects a block of coordinates, keeps the complement fixed, and solves the original objective globally on that block through a proximal combinatorial subproblem. In the formulation studied in the paper, this yields stronger stationarity notions than coordinate-wise methods, monotone descent, convergence in expectation under random block selection, and high empirical accuracy on sparsity-constrained and $\ell_0$-regularized least-squares problems (Yuan et al., 2019).

1. Problem class and formal objective

BDA is developed for sparse optimization in the composite form

$F(x) = f(x) + h(x),$

where $f$ is convex, continuously differentiable, and has an $L$-Lipschitz gradient,

$\|\nabla f(x)-\nabla f(y)\|_2 \le L\|x-y\|_2,$

while $h$ induces sparsity either through a hard cardinality constraint or an $\ell_0$ penalty. The two principal instances are sparsity-constrained least squares,

$$\min_{x\in\mathbb{R}^n}\ \frac12\|Ax-b\|_2^2 \quad\text{s.t.}\quad \|x\|_0 \le s,$$

with $A\in\mathbb{R}^{m\times n}$, $b\in\mathbb{R}^m$, and $F(x) = f(x) + h(x),$0, and sparse-regularized least squares,

$F(x) = f(x) + h(x),$1

with $F(x) = f(x) + h(x),$2 (Yuan et al., 2019).

Within the unified composite notation, the sparsity-constrained case is represented by

$F(x) = f(x) + h(x),$3

and the regularized case by

$F(x) = f(x) + h(x),$4

For rate results, $F(x) = f(x) + h(x),$5 is sometimes additionally assumed $F(x) = f(x) + h(x),$6-strongly convex:

$F(x) = f(x) + h(x),$7

A notable structural point is that the main convergence guarantees do not require restricted isometry or restricted strong convexity; the analysis is carried out under standard smoothness and, where needed, standard strong convexity.

The central design choice is to optimize the original objective on a selected block of variables rather than to apply a one-coordinate surrogate step. This suggests a deliberate middle ground between full combinatorial search, which is globally powerful but computationally prohibitive, and coordinate descent, which is cheap but vulnerable to poor local minima.

2. Block activation, selection rules, and subproblem structure

At iteration $F(x) = f(x) + h(x),$8, BDA selects a working set

$F(x) = f(x) + h(x),$9

and keeps the complement $f$0 fixed. The update is obtained by solving the block-restricted proximal subproblem

$f$1

where $f$2 is a proximal parameter (Yuan et al., 2019).

Two block-selection mechanisms are used. Under the random strategy, $f$3 is sampled uniformly among all $f$4 subsets of size $f$5, independently across iterations. This is the regime used to establish convergence in expectation to a block-$f$6 stationary point. Under the greedy strategy, indices are scored through one-coordinate objective changes around the current iterate. Writing

$f$7

the algorithm computes, for $f$8,

$f$9

and, for $L$0, the cost of deactivation,

$L$1

After sorting $L$2 and $L$3 in increasing order, the top-$L$4 indices are selected. The hybrid experimental notation DEC-RiGj means that $L$5 coordinates are chosen randomly and $L$6 greedily, with $L$7.

The block subproblem is still NP-hard in general because $L$8 is either an $L$9 penalty or an indicator of a sparsity set. BDA therefore performs a global combinatorial search only within the active block. Exhaustive support enumeration has worst-case complexity $\|\nabla f(x)-\nabla f(y)\|_2 \le L\|x-y\|_2,$0; for each support pattern, the corresponding convex subproblem on the active coordinates is solved, often in closed form or by a small convex solver when $\|\nabla f(x)-\nabla f(y)\|_2 \le L\|x-y\|_2,$1 is quadratic. The paper also notes exact mixed-integer formulations using a variational form of the $\|\nabla f(x)-\nabla f(y)\|_2 \le L\|x-y\|_2,$2 pseudo-norm and solvers such as branch-and-cut or cutting-plane methods with CPLEX or Gurobi. In practice, exhaustive search can be accelerated by pruning through objective lower bounds, but the worst-case block complexity remains exponential in $\|\nabla f(x)-\nabla f(y)\|_2 \le L\|x-y\|_2,$3.

The iterative procedure is monotone in the objective and satisfies the sufficient decrease inequality

$\|\nabla f(x)-\nabla f(y)\|_2 \le L\|x-y\|_2,$4

Stopping is based on the mean relative objective decrease over a moving window of length $\|\nabla f(x)-\nabla f(y)\|_2 \le L\|x-y\|_2,$5, or on a maximal iteration budget $\|\nabla f(x)-\nabla f(y)\|_2 \le L\|x-y\|_2,$6.

3. Stationarity hierarchy and convergence theory

A central contribution of BDA is its stronger stationarity notion. A point $\|\nabla f(x)-\nabla f(y)\|_2 \le L\|x-y\|_2,$7 is block-$\|\nabla f(x)-\nabla f(y)\|_2 \le L\|x-y\|_2,$8 stationary if for every block $\|\nabla f(x)-\nabla f(y)\|_2 \le L\|x-y\|_2,$9 with $h$0,

$h$1

Equivalently, no descent is possible by globally optimizing any $h$2-dimensional restriction of the original objective. This is stronger than coordinate-wise criteria because it rules out coupled multi-variable improvements that one-dimensional checks can miss (Yuan et al., 2019).

The hierarchy established in the paper is

$h$3

For the $h$4-regularized case, an $h$5-stationary point is defined through the quadratic majorizer

$h$6

by the condition

$h$7

This is the proximal-thresholding optimality condition commonly used in first-order methods. In the paper’s comparison, block-wise optimality is stronger than the coordinate-wise optimality notion associated with Amir Beck et al.’s method. More specifically, block-2 stationary points in the constrained case, and block-1 stationary points in the regularized case with coordinate-wise Lipschitz constants, imply $h$8-stationarity. A common misconception is therefore that BDA is merely a coordinate method with a larger step; its theoretical target is a strictly stronger notion of stationarity whenever the block size exceeds the purely coordinate regime.

Under random uniform block selection, the generated sequence satisfies

$h$9

and

$\ell_0$0

As $\ell_0$1, $\ell_0$2 converges in expectation to a block-$\ell_0$3 stationary point. The global optimum is recovered only in the extreme case $\ell_0$4, since block-$\ell_0$5 stationarity coincides with global optimality for both the constrained and regularized formulations.

For the sparsity-constrained case, assuming $\ell_0$6 is $\ell_0$7-strongly convex and $\ell_0$8, the paper defines

$\ell_0$9

and proves

$$\min_{x\in\mathbb{R}^n}\ \frac12\|Ax-b\|_2^2 \quad\text{s.t.}\quad \|x\|_0 \le s,$$0

together with

$$\min_{x\in\mathbb{R}^n}\ \frac12\|Ax-b\|_2^2 \quad\text{s.t.}\quad \|x\|_0 \le s,$$1

where $$\min_{x\in\mathbb{R}^n}\ \frac12\|Ax-b\|_2^2 \quad\text{s.t.}\quad \|x\|_0 \le s,$$2 is a block-$$\min_{x\in\mathbb{R}^n}\ \frac12\|Ax-b\|_2^2 \quad\text{s.t.}\quad \|x\|_0 \le s,$$3 stationary point.

For the sparse-regularized case, the theory separates support changes from fixed-support refinement. Defining

$$\min_{x\in\mathbb{R}^n}\ \frac12\|Ax-b\|_2^2 \quad\text{s.t.}\quad \|x\|_0 \le s,$$4

the support can change at most

$$\min_{x\in\mathbb{R}^n}\ \frac12\|Ax-b\|_2^2 \quad\text{s.t.}\quad \|x\|_0 \le s,$$5

times in expectation. Once the support is fixed, the paper provides iteration bounds in both convex and strongly convex regimes. This two-phase structure suggests that BDA first spends a finite budget on coordinated activation and deactivation events, and then behaves like a stabilized block proximal method on the final support.

4. Computational trade-offs and block-size effects

The dominant computational parameter is the block size $$\min_{x\in\mathbb{R}^n}\ \frac12\|Ax-b\|_2^2 \quad\text{s.t.}\quad \|x\|_0 \le s,$$6. Random block selection costs $$\min_{x\in\mathbb{R}^n}\ \frac12\|Ax-b\|_2^2 \quad\text{s.t.}\quad \|x\|_0 \le s,$$7 to sample indices. Greedy selection is more expensive because it evaluates all $$\min_{x\in\mathbb{R}^n}\ \frac12\|Ax-b\|_2^2 \quad\text{s.t.}\quad \|x\|_0 \le s,$$8 and $$\min_{x\in\mathbb{R}^n}\ \frac12\|Ax-b\|_2^2 \quad\text{s.t.}\quad \|x\|_0 \le s,$$9, which is $A\in\mathbb{R}^{m\times n}$0 to $A\in\mathbb{R}^{m\times n}$1 depending on the cost of the one-dimensional subproblems. For least squares with $A\in\mathbb{R}^{m\times n}$2 structure, these evaluations reduce to closed-form checks and small univariate optimizations (Yuan et al., 2019).

The block solve itself has worst-case cost

$A\in\mathbb{R}^{m\times n}$3

since there are $A\in\mathbb{R}^{m\times n}$4 activation patterns and each active-support pattern requires solving a small convex problem in at most $A\in\mathbb{R}^{m\times n}$5 variables. When $A\in\mathbb{R}^{m\times n}$6 is quadratic, these convex subproblems admit closed forms or cost $A\in\mathbb{R}^{m\times n}$7 if solved by small linear systems. Memory usage is $A\in\mathbb{R}^{m\times n}$8 to store $A\in\mathbb{R}^{m\times n}$9, $b\in\mathbb{R}^m$0, and $b\in\mathbb{R}^m$1, together with temporary per-iteration memory between $b\in\mathbb{R}^m$2 and $b\in\mathbb{R}^m$3 depending on the block solver.

This leads to the defining trade-off of the method. Larger blocks improve solution quality and stationarity strength because they capture coupled activation and deactivation effects, but the subproblem cost grows exponentially with $b\in\mathbb{R}^m$4. The paper identifies $b\in\mathbb{R}^m$5 as a typical practical range. It also describes adaptive strategies: start with small $b\in\mathbb{R}^m$6, increase it when progress stalls, cap it to respect time budgets, and use pruning or branch-and-bound within the block to reduce enumeration cost. Hybrid block selection, denoted DEC-RiGj, is presented as a way to balance exploration and exploitation by mixing random and greedy coordinates.

The proximal parameter $b\in\mathbb{R}^m$7 is another key control. It stabilizes the nonconvex updates and enforces sufficient decrease. The reported default is $b\in\mathbb{R}^m$8, with the practical observation that larger $b\in\mathbb{R}^m$9 may improve stability at the cost of slower progress.

5. Empirical behavior on least-squares benchmarks

The experiments cover both sparsity-constrained and sparse-regularized least squares, using synthetic datasets such as random-256-1024 and random-256-2048, real data from e2006-5000-$F(x) = f(x) + h(x),$00 with $F(x) = f(x) + h(x),$01, and corrupted variants in which $F(x) = f(x) + h(x),$02 of the entries of $F(x) = f(x) + h(x),$03 are inflated by $F(x) = f(x) + h(x),$04, thereby introducing ill-conditioning and outliers. Baselines include gradient-type methods—PGM, APGM, and QPM—greedy pursuit methods—OMP, ROMP, Subspace Pursuit, and CoSaMP—convex relaxation via PGM-$F(x) = f(x) + h(x),$05, and nonconvex relaxation via PGM-$F(x) = f(x) + h(x),$06 for $F(x) = f(x) + h(x),$07 (Yuan et al., 2019).

For sparsity-constrained problems, BDA in hybrid form, such as DEC-RiGj, consistently reaches lower objective values than PGM, APGM, and QPM, which often become trapped in poor local minima. Relative to OMP, ROMP, Subspace Pursuit, and CoSaMP, it is competitive on well-conditioned synthetic data and markedly better on corrupted and real datasets, including the e2006 variants. The reported convergence behavior also distinguishes random and greedy block selection: random selection tends to yield better quality than purely greedy selection, and combining both is preferable. Typical convergence times for constrained tasks are under approximately $F(x) = f(x) + h(x),$08 seconds.

For sparse-regularized problems, across $F(x) = f(x) + h(x),$09, BDA with configurations such as DEC-R10G2 generally attains the best or near-best objective values compared with PGM-$F(x) = f(x) + h(x),$10, APGM-$F(x) = f(x) + h(x),$11, PGM-$F(x) = f(x) + h(x),$12, and PGM-$F(x) = f(x) + h(x),$13. The nonconvex $F(x) = f(x) + h(x),$14 relaxation is reported to outperform $F(x) = f(x) + h(x),$15, yet BDA surpasses both in most settings, including corrupted datasets. The computational price is higher runtime: BDA requires more time than the baselines, with examples in the range of $F(x) = f(x) + h(x),$16–$F(x) = f(x) + h(x),$17 seconds versus $F(x) = f(x) + h(x),$18–$F(x) = f(x) + h(x),$19 seconds for competing methods. The paper attributes this directly to the exact block search of complexity $F(x) = f(x) + h(x),$20.

Taken together, the experiments support a consistent interpretation: BDA’s principal empirical advantage is not raw iteration speed but higher-quality descent in difficult regimes, particularly under ill-conditioning, correlations, and outlier contamination.

6. Practical usage, extensions, and limitations

The recommended usage is strongly problem dependent. When dimensionality is very high or computation is tight, the paper advises starting with small blocks, such as $F(x) = f(x) + h(x),$21–$F(x) = f(x) + h(x),$22. For ill-conditioned or highly correlated designs, including corrupted matrices, moderate block sizes such as $F(x) = f(x) + h(x),$23–$F(x) = f(x) + h(x),$24 are recommended to permit coordinated moves that escape poor basins. For small to medium problem sizes where high accuracy is critical, larger blocks such as $F(x) = f(x) + h(x),$25–$F(x) = f(x) + h(x),$26 are viable, provided runtime is monitored (Yuan et al., 2019).

Hybrid random-plus-greedy selection is recommended as the default practical strategy. The rationale is explicit in the experimental and theoretical discussion: greedy coordinates exploit large local improvements, while random coordinates improve exploration and reduce myopia. Pure random selection remains attractive when the theoretical convergence-in-expectation regime is prioritized, especially in highly nonconvex landscapes. Pure greedy selection may produce fast initial progress but is more prone to becoming trapped.

The default stopping rule tracks the mean relative objective decrease over a moving window, with representative values $F(x) = f(x) + h(x),$27, $F(x) = f(x) + h(x),$28, and $F(x) = f(x) + h(x),$29. These settings operationalize the monotone descent guarantee rather than replacing it.

The main limitation is the combinatorial block search. The worst-case $F(x) = f(x) + h(x),$30 dependence makes aggressive block sizes expensive, and this cost is intrinsic to the framework’s stronger local optimality. The paper discusses several mitigations: capping $F(x) = f(x) + h(x),$31, pruning supports by lower bounds, warm-starting small convex subproblems, and parallelizing support enumeration because many candidate subpatterns are independent. Another limitation is model generality. The main analysis and experiments focus on least-squares objectives; when $F(x) = f(x) + h(x),$32 is more complicated, such as logistic loss, the paper proposes a Newton-type block-$F(x) = f(x) + h(x),$33 update obtained by majorizing $F(x) = f(x) + h(x),$34 with a local quadratic

$F(x) = f(x) + h(x),$35

and solving

$F(x) = f(x) + h(x),$36

under the same block restriction. The resulting notion is a Newton block-$F(x) = f(x) + h(x),$37 stationary point, which the paper describes as stronger than $F(x) = f(x) + h(x),$38-stationarity but weaker than full block-$F(x) = f(x) + h(x),$39 stationarity.

In that sense, BDA is best understood not as a universal exact solver for sparse optimization, but as a structured local-global hybrid: exact on the activated block, inexact globally, and engineered to raise the stationarity target beyond coordinate-wise methods while retaining tractable per-iteration structure.