Block Coordinate Descent Framework

• Block Coordinate Descent is an optimization framework that systematically updates blocks of variables while keeping others fixed to solve large-scale problems.
• It employs diverse block selection strategies and update rules—such as cyclic, randomized, and gradient-based methods—to enhance convergence in both convex and nonconvex scenarios.
• The framework is widely applied in deep learning, signal processing, sparse regression, and distributed computing, backed by strong theoretical guarantees.

Block Coordinate Descent (BCD) Framework

Block Coordinate Descent (BCD) is a fundamental algorithmic paradigm for solving large-scale optimization problems with a block-structured variable space. At each iteration, BCD updates a subset (“block”) of coordinates or variables while holding the remainder fixed, exploiting problem structure to accelerate convergence and reduce per-iteration complexity. BCD is central in machine learning, signal processing, numerical analysis, communications, and scientific computing, and admits extensive theoretical guarantees and practical adaptations in both convex and nonconvex settings.

1. General BCD Principle and Formulation

Let $x = (x_1, ..., x_L)$ denote a partition of the optimization variables, where each $x_\ell \in \mathbb{R}^{n_\ell}$. BCD targets problems of the form: $\min_{x \in X} f(x),$ where $X = X_1 \times ... \times X_L$ and $f$ may be convex or nonconvex, smooth or nonsmooth, possibly with block-separable (or more general) structure (Hong et al., 2015).

The canonical BCD iteration selects a block index $\ell^k$ at step $k$ (e.g., cyclically, randomly, or greedily) and performs a (possibly inexact) minimization: $$x_{\ell^k}^{k+1} \in \arg\min_{x_{\ell^k} \in X_{\ell^k}} f(x_1^{k+1}, ..., x_{\ell^k-1}^{k+1}, x_{\ell^k}, x_{\ell^k+1}^{k}, ..., x_L^k)$$ with $x_j^{k+1} = x_j^k$ for $j \ne \ell^k$ (Hong et al., 2015, Palagi et al., 2020, Hong et al., 2013).

This block-minimization generalizes to restricted, proximal, variable-metric, and surrogate-based block updates (Hong et al., 2013, Briceño-Arias et al., 30 Oct 2025). BCD encompasses many algorithmic variants (e.g., coordinate descent, block gradient, block proximal algorithms, and alternating minimization).

2. Block Selection, Update Rules, and Algorithmic Variants

The selection of blocks and update rules critically determines BCD performance:

• Block partitioning: Variables may be organized by natural groups—neural network layers (Palagi et al., 2020, Lau et al., 2018, Akiyama, 26 Oct 2025), sensor/receiver groups in wireless networks (Liu et al., 2014), parameters/layers in LLMs (Liu et al., 23 May 2025), or arbitrary coordinate groupings.
• Block selection: Strategies include cyclic (fixed order), randomized, Gauss–Southwell-type (maximum gradient norm), and greedy maximum improvement (Nutini et al., 2017, Hong et al., 2015).
• Update rules:
  • Exact minimization: argmin within a block (Hong et al., 2015, Hong et al., 2013);
  • Gradient or proximal steps: e.g., $$x_\ell^{k+1} = x_\ell^k - \alpha^k \nabla_{x_\ell} f(x^k)$$ with appropriate step size or proximal operator (Briceño-Arias et al., 30 Oct 2025);
  • Quasi-Newton or Newton-type steps for second-order acceleration (Lee et al., 2018, Hong et al., 2015, Lauga, 29 Jan 2026);
  • Armijo-type or backtracking line-search adaptation (Palagi et al., 2020, Briceño-Arias et al., 30 Oct 2025).

Classical BCD can be extended for:

• Nonconvex programs via surrogate minimization, block-wise variable-metric, or inexact block updates (Briceño-Arias et al., 30 Oct 2025, Yuan et al., 2024, Briceño-Arias et al., 30 Oct 2025).
• Parallel/distributed settings by simultaneously updating disjoint variable blocks with appropriate synchronization or asynchrony (lock-free, staleness-tolerant), including A2BCD (Hannah et al., 2018, Hong et al., 2015).
• Flexible deterministic or hierarchical block update orders for multilevel or priority-driven algorithms (Briceño-Arias et al., 30 Oct 2025).

3. Convergence Properties and Complexity Guarantees

Theoretical analysis of BCD is deeply developed for convex and nonconvex problems:

• Stationarity: In general, under mild regularity, the iterates accumulate at stationary points, i.e., $\nabla_{x_\ell} f(x^*) = 0$ for all $\ell$ (Hong et al., 2015, Palagi et al., 2020).
• Global convergence for convex $f$ with unique block subproblem minimizers: iterates converge to the global minimum (Hong et al., 2013, Hong et al., 2015).
• Sublinear rates: For convex composite programs, BCD/BSUM achieves $O(1/r)$ convergence in objective value after $r$ iterations. The accelerated two-block variant achieves $O(1/r^2)$ (Hong et al., 2013).
• Linear rates: If the problem is strongly convex or if the Polyak–Łojasiewicz inequality holds, BCD achieves global geometric/linear rates (Hong et al., 2015, Hannah et al., 2018). The A2BCD method delivers optimal accelerated complexity for strongly convex and smooth $f$, even in asynchronous parallel regimes (Hannah et al., 2018).
• Nonconvex settings: Under coercivity, block-Lipschitz smoothness, and the Kurdyka–Łojasiewicz (KŁ) property (implied, e.g., by real-analyticity or semi-algebraicity), BCD sequences converge globally to critical points, with rates determined by the KŁ exponent (Lau et al., 2018, Zeng et al., 2018, Lauga, 29 Jan 2026, Briceño-Arias et al., 30 Oct 2025). In high-dimensional structured nonconvex settings, coordinate-wise stationarity can be strictly stronger than mere criticality (Yuan et al., 2024).
• Special structures: BCD convergence for functionals with non-separable constraints (e.g., coupled equality or norm constraints) is addressed via feasible or penalty-based block updates, with global $O(1/k)$ or $Q$-linear rates under error bound conditions (Yuan et al., 2024).

The following table summarizes canonical complexity results:

Setting	Rate	References
Nonsmooth convex (exact GSM)	$O(1/r)$	(Hong et al., 2013, Hong et al., 2015)
Strongly convex, smooth	$O(\log 1/\epsilon)$	(Hong et al., 2015, Hannah et al., 2018)
Nonconvex, KŁ property	Variable (finite/lin/sublin)	(Lau et al., 2018, Briceño-Arias et al., 30 Oct 2025, Lauga, 29 Jan 2026)
Accelerated two-block	$O(1/r^2)$	(Hong et al., 2013)

4. Applications and Algorithmic Specialization

BCD is foundational in numerous domains, with specialized algorithms tailored to data/model structure:

• Deep neural network training: Layer-wise BCD (batch or minibatch variants), block-proximal strategies, and block layer decomposition schemes have shown practical and theoretical advantages over standard first-order methods, especially for very deep networks, including global convergence (to stationarity or even global optimizers under sufficient conditions) and improved avoidance of bad local minima or saddle regions (Palagi et al., 2020, Lau et al., 2018, Zhang et al., 2017, Akiyama, 26 Oct 2025).
• Matrix/tensor factorization: Nonnegative matrix factorization via multiplicative updates is a special BCD instance (Hong et al., 2015).
• Sparse regression and graphical lasso: BCD underlies forward-backward splitting, primal-dual GLasso, and QUIC-type Newton block methods for sparse precision matrix estimation, with convergence guarantees in nonconvex regimes (Lauga, 29 Jan 2026).
• Signal processing/communications: BCD realizes efficient beamforming design, transceiver optimization, and MIMO wireless resource allocation, with tight stationarity guarantees (Liu et al., 2014, Hong et al., 2015).
• Discrete optimal transport: BCD-NS combines network simplex with block-structured subproblems, achieving exact optimality under blockwise feasibility and reducing memory/compute overhead (Li et al., 26 Jun 2025).
• Inverse and ill-posed problems: BCD with block-cyclic/loping updates delivers regularization and improved practical convergence in linear inverse problems (Rabanser et al., 2019).

5. Block Size, Parallelism, and Practical Guidelines

The performance of BCD depends significantly on block size, update scheduling, and implementation choices:

• Block granularity: Layer-wise blocks are natural in DNNs (Palagi et al., 2020), while in convex regularized problems, coordinate or group-wise blocks may be optimal (Hong et al., 2015, Briceño-Arias et al., 30 Oct 2025). Larger blocks can exploit second-order structure (e.g., block-diagonal Hessians), message passing for sparse problems, and reduce iteration count, but are more costly per update (Nutini et al., 2017).
• Update order: Cyclic, randomized, essentially cyclic, or priority-weighted/hierarchical block selection schemes exist; greedy rules (e.g., Gauss–Southwell, maximum improvement) can substantially accelerate convergence (Nutini et al., 2017).
• Parallelism: Synchronous and asynchronous parallel BCD methods allow updates of non-overlapping or overlapping blocks on multiple processors/GPUs. Under bounded staleness or delayed updates, convergence is preserved, and acceleration by asynchrony is possible (Hong et al., 2015, Hannah et al., 2018, Liu et al., 23 May 2025).
• Line search/adaptive stepsizes: Robust block-level line-search (e.g., Armijo-type) and quasi-Newton acceleration improve global convergence and local rate (Palagi et al., 2020, Briceño-Arias et al., 30 Oct 2025).
• Block selection and preconditioning: Blockwise Lipschitz sampling, inexact/blockwise convexification, and variable-metric block updates (using local Hessians or Fisher information blocks) can optimize per-block progress (Lee et al., 2018, Lauga, 29 Jan 2026).
• Stopping criteria: Global optimization residuals (norm of the gradient, relative decrease), blockwise residuals, and application- or epoch-limited stopping are commonly used (Palagi et al., 2020, Briceño-Arias et al., 30 Oct 2025).

Empirical studies indicate that BCD methods can outperform standard stochastic or batch gradient-based algorithms on both convergence speed and robustness, especially in regimes with large variable space and/or deep/structured models (Palagi et al., 2020, Lau et al., 2018, Akiyama, 26 Oct 2025).

6. Extensions, Limitations, and Current Directions

Ongoing research in BCD addresses:

• Extension to highly nonconvex and nonsmooth landscapes using the KŁ property, extension to non-Euclidean and product manifold settings, and careful surrogate or variable-metric design in proximal and Newton block updates (Lauga, 29 Jan 2026, Briceño-Arias et al., 30 Oct 2025, Li et al., 2020).
• Asynchronous and communication-efficient distributed BCD for large-scale machine learning and scientific computing (Hannah et al., 2018, Hong et al., 2015, Liu et al., 23 May 2025).
• Blockwise acceleration, active-set identification, and superlinear/finiteness results for problems with sparse or low-rank structure (Nutini et al., 2017).
• Specialized stopping and regularization rules for inverse and ill-posed problems where stability in the presence of noise is crucial (Rabanser et al., 2019).
• Applications in optimal transport, Markov chain block selection, and problems with coupled or nonseparable constraints (Li et al., 26 Jun 2025, Sun et al., 2018, Yuan et al., 2024).

The BCD paradigm persists as a unifying framework, adaptable to the geometry and structure of highly varied nonconvex, high-dimensional optimization problems, with well-characterized convergence guarantees under broad conditions (Hong et al., 2015, Hong et al., 2013, Briceño-Arias et al., 30 Oct 2025).