Coordinate-Wise Algorithms Overview

• Coordinate-wise algorithms are optimization methods that update a single coordinate or block per iteration, providing low per-step cost and scalability.
• They leverage decomposability and block-wise Lipschitz smoothness to achieve efficient convergence in convex optimization and large-scale applications.
• Recent developments include accelerated variants and parallel implementations that boost performance in machine learning, statistical estimation, and combinatorial optimization.

Coordinate-wise algorithms, also known as coordinate descent or block-coordinate methods, are a class of optimization and computational strategies that exploit decomposability of a problem along its coordinates or blocks. In each iteration, these algorithms update a single coordinate or a subset (block) of variables while keeping others fixed. This approach leverages problem structure to achieve computational and memory efficiency, especially in high-dimensional and large-scale contexts. Coordinate-wise approaches have found rigorous theoretical footing and significant empirical success across convex optimization, combinatorial linear programs, statistical estimation, machine learning, and computational geometry.

1. Fundamental Definitions and Algorithmic Design

Let $f:\mathbb{R}^d\to\mathbb{R}$ be a convex and differentiable function, and consider the unconstrained minimization problem: $\min_{x\in\mathbb{R}^d} f(x)$ Partitioning the variable $x$ into $p$ blocks $x^{(1)}, \dots, x^{(p)}$, with associated selection matrices $U_\ell\in\mathbb{R}^{d\times d_\ell}$, the $\ell$th block is updated using the partial gradient $$\nabla^{(\ell)}f(x)=U_\ell^{\top}\nabla f(x)\in\mathbb{R}^{d_\ell}$$ (Kamri et al., 22 Jul 2025).

Coordinate-wise smoothness is a key structural assumption: for prescribed block-wise constants $L_1,\dots, L_p$, a function $f$ is block-wise $L_\ell$-smooth if for all $h\in\mathbb{R}^{d_\ell}$,

$$\left\| \nabla^{(\ell)}f(x+U_\ell h) - \nabla^{(\ell)}f(x) \right\| \leq L_\ell\|h\|$$

This yields quadratic upper bounds for each coordinate update direction.

The basic update rules for coordinate-wise algorithms include:

• Cyclic Coordinate Descent (CCD): Sequentially update each block in a fixed order via one-step gradient step:

$$x_i = x_{i-1} - \gamma_\ell U_\ell \nabla^{(\ell)}f(x_{i-1})$$

• Alternating Minimization (AM): Minimize exactly over the chosen block:

$$x_i = \arg\min_{z = x_{i-1} + U_\ell \Delta x^{(\ell)}} f(z)$$

• Cyclic Accelerated Coordinate Descent (CACD): Incorporates Nesterov-type acceleration, but in deterministic cyclic updates (Kamri et al., 22 Jul 2025, Kamri et al., 2022).

Coordinate-wise minimization is equally fundamental for nonsmooth and separable objectives, where each iteration reduces to a one-dimensional or low-dimensional subproblem (Shi et al., 2016, Zhao et al., 2014).

2. Convergence Theory and Worst-Case Analysis

Substantial theoretical results have clarified both the efficiency and intrinsic limitations of coordinate-wise algorithms in the smooth convex setting:

• Sublinear Convergence: For functions with coordinate-wise Lipschitz gradients, coordinate descent algorithms (in essential cyclic or randomized selection) achieve an $O(1/K)$ rate in the objective's reduction after $K$ full cycles (Shi et al., 2016, Kamri et al., 2022, Kamri et al., 22 Jul 2025).
• Scale Invariance: The worst-case performance of CCD is scale-invariant with respect to the block-wise smoothness constants $L_\ell$, allowing normalization of analysis to $L=1$ (Kamri et al., 22 Jul 2025).
• Lower Bounds and Optimality: For $p$ blocks, the worst-case for cyclic coordinate descent is at least $p$ times the worst-case for full gradient descent, highlighting the worst-case loss of per-iteration efficiency (Kamri et al., 22 Jul 2025). Conversely, randomized block selection with acceleration (e.g., Random Accelerated Coordinate Descent, RACD) achieves the $O(1/K^2)$ accelerated rate in expectation, which deterministic cyclic schemes provably cannot match (Kamri et al., 2022, Kamri et al., 22 Jul 2025).
• Performance Estimation Problem (PEP) Framework: Automated SDP-based frameworks allow precise computation of the worst-case performance of coordinate-wise algorithms, significantly sharpening previous (often loose) analytical bounds and enabling fine-grained algorithmic tuning (Kamri et al., 2022, Kamri et al., 22 Jul 2025).
• Special Classes with Global Optimality: For certain classes of LPs and piecewise-affine convex programs, coordinate-wise minimization with relative interior selection yields global optima for all interior coordinate-wise local minima (Dlask et al., 2020).

3. Structural Extensions and Algorithmic Flexibility

Coordinate-wise methods can be generalized and tailored via several key schemes:

• Block Coordinate and Composite Updates: Grouping of variables into non-overlapping or overlapping blocks, coupled with block-wise exact or prox-linear subproblem minimization, enables efficient optimization in sparse and structured regimes (Zhao et al., 2014, Peng et al., 2016).
• Coordinate-Friendly Operators: Operators are said to be "coordinate-friendly" if both the per-block update and necessary data structure/caching can be maintained at cost $O(1/m)$ relative to the full operator (Peng et al., 2016). This principle preserves algorithmic modularity and applies broadly to linear maps, separable nonlinearities, and structured operator-splitting frameworks.
• Parallelism and Asynchrony: Synchronous and asynchronous Jacobi-type variants allow deploying coordinate-wise methods in multi-processor or distributed environments, with theoretical support for convergence under staleness and relaxed locking (Peng et al., 2016, Shi et al., 2016).
• Coordinate-wise Armijo and Step-size Adaptation: Adaptive step-size rules exploiting blockwise smoothness or local curvature, such as coordinate-wise Armijo line search, have led to improved empirical and theoretical performance, especially in non-uniformly scaled or ill-conditioned problems (Truong, 2020, Truong, 2019, Lin et al., 25 Nov 2024).

4. Applications in Statistical Estimation, Machine Learning, and Combinatorics

Coordinate-wise algorithms are ubiquitous in sparse estimation, machine learning, and combinatorial optimization.

• Sparse Estimation: Coordinate-wise and pathwise coordinate descent are canonical in solving high-dimensional regularized regression problems (Lasso, Fused Lasso, group Lasso), generalized linear models, and sparse inverse covariance selection, providing linear (and sometimes superlinear) convergence and match minimax estimation rates with low computational burden (Höfling et al., 2010, Zhao et al., 2014, Yuan et al., 2017, Peng et al., 2016).
• Combinatorial Optimization: Certain LP relaxations of combinatorial problems (e.g., Max-2SAT, vertex cover, min-cut) have special structure permitting exact solution by coordinate-wise minimization. This structure is characterized by at most two nonzero entries per constraint and bounded coefficients, leading to theoretically guaranteed global optimality (Dlask et al., 2020).
• Large-Scale Machine Learning and Imaging: Application domains include CT image reconstruction, total variation denoising, large-scale logistic regression, and deep reinforcement learning policy gradient variance reduction via coordinate-wise baselines (Peng et al., 2016, Kim, 2017, Zhong et al., 2021).
• Eigenvector Computation: Shift-and-invert reduction combined with coordinate descent enables efficient computation of the leading eigenvector of symmetric matrices, achieving convergence matching or surpassing power/lanczos methods, notably for slowly decaying spectra (Wang et al., 2017).
• Coordinate-wise Maxima and Computational Geometry: Efficient computation of coordinate-wise maxima (skyline problems) leverages self-improving strategies with instance-optimal expected runtime matching the information-theoretic lower bound up to additive factors (Clarkson et al., 2012, Clarkson et al., 2012).

5. Practical Algorithms, Implementation, and Empirical Performance

Key features enabling practical deployment of coordinate-wise methods include:

• Low Per-step Cost: Updates require only local data due to block separability or sparsity of the updates, often permitting memory- and cache-efficient implementation (Peng et al., 2016, Shi et al., 2016).
• Order-of-magnitude Empirical Speedup: For Fused Lasso, pathwise Lasso, and minimum distance estimation in regression, coordinate-wise algorithms demonstrate 10–100× speedup over generic solvers, and in some settings approach or outperform specialized interior-point or alternating direction methods (Höfling et al., 2010, Kim, 2017, Zhao et al., 2014).
• Scalability and Parallelization: Asynchronous and multi-threaded variants achieve near-linear speedup with increasing number of cores; synchronous Jacobi variants depend on load balance and communication overhead (Peng et al., 2016).
• Hybrid and Specialized Variants: For challenging problems (e.g., hard penalties, non-convexities, or ill-conditioned data), hybrids with active set screening, maximum-flow subroutines, block splitting, or learn-to-optimize adaptive step-size LSTMs have been introduced, yielding both theoretical convergence and empirical acceleration (Höfling et al., 2010, Lin et al., 25 Nov 2024, Zhao et al., 2014).

6. Recent Developments and Limitations

Recent advances highlight both the strengths and limits of coordinate-wise algorithms:

• Accelerated Cyclic vs. Randomized Selection: Provable acceleration (i.e., $O(1/K^2)$ rates) is fundamentally limited to randomized block-selection; deterministic cyclic acceleration fails to achieve such rates and can even be inferior in worst-case settings (Kamri et al., 2022, Kamri et al., 22 Jul 2025).
• Worst-case vs. Practical Performance Gap: Theoretical lower bounds (linearity in number of blocks, sublinear convergence) are often realized only on constructed adversarial instances, while in practical, real-world problems, coordinate-wise methods can outperform what worst-case analysis suggests (Kamri et al., 22 Jul 2025).
• Solvability and Exactness: Certain nonlinear or high-degree LP relaxations are not amenable to globally exact coordinate-wise minimization, and may become trapped at suboptimal stationary points unless the problem class conditions are met (Dlask et al., 2020).
• Scalability of SDP-based Analysis: While tightest bounds via PEP/SDP are available for small to medium $N,p$, computation grows exponentially in these parameters for extensions involving all possible block-selection paths (Kamri et al., 2022, Kamri et al., 22 Jul 2025).

7. Connections, Extensions, and Ongoing Challenges

Coordinate-wise algorithms remain an active focus for optimization theory, algorithmic research, and high-performance applied computing:

• The connection to operator splitting, primal–dual methods, and stochastic approximation widens the applicability to monotone inclusions, distributed optimization, and large-scale scientific computing (Peng et al., 2016).
• Insights from worst-case analysis inform algorithm design, step-size selection, and the choice between deterministic, randomized, or hybrid update schedules (Kamri et al., 2022, Kamri et al., 22 Jul 2025).
• Open challenges include adaptive block structuring, exploiting higher-order smoothness or curvature, extending global optimality certificates for broader classes of non-differentiable convex programs, and further automating per-application tuning through learning-based meta-optimization (Lin et al., 25 Nov 2024).

Coordinate-wise algorithms have thus achieved a rigorous and versatile toolkit status, supported by comprehensive theoretical guarantees and ongoing empirical successes, with a diverse array of extensions accommodating the demands of modern large-scale and high-dimensional optimization.