---
title: Coordinate-Wise Descent Algorithms
url: https://www.emergentmind.com/topics/coordinate-wise-descent-algorithms
type: topic
---

# Coordinate-Wise Descent Algorithms

Coordinate-wise descent algorithms are iterative optimization methods that minimize an objective function by updating one variable or one block of variables at a time, keeping the others fixed. These methods have seen renewed interest due to their scalability for high-dimensional problems and their ability to exploit problem structure for efficient computation. The landscape of coordinate-wise descent encompasses deterministic and randomized variants, primal and primal-dual frameworks, selection rules (cyclic, random, greedy), and recent extensions to nonconvex, online, distributed, and manifold settings.

## 1. Foundational Principles and Algorithmic Structure

At each iteration, a coordinate-wise descent algorithm selects a coordinate (or block) $i$ and solves an update subproblem along that direction. For a convex differentiable objective $f$ over $x\in\mathbb{R}^n$, the prototypical gradient step for coordinate $i$ is
\[
x_i^{(k+1)} = x_i^{(k)} - \alpha_k [\nabla f(x^{(k)})]_i,
\]
with the other coordinates unchanged. More generally, for compositional objectives $h(x)=f(x)+\sum_i r_i(x_i)$, or block forms, the update may use exact minimization along $i$, a coordinate-wise proximal mapping, or a surrogate quadratic approximation [1502.04759], [1610.00040].

The coordinate-selection rule critically affects convergence and efficiency:
- **Cyclic**: visit each coordinate in a fixed or permuted cyclic order;
- **Randomized**: select coordinates independently from a fixed or adaptively updated probability distribution;
- **Greedy (Gauss–Southwell)**: pick the coordinate with maximum partial derivative magnitude or maximal expected objective decrease.

Coordinate updates can be generalized to blocks—referred to as block coordinate descent (BCD)—with similar algorithmic templates and analysis [1502.04759], [1601.00863].

## 2. Convergence Theory and Worst-case Guarantees

For convex objectives with coordinate-wise Lipschitz gradients, randomized coordinate descent (RCD) with uniform sampling and stepsizes $\alpha_i \leq 1/L_i$ generates iterates with expected sublinear convergence $O(n/k)$ and, under strong convexity, linear convergence at rate $O((1 - \frac{\mu}{nL_{\max}} )^k )$ [1502.04759], [1610.00040].

Cyclic methods have convergence rates similar in order but with potentially larger constants. Beck–Tetruashvili's analysis for cyclic coordinate descent on $n$-dimensional convex objectives gives
\[
f(x^k)-f^* \leq \frac{4 n (1+ n L^2 / L_{\max}^2 ) R_0^2 }{k+8},
\]
for stepsizes $\alpha=1/L_{\max}$ [1502.04759], [2211.17018].

Recent advances use performance estimation problems (PEP) cast as semidefinite programs to compute tight numerical worst-case bounds for CCD and alternating minimization (AM), dramatically improving previous analytical constants (by factors of up to 10 for CCD) [2211.17018], [2507.16675]. Notably, for $p$ blocks and step size $\alpha=1/L$, the exact bound after $K$ cycles is
\[
f(x^{pK}) - f^* \leq \frac{C}{L(K+1)} R^2,
\]
with $C \approx 1.87$ for CCD, $C \approx 1.02$ for AM at $p=2$ [2211.17018]. Lower bounds show deterministic cyclic methods can be up to $p$ times slower than full gradient descent in worst case [2507.16675].

Accelerated randomized coordinate descent achieves $O(p^2/N^2)$ complexity [1502.04759], [1412.8060], [2507.16675], but deterministic cyclic-accelerated analogues are provably inefficient for worst-case performance; randomness is essential for acceleration in practice [2211.17018], [2507.16675].

## 3. Selection Rules, Sampling, and Adaptivity

ASR (adaptive selection rules) and importance sampling schemes dynamically assign coordinate-update frequencies based on per-coordinate progress rates or local Lipschitz constants, yielding dramatic speedups and often near-optimal allocation of computational resources [1401.3737], [1412.8060]. For instance, Nesterov's analysis suggests sampling coordinates proportional to their Lipschitz constants:
\[
p_i \propto L_i
\]
to minimize iteration complexity [1412.8060], [1401.3737]. Online adaptation mechanisms, e.g., ACF-CD, estimate optimal sampling frequencies during runtime, automatically adjusting to changing landscape and yielding substantial practical improvements [1401.3737].

Randomized block selection is robust across a vast range of settings (serial, parallel, distributed, or asynchronous) as long as probabilities are positive for all coordinates [1412.8060], [1601.00863], [2504.19004]. Arbitrary sampling and importance sampling allow coordinate updates to be tuned to the data distribution or computational architecture.

Greedy rules (Gauss–Southwell) maximize per-step reduction but incur additional cost, especially for nonseparable or large-block scenarios [1502.04759], [1610.00040]. In distributed and asynchronous settings (e.g., setwise CD in decentralized optimization), local greedy (GS) selection within accessible coordinate subsets affords iteration speedups proportional to set size $|S|$ compared to uniform sampling [2504.19004].

## 4. Extensions: Nonconvex, Fractional, Distributed

Source: https://www.emergentmind.com/topics/coordinate-wise-descent-algorithms