Cyclic Coordinate Descent

• Cyclic Coordinate Descent is a deterministic optimization method that updates one coordinate or block in a fixed cyclic order, applicable to smooth, composite, and constrained problems.
• In quadratic problems, CCD closely mirrors the Gauss–Seidel method, leveraging exact line searches and spectral properties to ensure convergence and finite-time model identification.
• The method generalizes to proximal, projected, and block-update schemes, providing practical insights despite potential worst-case inefficiencies compared to randomized variants.

Searching arXiv for recent and foundational CCD papers to support the article. arXiv search query: "cyclic coordinate descent worst-case analysis Gauss-Seidel finite identification coordinate descent" Cyclic Coordinate Descent (CCD) is a deterministic coordinate optimization scheme in which one coordinate, or one block of coordinates, is updated at a time in a fixed cyclic order and the process is repeated over successive sweeps. In the quadratic exact-line-search setting, it is identical to Gauss–Seidel on the associated linear system, while in composite and constrained settings it appears as a broader family of cyclic block, proximal, projection, and pairwise-update methods. Across the literature, CCD is analyzed from several viewpoints: as a Gauss–Seidel iteration for symmetric positive definite systems, as a prox-linear or projected method for separable nonsmooth objectives, as a cyclic block framework for constrained and nonconvex problems, and as a benchmark against randomized or permutation-randomized coordinate rules in worst-case complexity studies (Wright et al., 2017, Abbaszadehpeivasti et al., 2022, Kamri et al., 22 Jul 2025).

1. Canonical formulations and update rules

In its basic smooth unconstrained form, CCD updates a single coordinate by holding all others fixed and cycling deterministically through the indices. For block decompositions $$\mathbb{R}^d=\mathbb{R}^{d_1}\times\cdots\times\mathbb{R}^{d_p}$$, one standard cyclic block-gradient rule is

$$x_i = x_{i-1} - \gamma_\ell\, U_\ell \nabla^{(\ell)} f(x_{i-1}), \qquad \ell = \mathrm{mod}(i,p)+1,$$

with the standard choice $\gamma_\ell=1/L_\ell$ in the coordinate-wise smooth convex setting. In composite separable problems

$\Phi(x)=f(x)+\sum_{j=1}^p g_j(x_j),$

a proximal cyclic variant updates

$x_j^{(j,k)} \leftarrow \prox_{\gamma_j g_j}\!\left(x_j^{(j-1,k)} - \gamma_j \nabla_j f(x^{(j-1,k)})\right),$

with $0<\gamma_j\le 1/L_j$. A different but related convention appears in the finite-time $\ell_1$-regularized analysis, where “CCD” denotes the one-dimensional quadratic-upper-bound update and “CCM” denotes exact coordinate minimization; in that notation,

$$y^{(k,j)}_j \leftarrow S_{\lambda/L}\!\left( y^{(k,j-1)}_j - \frac{[\nabla f(y^{(k,j-1)})]_j}{L} \right).$$

This suggests that the label “CCD” is not fully uniform across papers: some authors reserve it for exact coordinate minimization, while others include prox-linear or upper-bound-based cyclic steps under the same heading (Kamri et al., 22 Jul 2025, Klopfenstein et al., 2020, Saha et al., 2010).

The same deterministic cyclic principle persists in more specialized settings. For unconstrained coordinate-wise smooth convex optimization, it is “one block partial gradient step per iteration,” in cyclic order. For separable nonsmooth composite objectives it is a prox-linear Gauss–Seidel method. For some constrained problems, single-coordinate feasibility is unavailable, and the cyclic idea survives through pairwise or blockwise feasible directions rather than basis-vector moves. A plausible implication is that CCD is best understood not as one unique update formula, but as a scheduling rule: deterministic, repeated, without-replacement traversal of coordinates or blocks.

2. Quadratic problems, exact line search, and the Gauss–Seidel viewpoint

The quadratic case provides the cleanest algebraic model of CCD. For

$$\min_{x\in \mathbb{R}^n} f(x), \qquad f(x)=\frac12 x^T A x,$$

with $A$ symmetric positive definite, exact line search along coordinate $$x_i = x_{i-1} - \gamma_\ell\, U_\ell \nabla^{(\ell)} f(x_{i-1}), \qquad \ell = \mathrm{mod}(i,p)+1,$$0 gives

$$x_i = x_{i-1} - \gamma_\ell\, U_\ell \nabla^{(\ell)} f(x_{i-1}), \qquad \ell = \mathrm{mod}(i,p)+1,$$1

With exact line search on a quadratic, the method is equivalent to Gauss–Seidel applied to $$x_i = x_{i-1} - \gamma_\ell\, U_\ell \nabla^{(\ell)} f(x_{i-1}), \qquad \ell = \mathrm{mod}(i,p)+1,$$2. Writing $$x_i = x_{i-1} - \gamma_\ell\, U_\ell \nabla^{(\ell)} f(x_{i-1}), \qquad \ell = \mathrm{mod}(i,p)+1,$$3, one CCD epoch has iteration matrix

$$x_i = x_{i-1} - \gamma_\ell\, U_\ell \nabla^{(\ell)} f(x_{i-1}), \qquad \ell = \mathrm{mod}(i,p)+1,$$4

so that after $$x_i = x_{i-1} - \gamma_\ell\, U_\ell \nabla^{(\ell)} f(x_{i-1}), \qquad \ell = \mathrm{mod}(i,p)+1,$$5 epochs,

$$x_i = x_{i-1} - \gamma_\ell\, U_\ell \nabla^{(\ell)} f(x_{i-1}), \qquad \ell = \mathrm{mod}(i,p)+1,$$6

This formulation makes the asymptotic per-epoch factor depend on $$x_i = x_{i-1} - \gamma_\ell\, U_\ell \nabla^{(\ell)} f(x_{i-1}), \qquad \ell = \mathrm{mod}(i,p)+1,$$7, directly connecting CCD convergence to the spectral properties of a non-symmetric Gauss–Seidel iteration matrix (Wright et al., 2017, Abbaszadehpeivasti et al., 2022).

The linear-algebraic interpretation extends further. For convex quadratics,

$$x_i = x_{i-1} - \gamma_\ell\, U_\ell \nabla^{(\ell)} f(x_{i-1}), \qquad \ell = \mathrm{mod}(i,p)+1,$$8

minimization is equivalent to solving $$x_i = x_{i-1} - \gamma_\ell\, U_\ell \nabla^{(\ell)} f(x_{i-1}), \qquad \ell = \mathrm{mod}(i,p)+1,$$9. In that setting the cyclic coordinate update is exactly Gauss–Seidel, and the same framework transfers to Kaczmarz and POCS after the usual reformulations. One paper makes the equivalence explicit and proves that the worst-case gap between cyclic and randomized variants for quadratics transfers immediately to Gauss–Seidel, Kaczmarz, and cyclic POCS. Another uses semidefinite-programming performance estimation to analyze cyclic coordinate descent and Gauss–Seidel for positive semidefinite systems with positive diagonal, including the semidefinite case $\gamma_\ell=1/L_\ell$0 rather than only the strictly positive definite case (Sun et al., 2016, Abbaszadehpeivasti et al., 2022).

This quadratic viewpoint also clarifies what CCD is not. It is not inherently stochastic, and it is not inherently tied to exact block minimization beyond this special setting. Exact line search on SPD quadratics gives the clean Gauss–Seidel identity; outside that regime, the cyclic schedule can be paired with prox-linear, projected, Armijo, or generalized-gradient substeps.

3. Convergence theory: finite-time rates, identification, and nonconvex stationarity

Finite-time and asymptotic guarantees for CCD depend strongly on the objective class and the update model. For $\gamma_\ell=1/L_\ell$1-regularized smooth convex problems,

$\gamma_\ell=1/L_\ell$2

one finite-time analysis proves $\gamma_\ell=1/L_\ell$3 convergence rates for two cyclic methods under the isotonicity assumption that

$\gamma_\ell=1/L_\ell$4

is isotone. Starting from a common super- or subsolution, the paper establishes the ordering

$\gamma_\ell=1/L_\ell$5

where $\gamma_\ell=1/L_\ell$6 are gradient-descent iterates, $\gamma_\ell=1/L_\ell$7 are CCD iterates, and $\gamma_\ell=1/L_\ell$8 are cyclic coordinate minimization iterates. In that setting, CCD and CCM inherit the same $\gamma_\ell=1/L_\ell$9 finite-time bound as gradient descent, with CCM at least as good as CCD at every iteration (Saha et al., 2010).

For broader convex composite problems,

$\Phi(x)=f(x)+\sum_{j=1}^p g_j(x_j),$0

proximal cyclic coordinate descent admits a different type of theory: finite-time model identification and local linear convergence. Under smoothness, separable nonsmooth structure, existence of minimizers, the non-degeneracy condition

$\Phi(x)=f(x)+\sum_{j=1}^p g_j(x_j),$1

and local $\Phi(x)=f(x)+\sum_{j=1}^p g_j(x_j),$2 regularity on active coordinates, cyclic coordinate descent identifies the model in finite time, meaning there exists $\Phi(x)=f(x)+\sum_{j=1}^p g_j(x_j),$3 such that for all $\Phi(x)=f(x)+\sum_{j=1}^p g_j(x_j),$4,

$\Phi(x)=f(x)+\sum_{j=1}^p g_j(x_j),$5

If restricted injectivity also holds,

$\Phi(x)=f(x)+\sum_{j=1}^p g_j(x_j),$6

then the identified phase is locally linear, with rate controlled by the spectral radius of the Jacobian of the epoch map restricted to the active subspace (Klopfenstein et al., 2020).

Nonconvex CCD theory is also substantial, though typically weaker in conclusion. For the $\Phi(x)=f(x)+\sum_{j=1}^p g_j(x_j),$7-regularized least-squares problem

$\Phi(x)=f(x)+\sum_{j=1}^p g_j(x_j),$8

a cyclic coordinate descent algorithm based on the nonconvex scalar proximal map satisfies a sufficient decrease inequality

$\Phi(x)=f(x)+\sum_{j=1}^p g_j(x_j),$9

under $x_j^{(j,k)} \leftarrow \prox_{\gamma_j g_j}\!\left(x_j^{(j-1,k)} - \gamma_j \nabla_j f(x^{(j-1,k)})\right),$0, and the full CCD sequence converges to a stationary point. Under additional conditions involving $x_j^{(j,k)} \leftarrow \prox_{\gamma_j g_j}\!\left(x_j^{(j-1,k)} - \gamma_j \nabla_j f(x^{(j-1,k)})\right),$1, $x_j^{(j,k)} \leftarrow \prox_{\gamma_j g_j}\!\left(x_j^{(j-1,k)} - \gamma_j \nabla_j f(x^{(j-1,k)})\right),$2, and the minimum active coefficient magnitude $x_j^{(j,k)} \leftarrow \prox_{\gamma_j g_j}\!\left(x_j^{(j-1,k)} - \gamma_j \nabla_j f(x^{(j-1,k)})\right),$3, the limit point is a local minimizer (Zeng et al., 2014).

A recurring misconception is that deterministic cyclic rules lack meaningful modern theory beyond basic convergence. The current literature does provide finite-time sublinear rates, finite identification results, explicit local linear rates after identification, and nonconvex stationarity theorems. What varies is the strength of the assumptions: isotonicity in one line of work, restricted injectivity and partial smoothness in another, or KL-based arguments and sufficient decrease in nonconvex settings.

4. Worst-case behavior, randomization, and performance-estimation analysis

The sharpest negative results for CCD arise in quadratic worst-case analysis. One paper proves that for convex quadratics with equal diagonal entries, cyclic coordinate descent can indeed be $x_j^{(j,k)} \leftarrow \prox_{\gamma_j g_j}\!\left(x_j^{(j-1,k)} - \gamma_j \nabla_j f(x^{(j-1,k)})\right),$4 slower than randomized coordinate descent in the worst case. In its notation,

$x_j^{(j,k)} \leftarrow \prox_{\gamma_j g_j}\!\left(x_j^{(j-1,k)} - \gamma_j \nabla_j f(x^{(j-1,k)})\right),$5

while

$x_j^{(j,k)} \leftarrow \prox_{\gamma_j g_j}\!\left(x_j^{(j-1,k)} - \gamma_j \nabla_j f(x^{(j-1,k)})\right),$6

and the gap holds for any fixed update order on the hard instance

$x_j^{(j,k)} \leftarrow \prox_{\gamma_j g_j}\!\left(x_j^{(j-1,k)} - \gamma_j \nabla_j f(x^{(j-1,k)})\right),$7

The same paper emphasizes a subtle point: for non-symmetric iteration matrices, spectral radius alone does not automatically furnish a lower bound on convergence rate, so the lower-bound construction requires more than a naive eigenvalue argument (Sun et al., 2016).

Random permutation can remove that worst case. For

$x_j^{(j,k)} \leftarrow \prox_{\gamma_j g_j}\!\left(x_j^{(j-1,k)} - \gamma_j \nabla_j f(x^{(j-1,k)})\right),$8

with exact line search and diagonal scaling $x_j^{(j,k)} \leftarrow \prox_{\gamma_j g_j}\!\left(x_j^{(j-1,k)} - \gamma_j \nabla_j f(x^{(j-1,k)})\right),$9, cyclic CCD, fully randomized CD, and random-permutation cyclic CD exhibit sharply different epoch behavior. CCD has epoch factor $0<\gamma_j\le 1/L_j$0 in the small-$0<\gamma_j\le 1/L_j$1 regime, whereas RCD behaves like $0<\gamma_j\le 1/L_j$2, and RPCD matches RCD asymptotically and can even be slightly better. In one analysis, the asymptotic RPCD recurrence has spectral radius

$0<\gamma_j\le 1/L_j$3

and another derives a more general diagonal-perturbed recurrence with bound

$0<\gamma_j\le 1/L_j$4

while reporting an epoch-wise rate “similar to $0<\gamma_j\le 1/L_j$5” in practice. These results identify a concrete structural failure mode of fixed-order cyclic updates and show that randomizing the order once per epoch disrupts the unfavorable deterministic Gauss–Seidel structure (Lee et al., 2016, Wright et al., 2017).

Recent performance-estimation work moderates the older pessimism without erasing it. Automated worst-case analysis via semidefinite programming yields much tighter bounds than the classical Beck–Tetruashvili-type analyses. For smooth convex functions under stronger bounded-level-set assumptions, a classical bound is

$0<\gamma_j\le 1/L_j$6

while PEP-based analyses report exact or tighter numerical worst-case bounds, including order-of-magnitude improvement for 2-block CCD in one study. Another recent analysis establishes a lower bound

$0<\gamma_j\le 1/L_j$7

showing that CCD’s worst case is at least the number of blocks times the worst case of full gradient descent when the budget is measured in partial updates. These papers also provide evidence that deterministic cyclic acceleration schemes modeled on accelerated randomized coordinate descent may be inefficient in the deterministic setting (Kamri et al., 2022, Abbaszadehpeivasti et al., 2022, Kamri et al., 22 Jul 2025).

This suggests a balanced interpretation. Worst-case theory is genuinely unfavorable to CCD on some structured families, and randomization can be provably decisive there. At the same time, modern numerical worst-case analysis indicates that the older cubic-in-$0<\gamma_j\le 1/L_j$8 or highly pessimistic bounds are not the final word on cyclic methods.

5. Generalizations beyond separable unconstrained problems

CCD extends well beyond unconstrained smooth convex minimization. One broad generalization is a constrained cyclic block coordinate framework with generalized gradient projections. For

$0<\gamma_j\le 1/L_j$9

the update is not exact block minimization but a generalized projection

$\ell_1$0

with block-separable metrics $\ell_1$1, descent directions $\ell_1$2, and Armijo backtracking. Under $\ell_1$3, block-separable closed convex constraints, and the metric assumptions (BH1)–(BH3), every limit point generated by the cyclic block generalized gradient projection method is stationary, even when $\ell_1$4 is possibly nonconvex (Bonettini et al., 2015).

A second family addresses coupling constraints that make single-coordinate updates infeasible. For

$\ell_1$5

the Almost Cyclic 2-Coordinate Descent method updates pairs $\ell_1$6 along

$\ell_1$7

so that the equality constraint is preserved automatically. One coordinate is cycled, the other is selected by a bound-distance rule that does not use first-order information. Global convergence to stationary points is established under a level-set assumption and standard stepsize conditions, and later work proves finite active-set identification and complexity guarantees under convexity and a quadratic growth condition (Cristofari, 2018, Cristofari, 2021).

A third line adapts the cyclic principle to geometries where coordinate axes are the wrong primitives. On bounded polytopes with vertices $\ell_1$8, PolyCD treats vertices as “coordinates” and cycles through updates of the form

$\ell_1$9

combining cyclic order with Frank–Wolfe-style feasible directions. PolyCD has an $$y^{(k,j)}_j \leftarrow S_{\lambda/L}\!\left( y^{(k,j-1)}_j - \frac{[\nabla f(y^{(k,j-1)})]_j}{L} \right).$$0 rate for smooth convex objectives, while PolyCDwA introduces away steps and achieves a global linear rate for smooth strongly convex objectives, with constants involving the facial distance $$y^{(k,j)}_j \leftarrow S_{\lambda/L}\!\left( y^{(k,j-1)}_j - \frac{[\nabla f(y^{(k,j-1)})]_j}{L} \right).$$1 (Mazumder et al., 2023).

Composite nonconvex block-coordinate variants also appear. The coordinate projected gradient descent method for

$$y^{(k,j)}_j \leftarrow S_{\lambda/L}\!\left( y^{(k,j-1)}_j - \frac{[\nabla f(y^{(k,j-1)})]_j}{L} \right).$$2

handles block coordinate-wise Lipschitz $$y^{(k,j)}_j \leftarrow S_{\lambda/L}\!\left( y^{(k,j-1)}_j - \frac{[\nabla f(y^{(k,j-1)})]_j}{L} \right).$$3, twice continuously differentiable nonseparable $$y^{(k,j)}_j \leftarrow S_{\lambda/L}\!\left( y^{(k,j-1)}_j - \frac{[\nabla f(y^{(k,j-1)})]_j}{L} \right).$$4, and separable convex-set indicator $$y^{(k,j)}_j \leftarrow S_{\lambda/L}\!\left( y^{(k,j-1)}_j - \frac{[\nabla f(y^{(k,j-1)})]_j}{L} \right).$$5. It updates one block cyclically using the partial gradient of $$y^{(k,j)}_j \leftarrow S_{\lambda/L}\!\left( y^{(k,j-1)}_j - \frac{[\nabla f(y^{(k,j-1)})]_j}{L} \right).$$6 and a novel adaptive stepsize defined through a polynomial equation, yielding the descent property

$$y^{(k,j)}_j \leftarrow S_{\lambda/L}\!\left( y^{(k,j-1)}_j - \frac{[\nabla f(y^{(k,j-1)})]_j}{L} \right).$$7

Under boundedness and the KL property, every limit point is stationary and the rate depends on the KL exponent $$y^{(k,j)}_j \leftarrow S_{\lambda/L}\!\left( y^{(k,j-1)}_j - \frac{[\nabla f(y^{(k,j-1)})]_j}{L} \right).$$8 (Chorobura et al., 1 Apr 2025).

6. Applications, implementations, and newer reinterpretations

CCD is also an implementation paradigm. In large $$y^{(k,j)}_j \leftarrow S_{\lambda/L}\!\left( y^{(k,j-1)}_j - \frac{[\nabla f(y^{(k,j-1)})]_j}{L} \right).$$9-regularized problems,

$$\min_{x\in \mathbb{R}^n} f(x), \qquad f(x)=\frac12 x^T A x,$$0

a generic parallel coordinate descent framework separates each iteration into Select, Propose, Accept, and Update, and positions sequential CCD as the singleton-selection special case. In that framework, CCD updates one coordinate at a time in cyclic order and therefore avoids the feature-correlation interference that constrains aggressive parallel schemes such as Shotgun, whose safe parallelism is bounded by

$$\min_{x\in \mathbb{R}^n} f(x), \qquad f(x)=\frac12 x^T A x,$$1

This implementation perspective highlights a persistent tradeoff: cyclic single-coordinate updates are sequential, but they are structurally simple and avoid the synchronization and interference issues that arise in many-coordinate parallel updates (Scherrer et al., 2012).

Several application-specific papers use CCD because its subproblems reduce to tractable one-dimensional problems. In phase retrieval, minimizing

$$\min_{x\in \mathbb{R}^n} f(x), \qquad f(x)=\frac12 x^T A x,$$2

by cyclic coordinate descent turns each step into minimization of a univariate quartic, achieved by solving a cubic equation for stationary points. CCD, randomized CD, and greedy CD are all proved to globally converge to a stationary point of the nonconvex problem; cyclic and randomized versions are also extended to sparse phase retrieval through $$\min_{x\in \mathbb{R}^n} f(x), \qquad f(x)=\frac12 x^T A x,$$3-regularized quartic subproblems, and the same quartic-coordinate framework is applied to blind equalization (Zeng et al., 2017).

In imaging, the guided filter has been interpreted as the CCD solver of a least-square objective function, and that interpretation is used to motivate new GF-like filters and rolling filtering schemes. The paper’s abstract states that this discovery allows modified objective functions whose first CCD pass defines new filters, and reports that the resulting filters and extensions produce state-of-the-art results (Dai, 2017).

Recent engineering-oriented work also reuses the cyclic principle in new computational forms. Enhanced cyclic coordinate descent for elastic-net penalized generalized linear models groups $$\min_{x\in \mathbb{R}^n} f(x), \qquad f(x)=\frac12 x^T A x,$$4 sequential coordinate updates, performs a Taylor expansion around the current iterate, and unrolls the usual coordinate recurrences into batched computations. The abstract reports average training-time improvements of $$\min_{x\in \mathbb{R}^n} f(x), \qquad f(x)=\frac12 x^T A x,$$5 on the regularization-path variant, while the paper emphasizes that $$\min_{x\in \mathbb{R}^n} f(x), \qquad f(x)=\frac12 x^T A x,$$6 recovers the original CD method and that the resulting ECCD avoids the convergence delay and numerical instability exhibited by block coordinate descent (Wang et al., 22 Oct 2025).

Finally, zeroth-order optimization has produced a reinterpretation of cyclic block updates. Coherent Coordinate Descent maintains a buffer of finite-difference coordinate gradients, advances the active coordinate cyclically,

$$\min_{x\in \mathbb{R}^n} f(x), \qquad f(x)=\frac12 x^T A x,$$7

and updates parameters by

$$\min_{x\in \mathbb{R}^n} f(x), \qquad f(x)=\frac12 x^T A x,$$8

The paper states that CoCD is equivalent to Block Cyclic Coordinate Descent with “warm starts,” achieves $$\min_{x\in \mathbb{R}^n} f(x), \qquad f(x)=\frac12 x^T A x,$$9 query complexity per step, and yields error bounds in which larger finite-difference radii can reduce the effective smoothness constant through an implicit smoothing effect (Liang et al., 14 May 2026).

Taken together, these developments suggest that CCD is less a single algorithm than a deterministic ordering principle that recurs across first-order, proximal, projected, Gauss–Seidel, polyhedral, and zeroth-order methods. Its main theoretical tension remains unchanged: fixed cyclic order can be structurally fragile in worst-case instances, yet the same deterministic order supports strong identification results, rich constrained generalizations, and highly efficient domain-specific implementations.