Enhanced Cyclic Coordinate Descent (ECCD)

• Enhanced Cyclic Coordinate Descent is a variant of cyclic coordinate descent that uses Taylor expansion to unroll scalar recurrences into batched linear algebra, improving efficiency in elastic net penalized models.
• ECCD leverages a tunable parameter 's' to transition from standard cyclic updates (s=1) to accelerated performance (s>1) without affecting convergence properties.
• The broader ECCD framework also integrates adaptive coordinate frequencies, variance reduction, and proximal updates to enhance stability and runtime performance across various optimization tasks.

to=arxiv_search 一本道高清无码json
{"query":"all: \"Enhanced Cyclic Coordinate Descent\" OR ti:\"Enhanced Cyclic Coordinate Descent\"","max_results":10,"sort_by":"submittedDate","sort_order":"descending"}{"query":"ti:\"Coordinate Descent Algorithms\" OR ti:\"A Primer on Coordinate Descent Algorithms\" OR ti:\"On the Worst-Case Analysis of Cyclic Coordinate-Wise Algorithms on Smooth Convex Functions\"","max_results":10,"sort_by":"relevance","sort_order":"descending"}{"query":"ti:\"Active-set identification with complexity guarantees of an almost cyclic 2-coordinate descent method with Armijo line search\" OR ti:\"Cyclic Block Coordinate Descent With Variance Reduction for Composite Nonconvex Optimization\"","max_results":10,"sort_by":"relevance","sort_order":"descending"}{"query":"ti:\"Interpreting and Extending The Guided Filter Via Cyclic Coordinate Descent\"","max_results":10,"sort_by":"relevance","sort_order":"descending"}{"query":"ti:\"Coordinate Descent with Online Adaptation of Coordinate Frequencies\" OR ti:\"Random Permutations Fix a Worst Case for Cyclic Coordinate Descent\"","max_results":10,"sort_by":"relevance","sort_order":"descending"}{"query":"ti:\"Enhanced Cyclic Coordinate Descent Methods for Elastic Net Penalized Linear Models\"","max_results":10,"sort_by":"relevance","sort_order":"descending"}{"query":"ti:\"On the Worst-Case Analysis of Cyclic Block Coordinate Descent type Algorithms\"","max_results":10,"sort_by":"relevance","sort_order":"descending"}{"query":"ti:\"A Cyclic Coordinate Descent Algorithm for lq Regularization\"","max_results":10,"sort_by":"relevance","sort_order":"descending"}{"query":"ti:\"A Cyclic Coordinate Descent Method for Convex Optimization on Polytopes\"","max_results":10,"sort_by":"relevance","sort_order":"descending"}{"query":"ti:\"On the Finite Time Convergence of Cyclic Coordinate Descent Methods\"","max_results":10,"sort_by":"relevance","sort_order":"descending"}{"query":"ti:\"Analyzing Random Permutations for Cyclic Coordinate Descent\"","max_results":10,"sort_by":"relevance","sort_order":"descending"}{"query":"ti:\"In this paper, we will disclose that the Guided Filter (GF) can be interpreted as the Cyclic Coordinate Descent\"","max_results":10,"sort_by":"relevance","sort_order":"descending"}I’ll build the encyclopedia entry around the explicit ECCD paper [2510.19999] and use the additional cyclic-coordinate literature you supplied for terminology, theory, and applications.
Enhanced Cyclic Coordinate Descent (ECCD) denotes cyclic coordinate descent methods augmented with structural modifications intended to improve efficiency, stability, or convergence behavior. The term is used explicitly in the context of elastic net penalized generalized linear models, where ECCD redesigns cyclic coordinate descent by performing a Taylor expansion around the current iterate, unrolling vector recurrences, and reformulating the resulting computations into batched linear algebra; in that formulation, a tunable integer parameter (s) controls the degree of unrolling, (s=1) recovers the original coordinate descent method, and (s>1) yields performance improvements without affecting convergence [2510.19999]. A broader reading of the literature suggests that ECCD is also a useful umbrella for cyclic schemes enhanced by coordinate-wise Lipschitz scaling, proximal or block updates, adaptive coordinate frequencies, random-permutation cycling, variance reduction, almost cyclic pair selection, and task-specific objective reformulations [1502.04759].

1. Definition, scope, and relation to classical cyclic coordinate descent

Classical cyclic coordinate descent updates one coordinate or block at a time in a fixed repeated order. In the smooth case, a prototypical update has the form
[
x^{k+1} = x^k - \alpha_k [\nabla f(x^k)]_{i_k} e_{i_k},
]
with cyclic selection
[
i_0=1,\quad i_{k+1} = [i_k \mod n] + 1.
]
For separable regularization, the corresponding coordinatewise proximal step is obtained by solving a one-dimensional subproblem [1502.04759].

Within this baseline, “enhancement” does not denote a single universally standardized algorithm. Several core coordinate-descent references explicitly note that Enhanced Cyclic Coordinate Descent is not a named algorithm in their taxonomy, but they identify the ingredients from which such a method is naturally built: refined step-size rules, coordinate-wise Lipschitz constants, block-separable structure, extrapolation, better sampling or ordering, proximal updates, structural exploitation, and parallelism [1502.04759]. This suggests that ECCD is best understood as a design family rather than a single canonical procedure.

The explicit 2025 ECCD formulation is narrower and more concrete. It targets generalized linear models with elastic net constraints and reworks the cyclic solver so that nonlinear operations arising in gradient computation are avoided by a Taylor expansion around the current iterate. This converts repeated scalar recurrences into more efficient batched computations and is presented as a solver that avoids the convergence delay and numerical instability exhibited by block coordinate descent [2510.19999].

2. Core algorithmic architecture in elastic net penalized generalized linear models

In the explicit ECCD formulation for generalized linear models, the optimization problem is
[
\min_{\beta, \beta_0}\;\; \mathcal{L}(\beta,\beta_0) = -\frac{1}{n}\,\ell(\beta,\beta_0) + \lambda \Bigl(\tfrac{1-\alpha}{2}|\beta|_2^2 + \alpha|\beta|_1\Bigr),
]
with design matrix (X \in \mathbb{R}^{n \times p}), coefficients (\beta \in \mathbb{R}^p), intercept (\beta_0), and exponential-family log-likelihood (\ell) [2510.19999].

Standard cyclic coordinate descent for this setting uses a local quadratic approximation coordinate by coordinate. For a coefficient (\beta_j), the update is
[
\beta_j^{(t+1)} \leftarrow \frac{ S!\Bigl( \frac{1}{n} g_j - \frac{1}{n}h_j\,\beta_j^{(t)},\; \lambda \alpha \Bigr) }{ -\frac{1}{n} h_j + \lambda (1-\alpha) },
]
where (g_j) and (h_j) are the first and second derivatives of the log-likelihood with respect to (\beta_j), and (S(z,\gamma)) is the soft-thresholding operator [2510.19999]. The computational bottleneck is the vector recurrence induced by updating the linear predictor (\eta = \beta_0 + X\beta), because every coordinate change nominally requires recomputation of nonlinear quantities such as (F'(\eta)) and (F''(\eta)).

ECCD replaces that per-coordinate recomputation by blockwise recurrence unrolling. Coordinates are processed in groups of size (s), and within each block the mean function is approximated by a first-order Taylor expansion around the block start:
[
F'q\bigl(\beta_0^{(k-1)s+\ell}\mathbf{1}_n + X\beta^{(k-1)s+\ell}\bigr)
\approx F'\bigl(\beta_0^{(k-1)s}\mathbf{1}_n + X\beta^{(k-1)s}\bigr)
+\sum{i=0}^{\ell-1} F''\bigl(\beta_0^{(k-1)s}\mathbf{1}_n + X\beta^{(k-1)s}\bigr)\odot (X e^{(k-1)s+i})\,\Delta\beta^{(k-1)s+i}.
]
This lets the method evaluate (F'(\cdot)) and (F''(\cdot)) once per block rather than once per coordinate, then express all internal updates through a small batched system [2510.19999].

The blockwise form introduces
[
A = X_s^\top \nabla F,\qquad B = X_s^\top \mathrm{diag}(\nabla^2 F)\,X_s,
]
where (X_s) is the submatrix corresponding to the current block. The numerator and denominator of each coordinate update inside the block are then written in terms of (A), (B), the current coefficient values, and the elastic net penalties, after which soft-thresholding is applied. The parameter (s) controls the trade-off: (s=1) yields the original cyclic coordinate descent method, while (s>1) amortizes nonlinear work across a block [2510.19999].

The resulting complexity statement is explicit. For one epoch, ECCD has cost
[
\mathcal{O}\Bigl(\frac{npC}{s}\Bigr) + \mathcal{O}(nps),
]
where (C) is the cost of one evaluation of (F'(\cdot)). Choosing (s=\sqrt{C}) yields (\mathcal{O}(np\sqrt{C})), compared with (\mathcal{O}(npC)) for standard cyclic coordinate descent [2510.19999].

3. Principal enhancement mechanisms in the broader literature

The literature supports several recurring enhancement motifs that fit naturally under ECCD. In many cases the term itself is absent, but the algorithmic ingredients are explicit.

Enhancement motif	Representative formulation	Source
Coordinate-wise or block Lipschitz scaling	(\alpha_i = 1/L_i) or exact one-dimensional minimization	[1502.04759]
Proximal and block cyclic updates	coordinatewise or blockwise proximal-gradient subproblems	[1610.00040]
Adaptive coordinate frequencies	online adaptation of (p_i) and (\pi_i = p_i/\sum_j p_j)	[1401.3737]
Random-permutation cycling	fresh random permutation each epoch	[1607.08320]
Variance-reduced cyclic block descent	recursive variance reduction in cyclic block updates	[2212.05088]
Almost cyclic pair updates	2-coordinate working sets for one linear equality constraint	[1806.07826]
Polytope-aware cyclic updates	vertex-based cyclic steps and away steps	[2303.07642]

Coordinate-wise scaling is one of the oldest and most pervasive enhancements. In convex smooth settings, cyclic or block coordinate descent can use component Lipschitz constants (L_i), exact line minimization, or line search, and for separable regularizers it admits coordinatewise proximal subproblems [1502.04759]. The same logic appears in the primer-style literature: block minimization, block proximal point, and block prox-linear updates turn plain cyclic descent into a more stable and more broadly applicable solver for composite objectives [1610.00040].

Adaptive ordering constitutes a different enhancement class. The adaptive coordinate frequencies method maintains preferences (p_i), probabilities (\pi_i = p_i/p_{\mathrm{sum}}), and a moving average of progress (\overline r), then updates
[
p_{\mathrm{new}} \leftarrow \Big[\exp\Big(c\Big(\frac{\Delta f}{\overline r}-1\Big)\Big)p_i\Big]{p{\min}}^{p_{\max}},
]
thereby increasing the frequency of coordinates that currently yield better-than-average objective decrease [1401.3737]. This suggests an ECCD interpretation in which a “cycle” is no longer uniformly one visit per coordinate, but an adaptively weighted schedule.

Random-permutation cycling is another minimal but consequential enhancement. Instead of a fixed cyclic order, one samples a fresh random permutation at the start of each epoch and then performs a full sweep in that order. On structured worst-case quadratics, this random-permutations cyclic coordinate descent performs well and can even outperform randomized coordinate descent in a certain regime [1607.08320]. Related analysis on convex quadratics reaches the same conclusion: random-permutation cyclic coordinate descent is an effective enhancement of standard CCD, especially for diagonally scaled rank-one perturbations of the identity [1706.00908].

Further extensions modify the geometry of the coordinate space itself. On one hand, almost cyclic 2-coordinate descent uses pairs ((p_i^k,j(k))), moves along (e_{p_i^k}-e_{j(k)}), and preserves a single linear equality constraint while computing only two partial derivatives per inner step [1806.07826]. On the other hand, PolyCD and PolyCDwA treat vertices of a polytope as “coordinates,” update cyclically over vertices, and incorporate away steps in the spirit of Frank–Wolfe [2303.07642]. In image processing, the guided filter has been interpreted as one cyclic coordinate descent iteration on a least-squares objective, and this viewpoint motivates modified objective functions and rolling filtering schemes [1705.10552].

4. Convergence theory, worst-case analysis, and common misconceptions

A recurrent misconception is that any enhancement of deterministic cyclic coordinate descent automatically inherits accelerated rates known for randomized methods. The recent worst-case literature argues against that view. Exact and numerical worst-case analyses for cyclic coordinate descent on smooth convex functions show sublinear (O(1/k))-type behavior, substantially sharper constants than older analyses, and improved understanding under natural initial-distance assumptions [2211.17018]. However, the same line of work also provides numerical evidence that a standard scheme that provably accelerates random coordinate descent to a (O(1/k^2)) complexity is actually inefficient when used in a deterministic cyclic algorithm [2211.17018].

The 2025 performance-estimation analysis of cyclic block coordinate descent sharpens this point. It shows the convergence of CCD under more natural assumptions than those typically made in the literature, uncovers a scale-invariance property of the worst case of CCD with respect to the coordinate-wise smoothness constants, and derives a lower bound on the worst-case performance of CCD equal to the number of blocks times the worst-case of full gradient descent over the class of smooth convex functions [2507.16675]. A plausible implication is that deterministic ECCD variants should primarily be expected to improve constants, structure exploitation, or empirical runtime, rather than universally change the worst-case asymptotic order.

For convex composite (\ell_1)-regularized problems, finite-time (O(1/k)) guarantees for cyclic coordinate descent and cyclic coordinate minimization were established under an isotonicity assumption, together with the fact that the iterates generated by the cyclic coordinate descent methods remain better than those of gradient descent uniformly over time [1005.2146]. For nonconvex (\ell_q) regularization, a cyclic coordinate descent algorithm with (0<q<1) converges globally to a stationary point as long as the stepsize is less than a positive constant, and under additional conditions converges to a local minimizer [1408.0578]. These results show that “enhancement” is compatible with nonconvex penalties, but only under explicit stepsize, regularity, and proximal-selection conditions.

In nonconvex composite optimization, cyclic block coordinate descent with variance reduction yields non-asymptotic gradient-norm guarantees, and under a Polyak–Łojasiewicz condition a faster linear convergence result is proved [2212.05088]. In singly linearly constrained smooth problems, almost cyclic 2-coordinate descent guarantees that every limit point is stationary under a simple Armijo line search, and finite active-set identification plus complexity guarantees are available under convexity and a quadratic growth condition [2103.04891]. The theoretical picture is therefore not uniform: ECCD is not one theorem but a collection of cyclic methods whose guarantees depend sharply on convexity, smoothness, separability, geometry, and the specific enhancement mechanism.

5. Application domains

The explicit ECCD formulation is rooted in elastic net penalized generalized linear models, including logistic and Poisson regression, where its Taylor-expanded recurrence unrolling is paired with strong rules, active-set updates, and regularization-path warm starts [2510.19999]. This is the most direct use of the label ECCD in the supplied literature.

Sparse modeling is another natural domain. Cyclic coordinate descent for (\ell_q) regularization, with (0<q<1), uses a nonconvex thresholding operator, a stepsize parameter (\mu\in(0,1/L_{\max})), and a tie-breaking rule at the threshold, leading to finite support and sign identification after finitely many iterations [1408.0578]. This places enhanced cyclic methods squarely within nonconvex sparse recovery.

Large-scale machine learning provides several further instantiations. Adaptive coordinate frequencies were evaluated on LASSO, linear SVM, multi-class SVM, and dual logistic regression, where the algorithm offers significant speed-ups over state-of-the-art training methods [1401.3737]. Variance-reduced cyclic block methods were developed for composite nonconvex optimization and experimentally demonstrated efficacy in training deep neural nets [2212.05088]. In constrained optimization over the simplex or the (\ell_1)-ball, PolyCD and PolyCDwA extend cyclic coordinate ideas to vertex-based polytope updates and are applied to (\ell_1)-constrained linear regression, (\ell_1)-constrained logistic regression, and kernel density estimation [2303.07642].

The image-processing literature offers a distinct application logic. Interpreting the guided filter as the cyclic coordinate descent solver of a least-square objective function implies a possible way to extend guided filtering by altering the objective and taking the first-pass iteration of the CCD solver of the modified objective functions. It also yields rolling filtering schemes derived from iterative minimizing procedures [1705.10552]. This suggests that ECCD is not restricted to statistical estimation; it can also serve as an energy-design principle in signal and image processing.

6. Empirical performance, implementation practice, and limitations

The most explicit empirical claim attached to the ECCD label is that the 2025 elastic-net GLM implementation shows consistent performance improvements of (3\times) in average for the regularization path variant on diverse benchmark datasets [2510.19999]. That paper also states that (s>1) yields performance improvements without affecting convergence, whereas (s=1) yields the original CD method, and that a key advantage of ECCD is avoidance of the convergence delay and numerical instability exhibited by block coordinate descent [2510.19999]. Its implementation is in C++ using Eigen to accelerate linear algebra computations [2510.19999].

Across adjacent cyclic-coordinate literature, the empirical pattern is similar but more nuanced. Adaptive coordinate frequencies are reported to offer significant speed-ups over state-of-the-art training methods, yet the same source also notes that the overhead can dominate on very easy, heavily regularized, or otherwise quickly solved problems [1401.3737]. PolyCDwA is reported to achieve strong computational performance for large-scale benchmark problems including (\ell_1)-constrained linear regression, (\ell_1)-constrained logistic regression, and kernel density estimation [2303.07642]. Variance-reduced cyclic block descent demonstrates efficacy in training deep neural nets, but its favorable complexity relies on recursive variance reduction and a Mahalanobis-norm Lipschitz analysis rather than on cyclicity alone [2212.05088].

Several limitations recur. First, the label ECCD is not yet uniform across the literature; in many foundational references it is better read as a synthesized category than as a standardized algorithm name [1502.04759]. Second, deterministic cyclic enhancements should not be presumed to inherit randomized acceleration results; recent worst-case analyses explicitly caution against that inference [2211.17018]. Third, many of the strongest guarantees are highly problem-specific: nonconvex (\ell_q) results rely on a particular proximal thresholding rule [1408.0578]; almost cyclic pair methods rely on a single linear coupling constraint and simple bounds [1806.07826]; and GLM-specific ECCD depends on the Taylor-expanded block recurrence structure [2510.19999].

Taken together, the literature supports a precise but plural view. ECCD is most concretely a Taylor-unrolled cyclic solver for elastic net penalized generalized linear models [2510.19999]. More broadly, it designates a research program in which cyclic coordinate descent is strengthened by structural modeling, adaptive scheduling, proximal or block formulations, stochastic variance reduction, or geometry-aware working sets. This broader interpretation is strongly supported by the surrounding coordinate-descent literature, even where the name itself is absent.