---
title: Randomized Coordinate Gradient Descent
url: https://www.emergentmind.com/topics/randomized-coordinate-gradient-descent
type: topic
---

# Randomized Coordinate Gradient Descent

Randomized coordinate gradient descent is a family of first-order methods that minimizes an objective by updating only a randomly selected coordinate or block at each iteration, rather than applying a full-gradient step. In its modern form, it covers smooth problems \(\min_x f(x)\), composite objectives of the form \(f(x)+\sum_i g_i(x_i)\), and block-structured variables \(x=(x_1,\dots,x_n)\); the basic iteration may be a partial-gradient step such as \(w_{t+1}=w_t-\eta_t \nabla_{i_t}F_S(w_t)e_{i_t}\) or a one-block proximal-gradient subproblem on the sampled block [2108.07414, 1407.0107, 1412.8060]. In several analyses, “randomized coordinate descent” and “randomized coordinate gradient descent” effectively coincide because the method is explicitly gradient-based, using partial derivatives or partial gradients rather than derivative-free coordinate searches [1407.0107, 2108.07414].

## 1. Canonical formulation and algorithmic core

The standard setting partitions the variable into coordinates or blocks and updates only one sampled part at a time. For a composite objective with block-separable regularizer,
\[
\min_x\; f(x)+g(x)=\frac{1}{I}\sum_{i=1}^I f_i(x)+\sum_{j=1}^J g_j(x_j),
\]
a representative randomized block-coordinate proximal-gradient step is
\[
x_{j_t}^{t+1}=\arg\min_{x_{j_t}} \left\langle \nabla_{j_t} f_{i_t}(x^t),x_{j_t}\right\rangle + g_{j_t}(x_{j_t}) + \frac{\eta_t}{2}\|x_{j_t}-x_{j_t}^t\|_2^2,
\]
with all other blocks unchanged; in the purely deterministic full-data version, \(\nabla_{j_t} f_{i_t}(x^t)\) is replaced by \(\nabla_{j_t} f(x^t)\) [1407.0107]. In the smooth unconstrained case, the update reduces to the familiar single-coordinate gradient step
\[
\theta_{n+1}=\theta_n-a_n e_{i_n}\,\partial_{i_n}f(\theta_n),
\]
with \(i_n\) sampled uniformly from \(\{1,\dots,d\}\) [2311.00088].

Quadratic least-squares problems provide the classical archetype. For
\[
L(\beta)=2\|y-X\beta\|^2,
\]
randomized coordinate descent chooses column \(j\) with probability proportional to \(\|X_j\|^2\) and performs
\[
\beta_{t+1}=\beta_t+\frac{X_j^Tr_t}{\|X_j\|^2}e_j,\qquad r_t:=y-X\beta_t,
\]
which is simultaneously an exact one-dimensional minimization and a coordinate-gradient step scaled by the inverse coordinate curvature [1406.5295]. This column-wise viewpoint is central in linear solvers, where coordinate methods act on features, in contrast with row-based randomized Kaczmarz methods that act on data points [1406.5295].

The basic rationale is computational. When only one coordinate or block is touched, per-iteration work can be far smaller than in full-gradient methods. This advantage is strongest when gradients are expensive to obtain in all directions at once, when the objective is anisotropic across coordinates, or when data and variable dimensions are both large [1407.0107, 2311.00088].

## 2. Sampling models, block smoothness, and coordinate geometry

Classical analyses rely on coordinate-wise smoothness constants. In the single-coordinate smooth setting,
\[
|\partial_i f(\theta+e_i h)-\partial_i f(\theta)|\le L_i |h|,\qquad \forall h\in\mathbb{R},
\]
and one defines
\[
L_{\max}:=\max_i L_i,\qquad L_{\mathrm{avg}}:=\frac1d\sum_{i=1}^d L_i,
\]
with the inequalities
\[
L_i\le L_{\mathrm{avg}}\le L_{\max}\le L\le dL_{\max}.
\]
These anisotropic constants explain why coordinate methods can outperform full-gradient descent when the global smoothness constant \(L\) is much larger than the average coordinate curvature \(L_{\mathrm{avg}}\) [2311.00088].

A more general framework replaces serial uniform sampling by arbitrary random subsets of blocks. ALPHA formalizes a sampling \(\hat S\subseteq[n]\) with marginal probabilities \(p_i=\mathbb{P}(i\in\hat S)\) and analyzes it through the Expected Separable Overapproximation condition
\[
\E[f(x+h_{[\hat S]})]\le f(x)+\langle \nabla f(x),h\rangle_p+\frac12\|h\|_{v\circ p}^2,
\]
which yields a unified theory for serial, minibatch, parallel, distributed, uniform, and importance-sampled coordinate methods [1412.8060]. In the serial case, the ESO vector \(v\) reduces to the usual block Lipschitz constants.

| Regime | Sampling and update pattern | Representative source |
|---|---|---|
| Serial coordinate/block | One coordinate or block per iteration; partial-gradient or proximal step | [2108.07414], [1407.0107] |
| Arbitrary or importance sampling | Random subset \(\hat S\) with marginals \(p_i\); ESO-based analysis | [1412.8060] |
| Online or stochastic block sampling | Random sample or mini-batch index and random coordinate block in the same iteration | [1407.0107] |
| Feasibility-preserving constrained sampling | Random pair or path of \(\tau\ge 2\) blocks so that updates remain feasible | [1504.06340], [1709.00982] |

Importance sampling follows naturally from this geometry. In ALPHA’s serial smooth setting, the bound suggests the optimal but generally impractical sampling law
\[
p_i^*=\frac{(L_i\|x_*^i-x_0^i\|_i^2)^{1/3}}{\sum_{j=1}^n (L_j\|x_*^j-x_0^j\|_j^2)^{1/3}},
\]
which shows that both curvature and distance-to-solution information matter [1412.8060]. This suggests that coordinate descent is best understood not as a single algorithm, but as a sampling-and-geometry design space.

## 3. Convergence theory in convex, strongly convex, and nonconvex regimes

For smooth convex problems, randomized coordinate methods admit the same qualitative rate hierarchy as full-gradient methods. ALPHA yields \(O(1/k)\) convergence in the nonaccelerated case and \(O(1/k^2)\) convergence in the accelerated case under arbitrary sampling, recovering gradient descent, coordinate descent, parallel coordinate descent, and accelerated coordinate descent as special cases [1412.8060]. In block-structured strongly convex settings, linear convergence follows from standard coordinate smoothness and strong convexity assumptions; for quadratic SPD systems, classical randomized coordinate descent has the rate
\[
\mathbb E\|x^k-x^*\|_A^2 \le \left(1-\frac{\lambda_{\min}^+(A)}{\operatorname{tr}(A)}\right)^k \|x^0-x^*\|_A^2,
\]
which makes explicit the dependence on the spectrum of the coordinate geometry [2506.09394].

A central refinement concerns heterogeneous blocks. Alternating Randomized Block Coordinate Descent updates one randomly sampled block among \(1,\dots,n-1\) by a block-gradient step and then exactly minimizes over a distinguished block \(n\). Its rates depend only on \(L_1,\dots,L_{n-1}\) and are independent of the least smooth block \(L_n\), provided that block can be optimized exactly; the accelerated version AAR-BCD achieves the corresponding \(O(1/k^2)\) behavior [1805.09185]. This shows that the special role usually associated with two-block alternating minimization is not limited to the case \(n=2\).

For nonconvex problems, the modern theory centers on stationarity rather than objective optimality. In the Bregman-relative-smoothness setting, randomized Bregman coordinate descent defines a full Bregman proximal map \(T(x)\) and proves the exact characterization
\[
x \text{ is stationary } \iff D_H(T(x),x)=0.
\]
Under blockwise relative smoothness, every limit point is stationary, the expected stationarity measure satisfies
\[
\min_{0\le l\le k}\mathbb E[D_H(T(x^l),x^l)] \le \frac{n}{k+1}(F(x^0)-F^*),
\]
and, when \(H\) is strongly convex in Euclidean norm, an \(\varepsilon\)-stationary point can be found in \(O(n\varepsilon^{-2})\) iterations; the convex rate improves to \(O(n\varepsilon^{-1})\), and relative strong convexity yields global linear convergence [2001.05202]. For structured nonsmooth nonconvex DC objectives \(f+\phi-h\), randomized coordinate subgradient descent achieves an \(O(1/K)\) rate in the expected squared norm of a block composite subgradient mapping, while all limit points are critical [1909.00918].

Strict-saddle behavior has also been analyzed. Randomized coordinate gradient descent with random coordinate choices and random stepsizes almost surely does not converge to strict saddle points; if all saddle points are strict and critical points are isolated, the iterates converge almost surely to local minima [2101.01323]. A later fixed-stepsize analysis establishes the same qualitative strict-saddle escape statement through a center-stable manifold argument [2508.07535]. This suggests that the saddle-avoidance picture known for full gradient descent extends to genuinely randomized coordinate dynamics.

## 4. Beyond the Euclidean Lipschitz model

The classical Euclidean model assumes coordinate-wise Lipschitz partial gradients. A major extension replaces that assumption by Bregman geometry and relative smoothness. In the composite problem
\[
F(x)=f(x)+r(x),\qquad r(x)=\sum_{i=1}^n r_i(x_i),
\]
the randomized Bregman coordinate descent update is
\[
T_i(x)=\arg\min_{u_i}\ \langle \nabla_i f(x),u_i-x_i\rangle + \frac{1}{\alpha}D_{h_i}(u_i,x_i)+r_i(u_i),
\]
where \(D_{h_i}\) is a block Bregman distance [2001.05202]. If one chooses the Euclidean reference \(h_i(x_i)=\frac12\|x_i\|^2\), then \(D_{h_i}(u_i,x_i)=\frac12\|u_i-x_i\|^2\), relative smoothness becomes ordinary coordinate-wise Lipschitz continuity, and the method reduces exactly to proximal randomized coordinate descent; when \(r\equiv 0\), it reduces further to standard randomized coordinate gradient descent [2001.05202].

The gain is that coordinate updates remain analyzable in models where partial gradients are not globally Lipschitz. The paper highlights Poisson inverse problems, KL-type objectives, matrix and tensor factorization, and relative-entropy regression as motivating examples, using non-Euclidean references such as Burg entropy and Boltzmann–Shannon entropy [2001.05202]. This directly contradicts the common assumption that coordinate descent is intrinsically tied to Euclidean quadratic upper models.

A still broader generalization transports the coordinate idea to measure space. Random Wasserstein Coordinate Descent updates a probability measure by restricting the Wasserstein gradient to one ambient axis,
\[
\mu_{k+1}=\bigl(\operatorname{Id}-\gamma_{i_k}U_{i_k}\nabla_{\mathbb W}E[\mu_k]\bigr)_\#\mu_k,
\]
with coordinatewise smoothness constants \(L_i\) and total curvature \(L_{\rm sum}=\sum_i L_i\) [2604.01606]. The resulting complexity mirrors the Euclidean theory: nonconvex and geodesically convex regimes have \(O(L_{\rm sum}/\varepsilon)\) complexity, while PL or geodesically strongly convex regimes yield \(O((L_{\rm sum}/m)\log(1/\varepsilon))\) convergence [2604.01606]. This suggests that “coordinate descent” can be abstracted from vector coordinates to geometry-adapted directions in nonlinear spaces.

A different structural extension enriches the direction set itself. Stochastic Spectral and Conjugate Descent shows that merely changing coordinate sampling probabilities is fundamentally limited on SPD quadratics, whereas augmenting coordinates with a few spectral or conjugate directions can interpolate between the classical randomized coordinate rate and a condition-number-independent \(O(n\log(1/\varepsilon))\) regime [1802.03703]. The paper’s SSCD scheme removes the influence of the smallest eigenvalues by sampling both coordinates and selected eigendirections. This is no longer pure coordinate descent, but it arises directly from an analysis of what classical randomized coordinate gradient descent cannot do by reweighting coordinates alone.

## 5. Stochastic, online, constrained, and linear-system variants

When both the number of samples and the number of features are large, coordinate randomization can be combined with stochastic sampling over data. ORBCD samples a mini-batch or data index \(i_t\), then samples a coordinate block \(j_t\), and performs a proximal step using only \(\nabla_{j_t} f_{i_t}(x^t)\) [1407.0107]. In the convex stochastic case it achieves an \(O(1/\sqrt{T})\) rate, in the strongly convex case an \(O((\log T)/T)\) rate, and with SVRG-style variance reduction—ORBCDVD—it converges geometrically in expectation for strongly convex objectives [1407.0107]. This places randomized coordinate gradient descent squarely at the intersection of SGD and block-coordinate proximal methods.

Linear coupling constraints require a different update geometry, because a single-block move typically breaks feasibility. For network problems with the constraint \(\sum_i x_i=0\), randomized block coordinate descent therefore updates \(\tau\ge 2\) variables at a time along a sampled path or edge, maintaining the zero-sum constraint exactly [1504.06340]. The expected complexity is \(O(N/(\tau\varepsilon))\) for smooth convex objectives, with linear convergence in the strongly convex case [1504.06340]. A later refinement for the two-block case sharpened the expected rate by extending Nesterov’s second technique to the linearly constrained setting and by constructing a basis for the feasible subspace [1709.00982].

Linear systems and least squares provide another major specialization. For overdetermined inconsistent systems, randomized coordinate descent on the columns of \(X\) converges linearly to the least-squares solution, while randomized Kaczmarz on rows does not; for underdetermined consistent systems, the preference reverses if one wants the minimum-norm solution [1406.5295]. More recent work modifies classical coordinate descent rather than the problem class. Subspace-constrained randomized coordinate descent restricts the iterates to an affine space induced by a Nyström approximation and obtains a rate controlled by the tail spectrum \(A^\circ=A-A\langle\mathcal S\rangle\), thereby avoiding degradation from large spectral outliers [2506.09394]. Randomized conjugate gradient least squares goes further by replacing the full gradient in a CGLS-type direction-correction scheme with a randomized coordinate gradient \(S_kS_k^\top \nabla f(x^k)\); the resulting RCGLS method has a linear convergence bound at least as good as standard randomized coordinate descent and admits a lightweight sparse implementation [2605.25034].

These developments show that “randomized coordinate gradient descent” is not a single update rule but a design principle: choose a low-cost partial first-order surrogate, preserve whatever structure the problem requires, and analyze the resulting dynamics in the natural geometry of the model.

## 6. Applications, statistical interpretation, and common misconceptions

The application range is broad because many modern objectives make full gradients expensive or unnatural. In parameterized quantum circuits, each partial derivative is typically obtained by a separate parameter-shift measurement, so the full gradient costs roughly \(d\) times one derivative evaluation; under a local PL condition and bounded unbiased noise, random coordinate descent has the same stochastic stability order as gradient descent and can be much cheaper in total partial-derivative cost when \(L\gg L_{\mathrm{avg}}\) [2311.00088]. The reported experiments include nearly \(4\times\) fewer partial derivative evaluations than gradient descent on a transverse-field Ising VQE instance and about \(3\times\) fewer on a distinct \(3\)-qubit, \(18\)-parameter ansatz [2311.00088].

Boosting provides a different reinterpretation. Randomized Gradient Boosting Machine views weak-learner coefficients as coordinates in a convex optimization problem and shows that classical GBM is greedy coordinate descent, while RGBM is a random-then-greedy hybrid that samples a subset of weak learners and then chooses the largest-magnitude coordinate gradient within that subset [1810.10158]. This does not fit the plain “sample one coordinate and update it immediately” template, but it demonstrates how randomized coordinate-gradient ideas extend to dictionary search in supervised learning.

The statistical role of coordinate randomization is not limited to optimization speed. Stability and Generalization for Randomized Coordinate Descent proves that, in convex empirical-risk minimization, the \(\ell_1\) on-average argument stability scales as
\[
O\!\left(\frac{1}{nd}\sum_t \eta_t\right),
\]
and in the strongly convex case as \(O(1/(n\sigma))\), leading to \(O(1/\sqrt n)\) and \(O(1/(n\sigma))\) excess-risk bounds with suitable early stopping [2108.07414]. The same work argues that RCD is more stable than SGD because only one coordinate is perturbed at each step [2108.07414]. This reframes randomized coordinate gradient descent as a learning algorithm with distinctive generalization behavior, not merely a computational primitive.

Several recurrent misconceptions are clarified by the literature. First, randomized coordinate descent is not inherently derivative-free: in ORBCD, ALPHA, and the stability analysis, it is explicitly a partial-gradient or partial proximal-gradient method [1407.0107, 1412.8060, 2108.07414]. Second, lower per-iteration cost does not by itself imply lower total complexity. In Langevin Monte Carlo, blindly replacing the full gradient by a one-coordinate unbiased estimator produces no computational gain, because the variance increases the required number of iterations; only after incorporating variance reduction through SAGA or SVRG does the coordinate strategy improve total cost, and in the underdamped case it can match the full-gradient iteration count while using one directional derivative per step [2007.14209]. Third, the usual coordinate-wise Lipschitz assumption is not the only viable analytical framework: Bregman relative smoothness and Wasserstein coordinate smoothness show that coordinate methods can be formulated in genuinely non-Euclidean geometries [2001.05202, 2604.01606].

Taken together, these results depict randomized coordinate gradient descent as a large methodological family unified by one principle: replace a full first-order update by a randomized partial one that is cheaper, structurally compatible with the model, and analyzable in an appropriate geometry. Its modern theory spans convex and nonconvex optimization, exact and proximal updates, Euclidean and Bregman geometries, stochastic and online learning, constrained network problems, linear solvers, boosting, quantum optimization, and even Wasserstein spaces.

Source: https://www.emergentmind.com/topics/randomized-coordinate-gradient-descent