Riemannian Block Coordinate Descent

• RBCD is a family of optimization methods that update manifold block variables one at a time while maintaining feasibility via retractions or the exponential map.
• It encompasses formulations based on product manifolds, tangent subspaces, and matrix-manifold coordinates for diverse applications.
• RBCD offers computational efficiency and provable convergence in tasks ranging from robust optimal transport to essential matrix estimation under manifold-specific conditions.

Riemannian Block Coordinate Descent (RBCD) denotes a family of block-wise optimization methods on Riemannian manifolds in which one updates one block variable, one tangent-space coordinate or block, or one tangent subspace at a time while maintaining feasibility through the exponential map or a retraction. In the recent literature, the term covers at least three closely related formulations: block coordinate descent on product manifolds $M=M_1\times\cdots\times M_b$, tangent-subspace methods that replace Euclidean coordinate blocks by subspaces $S_k\subseteq T_xM$, and coordinate descent algorithms on matrix manifolds built from manifold-specific tangent bases. The same acronym also names a specific fast, scalable algorithm for entropy-regularized Projection Robust Wasserstein distance on the Stiefel manifold (Peng et al., 2023, Gutman et al., 2019, Han et al., 2024, Huang et al., 2020).

1. Problem formulations and basic definitions

A standard product-manifold formulation considers

$$\min_{x\in M} F(x), \qquad M:=M_1\times\cdots\times M_b,$$

where each block $x_i\in M_i\subset \mathbb{R}^{n_i}$ and each $M_i$ is a smooth embedded submanifold in the convergence-rate setting. The Euclidean partial gradient $\nabla_iF(x)$ is projected onto $T_{x_i}M_i$ to obtain the Riemannian partial gradient $\widetilde{\nabla}_iF(x)$, and the full Riemannian gradient is $$\widetilde{\nabla}F(x)=[\widetilde{\nabla}_1F(x);\dots;\widetilde{\nabla}_bF(x)]$$. For general closed sets, stationarity is expressed by the tangent cone condition

$$\langle \nabla F(x),v\rangle \ge 0 \quad \forall v\in T_M(x),$$

which reduces to $S_k\subseteq T_xM$0 on smooth manifolds (Peng et al., 2023).

A second formulation, introduced as Tangent Subspace Descent (TSD), treats Euclidean coordinate blocks as tangent subspaces of a manifold. At iterate $S_k\subseteq T_xM$1, one selects a tangent subspace $S_k\subseteq T_xM$2 with orthogonal projection $S_k\subseteq T_xM$3, and updates by

$S_k\subseteq T_xM$4

or, more generally, with a retraction $S_k\subseteq T_xM$5,

$S_k\subseteq T_xM$6

In this formulation, RBCD is not tied to a product decomposition of variables; the “blocks” are tangent subspaces selected at the current iterate (Gutman et al., 2019).

A third formulation specializes to matrix manifolds. There, coordinates are chosen with respect to a basis $S_k\subseteq T_xM$7 spanning $S_k\subseteq T_xM$8, not necessarily orthonormal, and a block is a structured subset of entries or a tangent subspace. The generic update is

$S_k\subseteq T_xM$9

with $$\min_{x\in M} F(x), \qquad M:=M_1\times\cdots\times M_b,$$0. A central identity is

$$\min_{x\in M} F(x), \qquad M:=M_1\times\cdots\times M_b,$$1

which permits coordinate derivatives to be computed directly from the Euclidean gradient (Han et al., 2024).

When $$\min_{x\in M} F(x), \qquad M:=M_1\times\cdots\times M_b,$$2, these constructions reduce to classical Euclidean block coordinate descent: tangent spaces become the ambient space, $$\min_{x\in M} F(x), \qquad M:=M_1\times\cdots\times M_b,$$3, and the update becomes $$\min_{x\in M} F(x), \qquad M:=M_1\times\cdots\times M_b,$$4 (Gutman et al., 2019, Han et al., 2024).

2. Update rules and algorithmic families

The product-manifold literature distinguishes three principal block updates. Block Exact Minimization (BEM) updates block $$\min_{x\in M} F(x), \qquad M:=M_1\times\cdots\times M_b,$$5 by solving

$$\min_{x\in M} F(x), \qquad M:=M_1\times\cdots\times M_b,$$6

Block Majorization–Minimization (BMM) replaces the block objective by a block-$$\min_{x\in M} F(x), \qquad M:=M_1\times\cdots\times M_b,$$7 majorizer $$\min_{x\in M} F(x), \qquad M:=M_1\times\cdots\times M_b,$$8 and minimizes that surrogate. Block Riemannian Gradient Descent (BRGD) uses

$$\min_{x\in M} F(x), \qquad M:=M_1\times\cdots\times M_b,$$9

A blended scheme applies BRGD on some blocks and exact minimization on others (Peng et al., 2023).

The constrained block-Riemannian optimization literature places these methods in a block majorization-minimization template. At iteration $x_i\in M_i\subset \mathbb{R}^{n_i}$0, a block $x_i\in M_i\subset \mathbb{R}^{n_i}$1 is chosen, a surrogate $x_i\in M_i\subset \mathbb{R}^{n_i}$2 is built so that $x_i\in M_i\subset \mathbb{R}^{n_i}$3 and $x_i\in M_i\subset \mathbb{R}^{n_i}$4, and the update minimizes the surrogate over the feasible block set. A first-order quadratic tangent-space surrogate recovers RBCD as a prox-linear special case (Li et al., 2023).

On matrix manifolds, cyclic, random, and Gauss–Southwell selection rules are natural. A fixed-order cyclic sweep updates blocks $x_i\in M_i\subset \mathbb{R}^{n_i}$5, a random rule samples a block uniformly, and a Gauss–Southwell rule chooses the block maximizing $x_i\in M_i\subset \mathbb{R}^{n_i}$6. The same literature also introduces a first-order approximation variant, RCDlin, which anchors the Euclidean gradient at $x_i\in M_i\subset \mathbb{R}^{n_i}$7, performs $x_i\in M_i\subset \mathbb{R}^{n_i}$8 inner block updates with that fixed gradient, and evaluates $x_i\in M_i\subset \mathbb{R}^{n_i}$9 only once per outer iteration (Han et al., 2024).

Family	Representative update	Setting
BEM	exact minimization over one block	product manifolds
BMM	minimization of a block majorizer $M_i$0 or $M_i$1	constrained block-Riemannian optimization
BRGD	$M_i$2	smooth embedded product manifolds
TSD	$M_i$3	tangent-subspace blocks
RCDlin	$M_i$4 with anchored $M_i$5	matrix manifolds

The distinction among these schemes is substantive. In BEM and BMM, the block step may be a small exact or surrogate minimization. In BRGD and TSD, the step is explicitly first order. In matrix-manifold RCD, the blocks are often chosen to exploit closed-form retractions with very low per-update cost (Peng et al., 2023, Li et al., 2023, Gutman et al., 2019, Han et al., 2024).

3. Geometry of blocks on matrix and manifold domains

The Stiefel manifold,

$M_i$6

is central in RBCD. Its tangent space is

$M_i$7

and the Euclidean-metric projection is

$M_i$8

Common retractions include the QR retraction and the polar retraction. For matrix-manifold coordinate descent on Stiefel, basis directions of the form $M_i$9, with $\nabla_iF(x)$0, yield Givens-rotation updates

$\nabla_iF(x)$1

which modify only rows $\nabla_iF(x)$2 and $\nabla_iF(x)$3 and cost $\nabla_iF(x)$4 per update (Huang et al., 2020, Han et al., 2024).

The Grassmann manifold uses the horizontal tangent representation

$\nabla_iF(x)$5

The same Givens-style update is valid, and the matrix-manifold literature proves that it commutes with right actions, making it well defined on equivalence classes $\nabla_iF(x)$6 (Han et al., 2024).

TSD constructs blocks directly in tangent spaces. On $\nabla_iF(x)$7, the tangent space is $\nabla_iF(x)$8, and block subspaces are induced by partitions of an orthonormal skew basis into sets $\nabla_iF(x)$9. Under a “disjoint pairs” assumption, the exponential of the block generator decomposes into independent $T_{x_i}M_i$0 Givens blocks. On $T_{x_i}M_i$1, the randomized rule uses one-dimensional subspaces spanning skew and normal components, with sampled directions $T_{x_i}M_i$2 or $T_{x_i}M_i$3, where $T_{x_i}M_i$4 is sampled uniformly from the unit sphere (Gutman et al., 2019).

The matrix-manifold framework extends the same principle to other geometries. For fixed-rank SPSD matrices represented as $T_{x_i}M_i$5, the tangent space at $T_{x_i}M_i$6 is Euclidean, the gradient is $T_{x_i}M_i$7, and single-entry updates

$T_{x_i}M_i$8

have $T_{x_i}M_i$9 update cost. For SPD matrices with the Bures–Wasserstein metric, coordinate steps use basis directions $\widetilde{\nabla}_iF(x)$0 and alter only two rows and columns. For generalized hyperbolic manifolds, symplectic manifolds, doubly stochastic manifolds, and multinomial manifolds, the literature supplies manifold-specific bases, closed-form coordinate derivatives, and retractions such as Lorentz boosts, symplectic exponentials, coordinate Sinkhorn, and row normalization (Han et al., 2024).

A recurrent structural fact is that the block is defined in the tangent geometry, not merely in ambient coordinates. This is why apparently similar “coordinate descent” constructions can differ materially across manifolds.

4. Assumptions, convergence guarantees, and complexity regimes

The TSD analysis assumes a geodesically complete Riemannian manifold, smoothness in the form

$\widetilde{\nabla}_iF(x)$1

and either the deterministic $\widetilde{\nabla}_iF(x)$2-gap ensuring condition or the randomized $\widetilde{\nabla}_iF(x)$3-randomized norm condition. Under block smoothness and $\widetilde{\nabla}_iF(x)$4-gap ensuring, the deterministic multi-block method satisfies $\widetilde{\nabla}_iF(x)$5, any limit point is stationary, the stationarity rate is $\widetilde{\nabla}_iF(x)$6, and for $\widetilde{\nabla}_iF(x)$7-convex objectives the function-value rate is $\widetilde{\nabla}_iF(x)$8. Under the $\widetilde{\nabla}_iF(x)$9-randomized norm condition, the randomized single-block method yields expected stationarity

$$\widetilde{\nabla}F(x)=[\widetilde{\nabla}_1F(x);\dots;\widetilde{\nabla}_bF(x)]$$0

almost sure convergence of $$\widetilde{\nabla}F(x)=[\widetilde{\nabla}_1F(x);\dots;\widetilde{\nabla}_bF(x)]$$1 to zero, and an $$\widetilde{\nabla}F(x)=[\widetilde{\nabla}_1F(x);\dots;\widetilde{\nabla}_bF(x)]$$2 expected function-value bound under $$\widetilde{\nabla}F(x)=[\widetilde{\nabla}_1F(x);\dots;\widetilde{\nabla}_bF(x)]$$3-convexity (Gutman et al., 2019).

The product-manifold BRGD analysis assumes compact smooth submanifolds, block-$$\widetilde{\nabla}F(x)=[\widetilde{\nabla}_1F(x);\dots;\widetilde{\nabla}_bF(x)]$$4 Lipschitz smoothness of the Euclidean partial gradients, and smooth retractions with blockwise descent lemmas. With cyclic BRGD and stepsizes $$\widetilde{\nabla}F(x)=[\widetilde{\nabla}_1F(x);\dots;\widetilde{\nabla}_bF(x)]$$5, the theory gives

$$\widetilde{\nabla}F(x)=[\widetilde{\nabla}_1F(x);\dots;\widetilde{\nabla}_bF(x)]$$6

hence iteration complexity $$\widetilde{\nabla}F(x)=[\widetilde{\nabla}_1F(x);\dots;\widetilde{\nabla}_bF(x)]$$7 to reach $$\widetilde{\nabla}F(x)=[\widetilde{\nabla}_1F(x);\dots;\widetilde{\nabla}_bF(x)]$$8-stationarity. The blended BEM+RGD scheme admits an analogous bound with $$\widetilde{\nabla}F(x)=[\widetilde{\nabla}_1F(x);\dots;\widetilde{\nabla}_bF(x)]$$9 gradient-updated blocks (Peng et al., 2023).

The RBMM analysis addresses smooth nonconvex objectives over $$\langle \nabla F(x),v\rangle \ge 0 \quad \forall v\in T_M(x),$$0, with geodesic smoothness, compact sublevel sets, a uniform injectivity radius bound, and either $$\langle \nabla F(x),v\rangle \ge 0 \quad \forall v\in T_M(x),$$1-smooth, Riemannian proximal, or Euclidean proximal surrogates. Every limit point is stationary under the stated assumptions. For proximal surrogates, the worst-case iteration complexity for the paper’s $$\langle \nabla F(x),v\rangle \ge 0 \quad \forall v\in T_M(x),$$2-stationarity measure is $$\langle \nabla F(x),v\rangle \ge 0 \quad \forall v\in T_M(x),$$3; for $$\langle \nabla F(x),v\rangle \ge 0 \quad \forall v\in T_M(x),$$4-smooth surrogates, the general complexity is $$\langle \nabla F(x),v\rangle \ge 0 \quad \forall v\in T_M(x),$$5, improved to $$\langle \nabla F(x),v\rangle \ge 0 \quad \forall v\in T_M(x),$$6 under an upper quadratic surrogate-gap bound. On products of Euclidean and Stiefel manifolds, the assumptions become completely Euclidean while the analysis remains Riemannian (Li et al., 2023).

The matrix-manifold RCD and RCDlin analyses assume a compact neighborhood around a critical point, bounded basis and projection constants, and retraction $$\langle \nabla F(x),v\rangle \ge 0 \quad \forall v\in T_M(x),$$7-smoothness. Randomized RCD with one inner step and $$\langle \nabla F(x),v\rangle \ge 0 \quad \forall v\in T_M(x),$$8 satisfies

$$\langle \nabla F(x),v\rangle \ge 0 \quad \forall v\in T_M(x),$$9

whereas cyclic RCD yields

$S_k\subseteq T_xM$00

For RCDlin, the randomized rate becomes $S_k\subseteq T_xM$01 under a positive-correlation condition between anchored and true block derivatives (Han et al., 2024).

A distinct convergence template appears in non-orthogonal joint approximate diagonalization on $S_k\subseteq T_xM$02. There, gradient-based BCD-G algorithms choose the block according to the Riemannian gradient magnitude, use Wolfe line search on the Stiefel block, and use structured elementary transformations on the special linear block. Under bounded iterates, the whole sequence converges and every limit point is stationary, with the proof based on sufficient descent and the Łojasiewicz gradient inequality (Li et al., 2020).

These results are not interchangeable. Their rates, stationarity measures, and assumptions depend on whether blocks are product components, tangent subspaces, or manifold-specific coordinates.

5. The Projection Robust Wasserstein instance

A prominent specialized use of the acronym is the RBCD algorithm for computing the Projection Robust Wasserstein (PRW) distance. For empirical measures $S_k\subseteq T_xM$03 and $S_k\subseteq T_xM$04, the squared PRW distance is

$S_k\subseteq T_xM$05

The entropy-regularized version introduces

$S_k\subseteq T_xM$06

and maximizes over $S_k\subseteq T_xM$07 the inner regularized OT value. The key reformulation dualizes the inner RegOT and obtains a smooth minimization problem over two Euclidean blocks $S_k\subseteq T_xM$08 and one Stiefel block $S_k\subseteq T_xM$09: $S_k\subseteq T_xM$10 where

$S_k\subseteq T_xM$11

This removes the need to solve an entropy-regularized optimal transport problem in each iteration (Huang et al., 2020).

The resulting RBCD iteration has three blocks. The $S_k\subseteq T_xM$12- and $S_k\subseteq T_xM$13-updates are closed-form marginal scaling steps,

$S_k\subseteq T_xM$14

$S_k\subseteq T_xM$15

which are exactly the row and column scaling steps of Sinkhorn applied analytically to $S_k\subseteq T_xM$16. The $S_k\subseteq T_xM$17-block performs one Riemannian gradient step with retraction: $S_k\subseteq T_xM$18 with

$S_k\subseteq T_xM$19

Feasibility of the transport matrix is enforced at the end by a rounding procedure that corrects marginals and yields $S_k\subseteq T_xM$20.

The theoretical gain is substantial. The arithmetic complexity to obtain an $S_k\subseteq T_xM$21-stationary point is

$S_k\subseteq T_xM$22

and in the common case $S_k\subseteq T_xM$23 and $S_k\subseteq T_xM$24, this becomes $S_k\subseteq T_xM$25. The comparison paper reports $S_k\subseteq T_xM$26 complexity for RGAS, which solves a Sinkhorn subproblem to high accuracy at every iteration. Per iteration, RBCD costs $S_k\subseteq T_xM$27, with $S_k\subseteq T_xM$28- and $S_k\subseteq T_xM$29-updates of $S_k\subseteq T_xM$30, $S_k\subseteq T_xM$31 computed in $S_k\subseteq T_xM$32 without forming $S_k\subseteq T_xM$33, and QR or polar retraction in $S_k\subseteq T_xM$34.

The experiments cover synthetic fragmented hypercube and Gaussian models and real data including movie scripts, Shakespeare plays, and MNIST. On the fragmented hypercube with $S_k\subseteq T_xM$35, the reported runtimes include $S_k\subseteq T_xM$36 s, $S_k\subseteq T_xM$37 s, and $S_k\subseteq T_xM$38 s for RBCD at $S_k\subseteq T_xM$39, versus $S_k\subseteq T_xM$40 s, $S_k\subseteq T_xM$41 s, and $S_k\subseteq T_xM$42 s for RGAS. For large scale $S_k\subseteq T_xM$43, $S_k\subseteq T_xM$44, the reported runtime is $S_k\subseteq T_xM$45 s for RBCD versus $S_k\subseteq T_xM$46 s for RGAS. The paper also reports similar PRW values across methods, robustness to white noise in the Gaussian setting, and an adaptive variant, RABCD, with empirically faster convergence and the same order complexity up to a $S_k\subseteq T_xM$47 factor (Huang et al., 2020).

6. Applications, relations, and limitations

RBCD appears in a broad range of nonconvex manifold problems. The product-manifold framework covers low-dimensional structure problems such as maximal coding rate reduction, neural collapse, generalized PCA, and alternating projection; combinatorial structure problems such as homomorphic sensing, regression without correspondences, real phase retrieval, and robust point matching; geometric-vision problems such as essential matrix estimation and absolute pose estimation; and outlier-robust estimation via iteratively-reweighted least squares. The same theory recovers previously known results for optimal transport, matrix factorization, and Burer–Monteiro factorization, and yields explicit corollaries for GPCA, essential matrix estimation, PRWD, IRLS, and block-diagonal SDP factorizations (Peng et al., 2023).

The matrix-manifold coordinate literature emphasizes computational efficiency in applications such as orthogonal Procrustes on Stiefel, PCA on Grassmann, orthogonal deep network distillation, nearest matrix on the symplectic manifold, Lorentz embeddings, and weighted least squares on SPD or SPSD manifolds. Its practical message is that row-pair rotations, single-entry factor updates, and $S_k\subseteq T_xM$48 coordinate Sinkhorn updates can be markedly cheaper than full Riemannian gradient or trust-region steps, especially when retraction costs dominate (Han et al., 2024).

In non-orthogonal joint approximate diagonalization, RBCD operates on the product of the complex Stiefel manifold and the special linear group. One block uses Riemannian line-search descent on the Stiefel factor, and the other uses plane Givens, triangular, or diagonal transformations on the special linear factor, producing BCD-GLU, BCD-GQU, and BCD-GU. The square-case Jacobi analogues on $S_k\subseteq T_xM$49 are Jacobi-GLU and Jacobi-GQU. Under bounded iterates, the whole sequence converges and the gradient vanishes (Li et al., 2020).

Several limitations recur across the literature. First, nonconvexity remains intrinsic: the general guarantees are to stationary points or $S_k\subseteq T_xM$50-stationary points, not to global minimizers. Second, block or subspace selection matters. TSD gives counterexamples showing that poorly chosen subspaces can prevent convergence even if they span the tangent space at each step; the deterministic gap ensuring and randomized $S_k\subseteq T_xM$51-randomized norm conditions were introduced precisely to exclude such failures (Gutman et al., 2019). Third, low per-iteration cost does not remove all large-scale bottlenecks. In PRW, the $S_k\subseteq T_xM$52 dependence persists even though the method avoids solving a full Sinkhorn problem at every iteration, and the performance remains sensitive to $S_k\subseteq T_xM$53 and $S_k\subseteq T_xM$54 (Huang et al., 2020). Fourth, different RBCD papers use different block notions—product blocks, tangent subspaces, or manifold-specific coordinates—so results from one setting should not be transferred mechanically to another. A plausible implication is that “RBCD” is best understood as a methodological class rather than a single canonical algorithm.

Across these formulations, the common design principle is stable: restrict the search to a geometrically meaningful block, compute a blockwise Riemannian descent direction, and return to the manifold by $S_k\subseteq T_xM$55 or a retraction. The differences lie in how the blocks are defined, which assumptions are used to prove descent, and how aggressively the algorithm exploits manifold structure for closed-form or low-cost updates.