Riemannian Block Coordinate Descent
- RBCD is a family of optimization methods that update manifold block variables one at a time while maintaining feasibility via retractions or the exponential map.
- It encompasses formulations based on product manifolds, tangent subspaces, and matrix-manifold coordinates for diverse applications.
- RBCD offers computational efficiency and provable convergence in tasks ranging from robust optimal transport to essential matrix estimation under manifold-specific conditions.
Riemannian Block Coordinate Descent (RBCD) denotes a family of block-wise optimization methods on Riemannian manifolds in which one updates one block variable, one tangent-space coordinate or block, or one tangent subspace at a time while maintaining feasibility through the exponential map or a retraction. In the recent literature, the term covers at least three closely related formulations: block coordinate descent on product manifolds , tangent-subspace methods that replace Euclidean coordinate blocks by subspaces , and coordinate descent algorithms on matrix manifolds built from manifold-specific tangent bases. The same acronym also names a specific fast, scalable algorithm for entropy-regularized Projection Robust Wasserstein distance on the Stiefel manifold (Peng et al., 2023, Gutman et al., 2019, Han et al., 2024, Huang et al., 2020).
1. Problem formulations and basic definitions
A standard product-manifold formulation considers
where each block and each is a smooth embedded submanifold in the convergence-rate setting. The Euclidean partial gradient is projected onto to obtain the Riemannian partial gradient , and the full Riemannian gradient is . For general closed sets, stationarity is expressed by the tangent cone condition
which reduces to on smooth manifolds (Peng et al., 2023).
A second formulation, introduced as Tangent Subspace Descent (TSD), treats Euclidean coordinate blocks as tangent subspaces of a manifold. At iterate , one selects a tangent subspace with orthogonal projection , and updates by
$x_{k+1}=\Exp_{x_k}\big(-\alpha_k P_{S_k}\nabla f(x_k)\big),$
or, more generally, with a retraction ,
In this formulation, RBCD is not tied to a product decomposition of variables; the “blocks” are tangent subspaces selected at the current iterate (Gutman et al., 2019).
A third formulation specializes to matrix manifolds. There, coordinates are chosen with respect to a basis spanning , not necessarily orthonormal, and a block is a structured subset of entries or a tangent subspace. The generic update is
with . A central identity is
which permits coordinate derivatives to be computed directly from the Euclidean gradient (Han et al., 2024).
When , these constructions reduce to classical Euclidean block coordinate descent: tangent spaces become the ambient space, $\Exp_x(v)=x+v$, and the update becomes (Gutman et al., 2019, Han et al., 2024).
2. Update rules and algorithmic families
The product-manifold literature distinguishes three principal block updates. Block Exact Minimization (BEM) updates block by solving
Block Majorization–Minimization (BMM) replaces the block objective by a block- majorizer and minimizes that surrogate. Block Riemannian Gradient Descent (BRGD) uses
A blended scheme applies BRGD on some blocks and exact minimization on others (Peng et al., 2023).
The constrained block-Riemannian optimization literature places these methods in a block majorization-minimization template. At iteration , a block is chosen, a surrogate is built so that and , and the update minimizes the surrogate over the feasible block set. A first-order quadratic tangent-space surrogate recovers RBCD as a prox-linear special case (Li et al., 2023).
On matrix manifolds, cyclic, random, and Gauss–Southwell selection rules are natural. A fixed-order cyclic sweep updates blocks , a random rule samples a block uniformly, and a Gauss–Southwell rule chooses the block maximizing . The same literature also introduces a first-order approximation variant, RCDlin, which anchors the Euclidean gradient at , performs inner block updates with that fixed gradient, and evaluates only once per outer iteration (Han et al., 2024).
| Family | Representative update | Setting |
|---|---|---|
| BEM | exact minimization over one block | product manifolds |
| BMM | minimization of a block majorizer or | constrained block-Riemannian optimization |
| BRGD | smooth embedded product manifolds | |
| TSD | tangent-subspace blocks | |
| RCDlin | with anchored | matrix manifolds |
The distinction among these schemes is substantive. In BEM and BMM, the block step may be a small exact or surrogate minimization. In BRGD and TSD, the step is explicitly first order. In matrix-manifold RCD, the blocks are often chosen to exploit closed-form retractions with very low per-update cost (Peng et al., 2023, Li et al., 2023, Gutman et al., 2019, Han et al., 2024).
3. Geometry of blocks on matrix and manifold domains
The Stiefel manifold,
is central in RBCD. Its tangent space is
and the Euclidean-metric projection is
Common retractions include the QR retraction and the polar retraction. For matrix-manifold coordinate descent on Stiefel, basis directions of the form , with , yield Givens-rotation updates
which modify only rows and and cost per update (Huang et al., 2020, Han et al., 2024).
The Grassmann manifold uses the horizontal tangent representation
The same Givens-style update is valid, and the matrix-manifold literature proves that it commutes with right actions, making it well defined on equivalence classes (Han et al., 2024).
TSD constructs blocks directly in tangent spaces. On , the tangent space is , and block subspaces are induced by partitions of an orthonormal skew basis into sets . Under a “disjoint pairs” assumption, the exponential of the block generator decomposes into independent Givens blocks. On , the randomized rule uses one-dimensional subspaces spanning skew and normal components, with sampled directions or , where is sampled uniformly from the unit sphere (Gutman et al., 2019).
The matrix-manifold framework extends the same principle to other geometries. For fixed-rank SPSD matrices represented as , the tangent space at is Euclidean, the gradient is , and single-entry updates
have update cost. For SPD matrices with the Bures–Wasserstein metric, coordinate steps use basis directions and alter only two rows and columns. For generalized hyperbolic manifolds, symplectic manifolds, doubly stochastic manifolds, and multinomial manifolds, the literature supplies manifold-specific bases, closed-form coordinate derivatives, and retractions such as Lorentz boosts, symplectic exponentials, coordinate Sinkhorn, and row normalization (Han et al., 2024).
A recurrent structural fact is that the block is defined in the tangent geometry, not merely in ambient coordinates. This is why apparently similar “coordinate descent” constructions can differ materially across manifolds.
4. Assumptions, convergence guarantees, and complexity regimes
The TSD analysis assumes a geodesically complete Riemannian manifold, smoothness in the form
and either the deterministic -gap ensuring condition or the randomized -randomized norm condition. Under block smoothness and -gap ensuring, the deterministic multi-block method satisfies , any limit point is stationary, the stationarity rate is , and for -convex objectives the function-value rate is . Under the -randomized norm condition, the randomized single-block method yields expected stationarity
almost sure convergence of to zero, and an expected function-value bound under -convexity (Gutman et al., 2019).
The product-manifold BRGD analysis assumes compact smooth submanifolds, block- Lipschitz smoothness of the Euclidean partial gradients, and smooth retractions with blockwise descent lemmas. With cyclic BRGD and stepsizes , the theory gives
hence iteration complexity to reach -stationarity. The blended BEM+RGD scheme admits an analogous bound with gradient-updated blocks (Peng et al., 2023).
The RBMM analysis addresses smooth nonconvex objectives over , with geodesic smoothness, compact sublevel sets, a uniform injectivity radius bound, and either -smooth, Riemannian proximal, or Euclidean proximal surrogates. Every limit point is stationary under the stated assumptions. For proximal surrogates, the worst-case iteration complexity for the paper’s -stationarity measure is ; for -smooth surrogates, the general complexity is , improved to under an upper quadratic surrogate-gap bound. On products of Euclidean and Stiefel manifolds, the assumptions become completely Euclidean while the analysis remains Riemannian (Li et al., 2023).
The matrix-manifold RCD and RCDlin analyses assume a compact neighborhood around a critical point, bounded basis and projection constants, and retraction -smoothness. Randomized RCD with one inner step and satisfies
whereas cyclic RCD yields
For RCDlin, the randomized rate becomes under a positive-correlation condition between anchored and true block derivatives (Han et al., 2024).
A distinct convergence template appears in non-orthogonal joint approximate diagonalization on . There, gradient-based BCD-G algorithms choose the block according to the Riemannian gradient magnitude, use Wolfe line search on the Stiefel block, and use structured elementary transformations on the special linear block. Under bounded iterates, the whole sequence converges and every limit point is stationary, with the proof based on sufficient descent and the Łojasiewicz gradient inequality (Li et al., 2020).
These results are not interchangeable. Their rates, stationarity measures, and assumptions depend on whether blocks are product components, tangent subspaces, or manifold-specific coordinates.
5. The Projection Robust Wasserstein instance
A prominent specialized use of the acronym is the RBCD algorithm for computing the Projection Robust Wasserstein (PRW) distance. For empirical measures and , the squared PRW distance is
The entropy-regularized version introduces
and maximizes over the inner regularized OT value. The key reformulation dualizes the inner RegOT and obtains a smooth minimization problem over two Euclidean blocks and one Stiefel block : where
This removes the need to solve an entropy-regularized optimal transport problem in each iteration (Huang et al., 2020).
The resulting RBCD iteration has three blocks. The - and -updates are closed-form marginal scaling steps,
which are exactly the row and column scaling steps of Sinkhorn applied analytically to . The -block performs one Riemannian gradient step with retraction: with
Feasibility of the transport matrix is enforced at the end by a rounding procedure that corrects marginals and yields .
The theoretical gain is substantial. The arithmetic complexity to obtain an -stationary point is
and in the common case and , this becomes . The comparison paper reports complexity for RGAS, which solves a Sinkhorn subproblem to high accuracy at every iteration. Per iteration, RBCD costs , with - and -updates of , computed in without forming , and QR or polar retraction in .
The experiments cover synthetic fragmented hypercube and Gaussian models and real data including movie scripts, Shakespeare plays, and MNIST. On the fragmented hypercube with , the reported runtimes include $0.18$ s, $1.83$ s, and $8.51$ s for RBCD at , versus $0.40$ s, $2.90$ s, and $10.25$ s for RGAS. For large scale , , the reported runtime is $354.91$ s for RBCD versus $427.24$ s for RGAS. The paper also reports similar PRW values across methods, robustness to white noise in the Gaussian setting, and an adaptive variant, RABCD, with empirically faster convergence and the same order complexity up to a factor (Huang et al., 2020).
6. Applications, relations, and limitations
RBCD appears in a broad range of nonconvex manifold problems. The product-manifold framework covers low-dimensional structure problems such as maximal coding rate reduction, neural collapse, generalized PCA, and alternating projection; combinatorial structure problems such as homomorphic sensing, regression without correspondences, real phase retrieval, and robust point matching; geometric-vision problems such as essential matrix estimation and absolute pose estimation; and outlier-robust estimation via iteratively-reweighted least squares. The same theory recovers previously known results for optimal transport, matrix factorization, and Burer–Monteiro factorization, and yields explicit corollaries for GPCA, essential matrix estimation, PRWD, IRLS, and block-diagonal SDP factorizations (Peng et al., 2023).
The matrix-manifold coordinate literature emphasizes computational efficiency in applications such as orthogonal Procrustes on Stiefel, PCA on Grassmann, orthogonal deep network distillation, nearest matrix on the symplectic manifold, Lorentz embeddings, and weighted least squares on SPD or SPSD manifolds. Its practical message is that row-pair rotations, single-entry factor updates, and coordinate Sinkhorn updates can be markedly cheaper than full Riemannian gradient or trust-region steps, especially when retraction costs dominate (Han et al., 2024).
In non-orthogonal joint approximate diagonalization, RBCD operates on the product of the complex Stiefel manifold and the special linear group. One block uses Riemannian line-search descent on the Stiefel factor, and the other uses plane Givens, triangular, or diagonal transformations on the special linear factor, producing BCD-GLU, BCD-GQU, and BCD-GU. The square-case Jacobi analogues on are Jacobi-GLU and Jacobi-GQU. Under bounded iterates, the whole sequence converges and the gradient vanishes (Li et al., 2020).
Several limitations recur across the literature. First, nonconvexity remains intrinsic: the general guarantees are to stationary points or -stationary points, not to global minimizers. Second, block or subspace selection matters. TSD gives counterexamples showing that poorly chosen subspaces can prevent convergence even if they span the tangent space at each step; the deterministic gap ensuring and randomized -randomized norm conditions were introduced precisely to exclude such failures (Gutman et al., 2019). Third, low per-iteration cost does not remove all large-scale bottlenecks. In PRW, the dependence persists even though the method avoids solving a full Sinkhorn problem at every iteration, and the performance remains sensitive to and (Huang et al., 2020). Fourth, different RBCD papers use different block notions—product blocks, tangent subspaces, or manifold-specific coordinates—so results from one setting should not be transferred mechanically to another. A plausible implication is that “RBCD” is best understood as a methodological class rather than a single canonical algorithm.
Across these formulations, the common design principle is stable: restrict the search to a geometrically meaningful block, compute a blockwise Riemannian descent direction, and return to the manifold by $\Exp$ or a retraction. The differences lie in how the blocks are defined, which assumptions are used to prove descent, and how aggressively the algorithm exploits manifold structure for closed-form or low-cost updates.