--- title: Orthogonality-Constrained Update Rules url: https://www.emergentmind.com/topics/orthogonality-constrained-update-rules type: topic --- # Orthogonality-Constrained Update Rules Orthogonality-constrained update rules are the foundational mechanisms underlying optimization procedures on matrix manifolds where the variables are restricted to satisfy orthogonality constraints, notably the Stiefel manifold \( \mathrm{St}(n, p) = \{ X \in \mathbb{R}^{n \times p} : X^\top X = I_p \} \). Such constraints are pervasive in problems across statistics, machine learning, signal processing, quantum chemistry, and computational physics. The design of update rules that preserve or efficiently control orthogonality is central to the scalability, convergence, and correctness of algorithms in these domains. ## 1. Riemannian Geometry and Tangent-space Projections Orthogonality constraints endow the feasible set with a Riemannian manifold structure, which leads to algorithms that perform ascent or descent intrinsically on the manifold. The canonical approach replaces the Euclidean gradient with the Riemannian gradient, computed as the projection of the Euclidean gradient onto the tangent space: \[ \mathrm{grad} \, f(X) = \nabla f(X) - X \cdot \mathrm{sym}(X^\top \nabla f(X)), \] where \( \mathrm{sym}(A) = \frac{1}{2}(A + A^\top) \), and \( \nabla f(X) \) denotes the Euclidean gradient [1611.05927, 1811.11733, 2505.12378]. Steps are taken in the tangent bundle, which enforces infinitesimal feasibility. In Riemannian methods, each iteration typically consists of: - Computing the Riemannian gradient or a geometry-consistent search direction. - Taking a step along this direction. - Applying a retraction, which maps the resulting point back onto the feasible manifold. Typical retractions include QR-based retractions, polar decompositions, or Cayley transforms. QR retraction is implemented as follows: given \( X \) and tangent vector \( \xi \), set \( Y = X + \xi \), find the thin QR factorization \( Y = Q R \), and set the next iterate as \( Q \) (with sign correction as needed) [1611.05927, 1811.11733, 1809.00452]. ## 2. Structured Update Rules and Algorithmic Variants Several algorithmic families have been developed to exploit the manifold structure: - **Manifold SGD and Backpropagation**: In deep learning, Stiefel layers enforce orthogonality on fully connected or convolutional filters by integrating the Riemannian projection and retraction into the SGDs or backpropagation routines. This results in update rules that maintain \( W^\top W = I \) throughout training [1611.05927]. - **Block-Coordinate and Randomized Updates**: Block coordinate descent (BCD) methods update a subset (block) of rows or columns and resolve the subproblem on the corresponding lower-dimensional Stiefel manifold, e.g., OBCD [2304.03641]. Randomized submanifold methods apply local manifold optimization on randomly chosen subspaces, greatly reducing per-iteration complexity and enabling scalability for large \( n \) or \( p \) [2505.12378]. - **Augmented Lagrangian and Penalty Methods**: Proximal linearized augmented Lagrangian (PLAM) or column-wise block minimization (PCAL) methods replace expensive per-iteration retraction with a penalty term and update the dual variables in closed form. Iterates remain close to the manifold and a final retraction provides exact feasibility [1810.03930]. - **Non-feasible ("Infeasible") and Landing Algorithms**: The landing algorithm introduces an "attracting field" penalizing deviations from orthogonality so that iterates are driven toward the manifold, while ensuring the limiting point and convergence rate coincide with those of exact manifold methods. The update has the form \[ X_{k+1} = X_k - \eta [\, f_1(X_k) + \lambda X_k (X_k^\top X_k - I) \,], \] where \( f_1(X) \) is the off-manifold Riemannian gradient [2303.16510]. - **Augmented ADMM and Variants for Nonsmooth Problems**: Algorithms such as OADMM apply ADMM splitting to structured optimization under orthogonality, using either exact projection onto the Stiefel manifold via SVD or retraction with a Riemannian gradient. Both variants ensure \( X^\top X = I \) is satisfied after each update [2405.15129]. - **Quasi-Newton and Second-order Methods**: Quasi-Newton frameworks (e.g., LSR1, L-BFGS) approximate the Riemannian Hessian or exploit the structure of the problem (e.g., cheap/high-cost Hessian splits) and resolve the subproblem with orthogonality-preserving solvers [1809.00452]. ## 3. Multiplicative and Nonnegative-Orthogonality Updates In the context of nonnegative matrix factorization and clustering, orthogonality is imposed alongside nonnegativity. Multiplicative update rules preserve nonnegativity automatically and are modified by adding orthogonality penalties to the objective: \[ \begin{aligned} &\text{Objective: } \frac12 \|A - B S C\|_F^2 + \frac\alpha2 \|C C^\top - I\|_F^2 + \frac\beta2 \|B^\top B - I\|_F^2 \\ &\text{Updates:} \begin{cases} b_{mp} \leftarrow b_{mp} \frac{ (A C^\top S^\top)_{mp} + \beta B_{mp} }{ (B S C C^\top S^\top)_{mp} + \beta (B B^\top B)_{mp} }, \\ c_{qn} \leftarrow c_{qn} \frac{ (S^\top B^\top A)_{qn} + \alpha C_{qn} }{ (S^\top B^\top B S C)_{qn} + \alpha (C C^\top C)_{qn} }. \end{cases} \end{aligned} \] These rules, together with majorization-minimization arguments, guarantee convergence to stationary points under increasing penalty parameters [1710.11478]. Support-set algorithms for nonnegative and orthogonality constraints exploit the property that every feasible \( X \) with \( X \ge 0 \) and \( X^\top X = I \) can have at most one positive entry per row. A block-wise support update, coupled with closed-form column updates, ensures all iterates remain feasible [2511.03443]. ## 4. Adaptive Step Size, Retraction, and Complexity Considerations Adaptive step size strategies have been developed to eliminate the need for computationally expensive backtracking line searches. For the Stiefel or Grassmann setting, the step is computed via an explicit Taylor expansion, using function and Hessian information: \[ \alpha_k = \min \left\{ -\frac{\langle\nabla_G E(U_k), D_k\rangle}{\nabla_G^2 E(U_k)[D_k, D_k]}, \, \frac{\vartheta_k}{\|D_k\|} \right\}. \] This leads to provable convergence and significant wall-clock time savings, especially in electronic structure problems [1906.02883]. Retraction dominates the computational cost of feasible manifold methods. Randomized and block submanifold updates can reduce per-iteration complexity from \( O(np^2) \) (full QR) to \( O(n r^2 + r^3) \) for block size \( r \ll n \), allowing scalable large-dimensional optimization [2505.12378]. ## 5. Stochastic, Variance-reduced, and Distributed Updates Stochastic (SGD-type) and variance-reduction (e.g., SAGA, SVRG) variants of manifold methods use unbiased or variance-reduced Riemannian gradients. Recent advances enable the same sample complexity and convergence rates as full Riemannian updates while avoiding full retraction at each iteration [2303.16510]. In online PCA, implicit and distributed updates eliminate the explicit orthonormalization step entirely and lead to minibatch, parallelizable learning schemes [1909.04803]. Empirical results demonstrate that, for a wide class of machine-learning tasks, these infeasible but manifold-attracting updates offer strict control over orthogonality violations while maintaining competitive accuracy and time-to-solution compared to classical feasible methods [2303.16510]. ## 6. Convergence Theory and Global/Local Rates A broad spectrum of theoretical guarantees exist for orthogonality-constrained update rules. - Feasible Riemannian gradient methods on compact manifolds with L-smooth objectives satisfy non-asymptotic sublinear rates, e.g., \( \min_{i