Low-Rank Orthogonalization

• Low-rank orthogonalization is a set of methods that combine low-dimensional factorization with orthogonal constraints to control geometry and eliminate scaling ambiguities.
• It is applied in neural training, tensor decompositions, and matrix pursuits to enable closed-form local solvers and improve numerical stability.
• This approach enhances conditioning and convergence by stabilizing updates and converting ill-posed problems into tractable, convex formulations.

Low-rank orthogonalization denotes a family of methods in which low-dimensional structure is represented, updated, or recovered together with orthogonality constraints or explicit orthogonalization operators. In the cited literature, the phrase covers factorized neural weights $W=AB^\top$ whose updates are spectrally normalized and orthogonalized, greedy matrix and tensor pursuits that add low-rank atoms and then orthogonally refit them, tensor-network algorithms that enforce left/right gauge orthogonality, orthogonal dictionary and Procrustes formulations for compression, and convex reformulations in which orthogonal projection matrices encode admissible low-rank subspaces (Janson et al., 12 Feb 2026, Wang et al., 2014, Takeda et al., 30 Jun 2026, Makhov et al., 16 Feb 2026, Bertsimas et al., 2021). The unifying theme is that low-rank structure alone reduces dimensionality, whereas orthogonalization controls geometry: it removes scaling ambiguities, improves conditioning, yields exact or closed-form local solvers, and can convert otherwise unstable or ill-posed low-rank procedures into stable and tractable ones.

1. Core formulations and geometric role

A recurring starting point is the factorization

$W=AB^\top,$

with $A\in\mathbb{R}^{m\times r}$, $B\in\mathbb{R}^{n\times r}$, and $r<\min(m,n)$. In native low-rank training, this parameterization introduces the scaling ambiguity

$$W=(\lambda A)(\lambda^{-1}B)^\top,\qquad \lambda>0,$$

so the factors can drift even when the represented matrix is unchanged. The induced product update

$$\Delta W = \Delta A\, B^\top + A\, \Delta B^\top + \Delta A\, \Delta B^\top$$

shows why this matters: small factor updates can still yield a large effective update in $W$ when the current factor norms are large, especially in spectral norm (Janson et al., 12 Feb 2026).

Orthogonalization enters as a geometry-shaping operator. In Muon-style formulations, if a gradient or momentum matrix has SVD $G_t=U\Sigma V^\top$, the orthogonalized update is

$O_t=\mathrm{Ortho}(G_t)=UV^\top,$

which normalizes all singular values to $W=AB^\top,$0 and constrains the update direction to a controlled singular-value profile (Janson et al., 12 Feb 2026). Closely related constructions appear in compression and dictionary learning, where an orthogonal dictionary satisfies $W=AB^\top,$1 and admits a Procrustes update $W=AB^\top,$2 after the thin SVD of $W=AB^\top,$3 (Makhov et al., 16 Feb 2026).

In matrix completion and tensor algorithms, orthogonality is often expressed as an invariance or residual condition rather than as a direct matrix-sign operator. In Orthogonal Rank-One Matrix Pursuit, after a new rank-one atom is selected, all coefficients are recomputed so that the residual is orthogonal to every previously selected basis matrix,

$W=AB^\top,$4

which is the matrix analogue of residual orthogonality in OMP (Wang et al., 2014). In TT decomposition, a mixed gauge imposes

$W=AB^\top,$5

so that each local ALS subproblem becomes an exact projection rather than a Gram-system solve (Takeda et al., 30 Jun 2026). In convex low-rank reformulations, orthogonality is encoded by a projection matrix

$W=AB^\top,$6

with $W=AB^\top,$7 ensuring that $W=AB^\top,$8 lies in the selected subspace (Bertsimas et al., 2021).

2. Stabilizing low-rank neural training and matrix optimizers

The most explicit recent use of low-rank orthogonalization in foundation-model training is Spectron, introduced to stabilize native low-rank LLM pretraining. The method identifies uncontrolled growth in the spectral norm of the effective weight update as the dominant instability mechanism. Assuming bounds

$W=AB^\top,$9

the paper derives

$A\in\mathbb{R}^{m\times r}$0

and for typical learning rates with $A\in\mathbb{R}^{m\times r}$1,

$A\in\mathbb{R}^{m\times r}$2

To enforce a target bound $A\in\mathbb{R}^{m\times r}$3, Spectron sets

$A\in\mathbb{R}^{m\times r}$4

Its update loop orthogonalizes momentum buffers with Newton–Schulz, estimates $A\in\mathbb{R}^{m\times r}$5 and $A\in\mathbb{R}^{m\times r}$6 by one power-iteration step, and rescales the orthogonalized factor updates accordingly. On a 94M factorized model, naive training yields about $A\in\mathbb{R}^{m\times r}$7 perplexity / $A\in\mathbb{R}^{m\times r}$8 val loss, spectral renormalization alone $A\in\mathbb{R}^{m\times r}$9, orthogonalization alone $B\in\mathbb{R}^{n\times r}$0, and the full method $B\in\mathbb{R}^{n\times r}$1. On a 454M factorized model, Spectron reaches perplexity $B\in\mathbb{R}^{n\times r}$2 versus $B\in\mathbb{R}^{n\times r}$3 for self-guided training and $B\in\mathbb{R}^{n\times r}$4 for naive AdamW, and a 454M factorized model trained for equal FLOPs can match the validation loss of a 780M dense model with roughly $B\in\mathbb{R}^{n\times r}$5 fewer parameters (Janson et al., 12 Feb 2026).

A related line of work treats orthogonalization as the core update geometry of matrix optimizers. SUMO restricts optimization to a dynamically adapted low-dimensional subspace and then orthogonalizes the moment exactly by SVD in that subspace, rather than approximating the polar factor with Newton–Schulz. The paper gives the approximation bound

$B\in\mathbb{R}^{n\times r}$6

for Newton–Schulz after $B\in\mathbb{R}^{n\times r}$7 iterations, emphasizing its dependence on the condition number of the moment matrix; exact SVD avoids this reduced-space orthogonalization error. The same paper reports that SUMO with SVD converges about $B\in\mathbb{R}^{n\times r}$8 faster than the Newton–Schulz5 variant on QNLI and gives better validation perplexity than GaLore in LLaMA pretraining, with $B\in\mathbb{R}^{n\times r}$9 versus $r<\min(m,n)$0 on the 1B model (Refael et al., 30 May 2025). Low-rank Muon replaces full $r<\min(m,n)$1 by

$r<\min(m,n)$2

and proves

$r<\min(m,n)$3

so the orthogonalization is exact on the projected low-rank approximation $r<\min(m,n)$4. Its pretraining experiments show better validation perplexity than vanilla Muon for larger GPT-2 and LLaMA models (He et al., 15 Sep 2025). Pro-KLShampoo combines a low-rank-plus-flat-tail KL-Shampoo factor with complement orthogonalization, using the identity

$r<\min(m,n)$5

and reports final-loss improvements such as $r<\min(m,n)$6 versus $r<\min(m,n)$7 on LLaMA 450M together with lower peak memory and faster wall-clock convergence (Sun et al., 7 May 2026). In zeroth-order optimization, ZO-Muon projects noisy gradients into a column-orthonormal subspace $r<\min(m,n)$8, applies $r<\min(m,n)$9 to the reduced estimate, and lifts the result back via $$W=(\lambda A)(\lambda^{-1}B)^\top,\qquad \lambda>0,$$0; it requires only $$W=(\lambda A)(\lambda^{-1}B)^\top,\qquad \lambda>0,$$1 of the queries to reach the same SST-2 performance as MeZO and improves accuracy by $$W=(\lambda A)(\lambda^{-1}B)^\top,\qquad \lambda>0,$$2 on ViT-B fine-tuning on CIFAR-100 (Lang et al., 19 Feb 2026).

Orthogonality has also been used as a schedule for reducing trainable degrees of freedom after the basis has stabilized. OIALR tracks the SVD of ordinary dense training and reports that the orthogonal basis changes substantially in the first few epochs and then stabilizes toward $$W=(\lambda A)(\lambda^{-1}B)^\top,\qquad \lambda>0,$$3 in a basis-stability score. The method delays the switch to a low-rank $$W=(\lambda A)(\lambda^{-1}B)^\top,\qquad \lambda>0,$$4 parameterization, freezes $$W=(\lambda A)(\lambda^{-1}B)^\top,\qquad \lambda>0,$$5 and $$W=(\lambda A)(\lambda^{-1}B)^\top,\qquad \lambda>0,$$6, trains $$W=(\lambda A)(\lambda^{-1}B)^\top,\qquad \lambda>0,$$7, and periodically refreshes the basis. On ViT-B/16 on ImageNet-2012, it reports $$W=(\lambda A)(\lambda^{-1}B)^\top,\qquad \lambda>0,$$8 top-1 and $$W=(\lambda A)(\lambda^{-1}B)^\top,\qquad \lambda>0,$$9 top-5 with $$\Delta W = \Delta A\, B^\top + A\, \Delta B^\top + \Delta A\, \Delta B^\top$$0 trainable parameters and $$\Delta W = \Delta A\, B^\top + A\, \Delta B^\top + \Delta A\, \Delta B^\top$$1 compression, compared with a baseline of $$\Delta W = \Delta A\, B^\top + A\, \Delta B^\top + \Delta A\, \Delta B^\top$$2 top-1 and $$\Delta W = \Delta A\, B^\top + A\, \Delta B^\top + \Delta A\, \Delta B^\top$$3 top-5 (Coquelin et al., 2024).

3. Greedy matrix pursuit and orthogonal refitting

In matrix completion, low-rank orthogonalization appears as a pursuit strategy that repeatedly selects rank-one directions and then orthogonally re-optimizes their coefficients. OR1MP formulates the observed-entry completion problem as a rank-constrained least-squares objective and, at iteration $$\Delta W = \Delta A\, B^\top + A\, \Delta B^\top + \Delta A\, \Delta B^\top$$4, chooses the next atom by

$$\Delta W = \Delta A\, B^\top + A\, \Delta B^\top + \Delta A\, \Delta B^\top$$5

so the selected matrix is the outer product of the top singular vector pair of the residual. The defining orthogonalization step is the joint least-squares refit over all previously chosen atoms. This produces a residual orthogonal to the selected span and yields the contraction

$$\Delta W = \Delta A\, B^\top + A\, \Delta B^\top + \Delta A\, \Delta B^\top$$6

hence linear convergence. The economic variant EOR1MP replaces the full refit with a two-variable update involving only the current estimate and the new atom, reducing storage from $$\Delta W = \Delta A\, B^\top + A\, \Delta B^\top + \Delta A\, \Delta B^\top$$7 to $$\Delta W = \Delta A\, B^\top + A\, \Delta B^\top + \Delta A\, \Delta B^\top$$8 while preserving the same linear rate (Wang et al., 2014).

A different generalization appears in rank-revealing factorization under arbitrary norms. The factorization still has the form

$$\Delta W = \Delta A\, B^\top + A\, \Delta B^\top + \Delta A\, \Delta B^\top$$9

with $W$0 a permutation, $W$1 upper triangular with nonincreasing diagonal entries, and $W$2 built from normalized residuals. The pivot rule is

$W$3

so column selection is again driven by residual novelty. In the Euclidean case, the new column is the orthogonal residual and $W$4; in arbitrary norms, orthogonality is replaced by best approximation in the chosen norm. The paper uses repeated

$W$5

subproblems, implemented as linear programs, to produce deterministic low-rank approximations with $W$6 error control and a rank-revealing diagonal in $W$7 (Atcheson, 2019). A plausible implication is that low-rank orthogonalization is not restricted to Euclidean geometry: the same greedy-and-refit pattern survives when orthogonality is replaced by norm-dependent residual optimality.

4. Tensor trains, orthogonal tensor approximation, and gauge constraints

In TT decomposition, orthogonalization is often the mechanism that makes local low-rank updates exact rather than merely heuristic. Online TT-ALS maintains a mixed gauge in which cores to the left of the current site are left-orthogonal and those to the right are right-orthogonal. Under this condition, the local environments satisfy

$W$8

and the local objective

$W$9

has the exact minimizer

$G_t=U\Sigma V^\top$0

The paper proves monotonic decrease of the local objective, a temporal smoothness bound $G_t=U\Sigma V^\top$1, and a per-time-step complexity

$G_t=U\Sigma V^\top$2

whose dominant term becomes $G_t=U\Sigma V^\top$3 under $G_t=U\Sigma V^\top$4. Its ablation on a dynamic background sequence shows that without orthogonality the relative error degrades as $G_t=U\Sigma V^\top$5 at $G_t=U\Sigma V^\top$6, whereas with orthogonality it remains $G_t=U\Sigma V^\top$7 (Takeda et al., 30 Jun 2026).

A complementary numerical study examines six orthogonalization kernels in TT format—CGS, MGS, CGS2, MGS2, Gram, and Householder—combined with TT-rounding to control rank growth. TT-rounding satisfies

$G_t=U\Sigma V^\top$8

and the experiments indicate that classical round-off bounds persist with the unit round-off $G_t=U\Sigma V^\top$9 replaced by the TT-rounding accuracy $O_t=\mathrm{Ortho}(G_t)=UV^\top,$0. The paper reports that TT-MGS2 usually gives the best orthogonality, TT-Householder is the most stable and often has the best memory profile among the stable methods, and TT-CGS and TT-Gram deteriorate rapidly on ill-conditioned inputs (Coulaud et al., 2022).

More general orthogonal tensor approximation methods use explicit low-rank orthogonalization steps. A modified algorithm for orthogonal low-rank CP approximation computes one orthonormal factor by SVD, then constructs the remaining orthonormal factors through a split–gather process in which intermediate vectors are assembled into a matrix $O_t=\mathrm{Ortho}(G_t)=UV^\top,$1 and orthogonalized by polar decomposition $O_t=\mathrm{Ortho}(G_t)=UV^\top,$2. Its main contribution is an approximation lower bound valid for any number of orthonormal modes $O_t=\mathrm{Ortho}(G_t)=UV^\top,$3, rather than only $O_t=\mathrm{Ortho}(G_t)=UV^\top,$4 (Yang, 2020). JLROA, by contrast, treats low-rank orthogonal approximation of symmetric tensors as a Jacobi problem on $O_t=\mathrm{Ortho}(G_t)=UV^\top,$5, maximizing the sum of squared diagonal coefficients via successive Givens rotations. It includes Jacobi CoM2 as the special case $O_t=\mathrm{Ortho}(G_t)=UV^\top,$6 and proves that any accumulation point is stationary; for the best rank-2 orthogonal approximation of 3rd-order symmetric tensors, it further establishes a stronger uniqueness-of-limit-point result under additional conditions (Li et al., 2019).

5. Compression, representation stabilization, and learned orthogonal transforms

In transformer compression, orthogonality is used to retain low-rank efficiency without forcing all columns into a single shared subspace. COMPOT first whitens the calibration-aware objective

$O_t=\mathrm{Ortho}(G_t)=UV^\top,$7

by writing $O_t=\mathrm{Ortho}(G_t)=UV^\top,$8 and $O_t=\mathrm{Ortho}(G_t)=UV^\top,$9. It then solves

$W=AB^\top,$00

where orthogonality makes both alternating steps closed-form: $W=AB^\top,$01 from the thin SVD of $W=AB^\top,$02. The method combines this with one-shot dynamic allocation across layers. On Llama3-8B at compression ratio $W=AB^\top,$03, it reports average accuracy $W=AB^\top,$04 and WikiText PPL $W=AB^\top,$05, versus $W=AB^\top,$06 and $W=AB^\top,$07 for SVD-LLM; on Whisper Large V3 at CR $W=AB^\top,$08, it reports WER $W=AB^\top,$09 on test-clean and $W=AB^\top,$10 on test-other, versus $W=AB^\top,$11 and $W=AB^\top,$12 for SVD-LLM (Makhov et al., 16 Feb 2026).

A different use case is embedding-space stabilization across retraining cycles. Orthogonal low-rank embedding stabilization begins with a recommender factorization $W=AB^\top,$13, computes QR decompositions $W=AB^\top,$14, $W=AB^\top,$15, performs SVD on the small matrix $W=AB^\top,$16, and forms transforms $W=AB^\top,$17 and $W=AB^\top,$18 such that

$W=AB^\top,$19

An orthogonal Procrustes alignment to a reference run then yields stabilized embeddings $W=AB^\top,$20 satisfying

$W=AB^\top,$21

Because the final alignment is orthogonal, dot products are preserved exactly. On the reported recommender experiments, same-user cosine similarity rises from $W=AB^\top,$22 to $W=AB^\top,$23 for period A vs B, same-item cosine similarity from $W=AB^\top,$24 to $W=AB^\top,$25, and rank correlation from $W=AB^\top,$26 to $W=AB^\top,$27 (Zielnicki et al., 11 Aug 2025).

In multi-dimensional inverse problems, OTLRM replaces a hand-designed transform in t-SVD/TNN with a learnable orthogonal transform built from Householder reflections,

$W=AB^\top,$28

and combines it with a generative low-rank factorization and a learned dense rank estimator $W=AB^\top,$29. The construction avoids differentiating through hard SVD/SVT layers while preserving transform-domain orthogonality. The paper reports $W=AB^\top,$30–$W=AB^\top,$31 dB gains in low-sampling tensor completion regimes and an average PSNR around $W=AB^\top,$32 on Brain MRI completion, better than DTNN and HLRTF in its table (Wang et al., 2024).

6. Structural theory, convex reformulations, and exact recovery

Low-rank orthogonalization also has a convex-analytic interpretation. A central result in low-rank optimization introduces an orthogonal projection matrix

$W=AB^\top,$33

as the matrix analogue of a binary selector and imposes $W=AB^\top,$34 to encode $W=AB^\top,$35. The convex hull of these projection matrices is explicit: $W=AB^\top,$36 Combined with the matrix perspective

$W=AB^\top,$37

this yields matrix perspective reformulations that recover convex hulls of low-rank epigraphs and produce SDP or conic relaxations for reduced rank regression, non-negative matrix factorization, and factor analysis (Bertsimas et al., 2021).

The tensor analogue exposes both the strengths and the nonclassical behavior of orthogonality. Orthogonal tensor decomposition defines

$W=AB^\top,$38

and the associated orthogonal rank $W=AB^\top,$39. The paper on orthogonal decompositions proves that orthogonal rank can exceed ordinary tensor rank, that a subtensor may have larger orthogonal rank than the whole tensor, and that orthogonal rank is lower semicontinuous. The lower semicontinuity implies that

$W=AB^\top,$40

always has a solution. Its OD-ALM algorithm uses an augmented Lagrangian to enforce pairwise orthogonality of rank-one tensor terms and then applies a mode-selective postprocessing orthogonalization to make the final decomposition exactly orthogonal (Zeng, 2021).

For fourth-order CPS tensors, orthogonality yields an exact spectral decomposition. Such a tensor admits

$W=AB^\top,$41

where $W=AB^\top,$42, the $W=AB^\top,$43 are symmetric matrices, and

$W=AB^\top,$44

Because the square unfolding $W=AB^\top,$45 is Hermitian, this is precisely the matrix spectral decomposition recast in tensor form. The successive matrix outer product rank-one approximation algorithm recovers the decomposition exactly when the $W=AB^\top,$46 are distinct. The same paper proves

$W=AB^\top,$47

which makes matrix rank a tractable surrogate for low-CP-rank completion, and establishes a rank-one equivalence between a rank-one CPS tensor and a rank-one matrix unfolding (Sabir et al., 2021).

Taken together, these results suggest a precise interpretation of low-rank orthogonalization. It is not merely a low-rank factorization with an extra constraint. Rather, it is a geometry-selection mechanism: it identifies a subspace, basis, gauge, or dictionary in which the reduced representation becomes stable, exactly solvable, linearly convergent, or convexifiable. The specific orthogonalization operator varies—from Gram-Schmidt, Householder, and polar decomposition to matrix sign, Procrustes, and projection-matrix reformulation—but in each case the low-rank structure supplies dimensional economy while orthogonality supplies control.