Approximately Orthogonalized Updates

• Approximately orthogonalized updates are designed to isolate useful descent directions by subtracting redundant or destabilizing components from raw update signals.
• They are applied across various domains such as residual networks, matrix optimizers, and recurrent models using techniques like polar decomposition, Newton–Schulz iteration, or manifold projections.
• Empirical studies show these methods improve convergence rates, training stability, and performance metrics in tasks ranging from image classification to video modeling.

Approximately orthogonalized updates are update rules that modify a raw increment before it is applied so that the effective step is constrained by an orthogonality condition, a semi-orthogonality condition, a manifold projection, or an approximate Neyman-orthogonality relation. In recent work, this idea appears in at least four technically distinct but geometrically related forms: residual-stream updates that discard the component parallel to the incoming representation, matrix-valued optimizer steps that replace momentum by a polar-factor direction or an approximation thereof, recurrent-state updates retracted to the Stiefel manifold, and nuisance-robust stochastic gradients corrected by an estimated orthogonalizing operator (Oh et al., 17 May 2025, Zhang et al., 3 Sep 2025, Wang et al., 27 Mar 2026, Yu et al., 28 Aug 2025).

1. Scope of the concept

The literature does not use a single universal formalism for “approximately orthogonalized updates.” Instead, the phrase denotes a family of constructions whose common objective is to suppress redundant or destabilizing directions while preserving useful descent or representational structure. In deep architectures, the reference object is often the current residual stream; in matrix optimization it is the singular geometry of the update; in recurrent models it is the orthonormality constraint $S^\top S = I$; and in orthogonal statistical learning it is the nuisance sensitivity of the score itself (Oh et al., 17 May 2025, Xu et al., 29 Jan 2026, Wang et al., 27 Mar 2026, Yu et al., 28 Aug 2025).

Domain	Raw update	Orthogonalization target
Residual networks	$F(x_l)$	Remove component parallel to $x_l$
Matrix optimizers	$G_t$ or $M_t$	Use $UV^\top$ or an approximation
Recurrent state evolution	$S_t^{\rm Euc}$	Retract to $S^\top S=I$
Nuisance-aware SGD	$S(\theta,g;z)$	Cancel first-order $g$-sensitivity

This breadth matters because the adjective “orthogonalized” can otherwise be misleading. In some papers it refers to Euclidean orthogonality of vectors or matrix columns; in others it refers to orthogonality under a non-Euclidean metric, such as a Fisher geometry; and in the nuisance literature it refers to orthogonality of a score with respect to perturbations in an auxiliary parameter rather than to any matrix factorization (Xu et al., 29 Jan 2026, Yu et al., 28 Aug 2025).

2. Core algebraic forms

A canonical representation-level construction is the Orthogonal Residual Update. If $F(x_l)$0 is the residual stream at layer $F(x_l)$1 and $F(x_l)$2 is the module, the standard residual rule is

$F(x_l)$3

The orthogonalized variant decomposes $F(x_l)$4 into components parallel and orthogonal to $F(x_l)$5, and applies only the orthogonal part,

$F(x_l)$6

followed by

$F(x_l)$7

This removes the part of the module output that merely rescales or anti-aligns the existing stream direction (Oh et al., 17 May 2025).

A canonical optimizer-level construction uses the polar factor. If $F(x_l)$8 is a matrix momentum or gradient, the orthogonalized direction is

$F(x_l)$9

AdaGO formulates this as the spectral-norm steepest-descent solution, while OptMuon uses the same Muon-style polar-factor direction but scales it by a closed-loop coefficient schedule, and FISMO applies the polar operation after whitening by Kronecker-factored Fisher factors (Zhang et al., 3 Sep 2025, Yuan, 7 Jun 2026, Xu et al., 29 Jan 2026).

A recurrent-state version appears in Orthogonalized State Update. There the unconstrained Euclidean state proposal

$x_l$0

is projected back onto the Stiefel manifold

$x_l$1

by solving

$x_l$2

implemented approximately via a polar-type Newton–Schulz iteration rather than a full SVD (Wang et al., 27 Mar 2026).

A statistically distinct construction appears in orthogonal statistical learning. Given an original oracle $x_l$3, one forms an approximately orthogonalized oracle

$x_l$4

Here the point is not matrix orthonormality but approximate Neyman orthogonality: the residual first-order sensitivity to nuisance perturbations is made small when $x_l$5 approximates the true orthogonalizer $x_l$6 (Yu et al., 28 Aug 2025).

3. Residual-stream and state-space applications

In feedforward deep networks, Orthogonal Residual Update was proposed as a modification of residual connections intended to guide modules toward injecting new representational directions. The stated empirical setting includes ResNetV2 with depths $x_l$7, Vision Transformers ViT-S and ViT-B, and datasets CIFAR-10, CIFAR-100, TinyImageNet, and ImageNet-1k. The reported mean$x_l$8std results over five runs include ViT-B on ImageNet-1k improving from $x_l$9 to $G_t$0 and ViT-S on ImageNet-1k improving from $G_t$1 to $G_t$2; on CIFAR-100, ViT-B improves from $G_t$3 to $G_t$4, while ResNetV2 gains are described as smaller but consistent, such as ResNetV2-34 on TinyImageNet moving from $G_t$5 to $G_t$6 (Oh et al., 17 May 2025).

The same work reports faster loss reduction in early iterations and lower time-to-accuracy, together with ablations indicating that training wholly with orthogonal updates gives the best final performance, that accuracy grows roughly linearly with the probability $G_t$7 in a probabilistic orthogonality variant, that applying orthogonal updates to more layers yields higher accuracy with “all-layers” best, and that $G_t$8 gave the most stable runs among values from $G_t$9 to $M_t$0 (Oh et al., 17 May 2025). The paper also distinguishes feature-wise orthogonalization from global orthogonalization, with the former described as simple and efficient for Transformer blocks and the latter recommended for CNNs or global features.

In recurrent video modeling, Orthogonalized State Update addresses a different failure mode: progressive singular-value decay of the state matrix, termed rank collapse. The mechanism constrains state evolution on the Stiefel manifold and uses an approximate polar decomposition by Newton–Schulz iteration to re-orthonormalize the intermediate state. The reported consequences include constant-spectral-norm behavior, $M_t$1, a mean singular-value drift stabilized at approximately $M_t$2 versus $M_t$3 in an uncontrolled baseline, a $M_t$4 mDice gain over the anatomically-prior baseline and $M_t$5 when combined with APFE, with approximately $M_t$6 fps inference and $M_t$7 h to converge (Wang et al., 27 Mar 2026). This suggests that approximately orthogonalized state updates are useful not only for gradient transport but also for long-horizon memory geometry.

4. Orthogonalized optimizers and adaptive scaling

A major contemporary use of approximately orthogonalized updates is in matrix-valued optimization. OptMuon combines Muon-style polar-factor directions with a trajectory-dependent AdaGrad-Norm-type coefficient schedule,

$M_t$8

where the scalar coefficient is determined by observed gradient and momentum history rather than by smoothness, variance, or bounded-gradient constants. Under the stated assumptions, OptMuon-A achieves

$M_t$9

OptMuon-I achieves

$UV^\top$0

and both reduce automatically to $UV^\top$1 in the zero-noise regime without retuning (Yuan, 7 Jun 2026).

AdaGO retains exact orthogonalization of the momentum-averaged gradient but uses a single scalar AdaGrad-type accumulator. With

$UV^\top$2

and an adaptive scalar $UV^\top$3 based on accumulated clamped Frobenius norms, the update is $UV^\top$4. The stated theoretical results include an $UV^\top$5 stochastic stationarity rate in nuclear norm and an $UV^\top$6 deterministic rate, while the implementation adds only one scalar state variable beyond Muon (Zhang et al., 3 Sep 2025).

FISMO generalizes strict isotropic orthogonalization to a Fisher-structured setting. It solves a trust-region problem under a Kronecker-factored Fisher metric and arrives at the closed-form update

$UV^\top$7

The paper characterizes this as exact orthonormality under the local Fisher metric together with retained Fisher-implied anisotropy in the original coordinates. Its main convergence theorem gives an $UV^\top$8 rate for the average expected nuclear norm of the gradient term, with an additional $UV^\top$9 variance term. The reported spectral-condition analysis places Adam above $S_t^{\rm Euc}$0, Muon near $S_t^{\rm Euc}$1, and FISMO in a moderate band of $S_t^{\rm Euc}$2–$S_t^{\rm Euc}$3 (Xu et al., 29 Jan 2026).

Other optimizer variants focus on robustness or cost. DP-Muon clips per-example matrix gradients, adds Gaussian noise, accumulates momentum, and then applies finite-step Newton–Schulz orthogonalization as post-processing, thereby inheriting the privacy guarantee of the same-lot subsampled Gaussian accountant with no additional privacy cost from the orthogonalization stage; its bias-corrected variant DP-MuonBC cancels the leading $S_t^{\rm Euc}$4 heat-smoothing bias term and leaves an $S_t^{\rm Euc}$5 residual (Kim et al., 13 May 2026). ROOT uses dimension-specific quintic Newton iterations together with proximal soft-thresholding of the momentum matrix before orthogonalization, and reports lower relative orthogonalization error than Muon and classic fixed-coefficient Newton–Schulz under a budget of five iterations, along with training-loss and zero-shot gains in the stated benchmarks (He et al., 25 Nov 2025). AuON departs further from explicit semi-orthogonalization: it enforces a strict spectral trust region by a linear-time normalization and hyperbolic-cosine scaling pass, and a hybrid variant adds a single Newton–Schulz step (Maity, 29 Sep 2025).

5. Approximation mechanisms and cost models

Approximate orthogonalization is typically motivated by the cost of exact SVD- or QR-based projections. In Orthogonal Residual Update, the extra computation for orthogonalizing a sequence of length $S_t^{\rm Euc}$6 and hidden size $S_t^{\rm Euc}$7 is $S_t^{\rm Euc}$8, compared with $S_t^{\rm Euc}$9 for a Transformer block or $S^\top S=I$0 for an MLP, and is therefore described as negligible (Oh et al., 17 May 2025). This is the simplest cost regime: a rank-one correction along an already available stream direction.

For matrix optimizers and manifold retractions, the dominant approximation engine is Newton–Schulz. DP-Muon uses a short Newton–Schulz routine whose finite-step output has operator norm at most $S^\top S=I$1 and whose deviation from the exact polar factor is bounded by

$S^\top S=I$2

with doubly-exponential decay in the number of steps $S^\top S=I$3 under the stated initialization condition (Kim et al., 13 May 2026). OSA likewise uses a low-order Newton–Schulz projection onto the Stiefel manifold rather than a full SVD (Wang et al., 27 Mar 2026).

A central line of work concerns optimizing the polynomial itself. CANS observes that the classical Newton–Schulz cubic

$S^\top S=I$4

is generally not minimax-optimal on the relevant singular-value interval. It replaces this fixed polynomial by Chebyshev-optimized odd polynomials computed either in closed form for degree $S^\top S=I$5 or by a Remez algorithm for higher degree. The paper states that cubic-CANS satisfies

$S^\top S=I$6

whereas classical Newton–Schulz has asymptotic constant $S^\top S=I$7. In a random $S^\top S=I$8 Gaussian example, standard Newton–Schulz needs approximately $S^\top S=I$9 matrix multiplications to reach $S(\theta,g;z)$0 error, cubic CANS approximately $S(\theta,g;z)$1, and quintic CANS approximately $S(\theta,g;z)$2 (Grishina et al., 12 Jun 2025).

AuON represents a different approximation philosophy. Instead of approximating $S(\theta,g;z)$3, it bounds the update inside the unit spectral-norm ball by

$S(\theta,g;z)$4

and proves $S(\theta,g;z)$5. All operations are $S(\theta,g;z)$6 in the number of elements, in contrast to $S(\theta,g;z)$7 or $S(\theta,g;z)$8 costs associated with semi-orthogonal Newton–Schulz or full factorizations (Maity, 29 Sep 2025). This suggests that the design space includes not only approximations to orthogonal factors but also cheaper surrogates that reproduce a trust-region or spectrum-homogenizing effect.

6. Stability, convergence, and generalization

Theoretical analyses emphasize that orthogonalization modifies geometry without necessarily destroying the favorable transport properties of the underlying architecture or optimizer. For residual networks, the Jacobian of the orthogonalized update takes the form

$S(\theta,g;z)$9

so the identity skip path remains intact. The associated appendix analysis states that subtracting the parallel term introduces only low-rank modifications to the Jacobian and does not block propagation of information about the parallel component entirely (Oh et al., 17 May 2025). This directly addresses a common concern that orthogonalized residuals might eliminate the benefits of residual learning.

In stochastic optimization, the main theoretical theme is decoupling direction from magnitude. OptMuon uses a closed-loop scalar schedule derived from observed $g$0 values and a running-maximum correction to avoid permanent coefficient collapse after isolated gradient spikes (Yuan, 7 Jun 2026). AdaGO keeps the orthogonalized direction exact while adapting only the scalar step size via accumulated Frobenius norms (Zhang et al., 3 Sep 2025). FISMO performs orthogonalization after Fisher whitening, so the update is orthogonal under a local Fisher metric rather than in the ambient Euclidean sense (Xu et al., 29 Jan 2026). These formulations indicate that “orthogonalized update” need not imply a fixed-magnitude Euclidean step.

The distributed generalization analysis in FedMuon makes the regularity requirements explicit. The orthogonalization map must be Lipschitz along paired trajectories: $g$1 The paper studies three maps: $g$2, globally Frobenius-Lipschitz with constant $g$3; finite-step Newton–Schulz, globally Lipschitz with constant $g$4; and exact matrix sign, which is not globally Lipschitz and therefore requires either a spectral tube $g$5 or Gaussian smoothing. In the uniform full-participation full-batch regime, the resulting upper-tail bound yields $g$6 scaling when the amplification terms are controlled (Chang et al., 1 Jun 2026). The paper’s one-dimensional counterexample shows that a gap, smoothing, or regularity condition is necessary.

The statistical orthogonality literature reaches an analogous conclusion in a different language. Exact orthogonality removes nuisance sensitivity to second order, approximate orthogonality interpolates between exact and non-orthogonal behavior, and the error term depends jointly on nuisance estimation error and orthogonalizer estimation error: $g$7 This is not matrix orthogonalization, but it is structurally parallel: an update is corrected so that a first-order interference term is suppressed (Yu et al., 28 Aug 2025).

7. Related geometries, limitations, and common misunderstandings

Approximately orthogonalized updates have deep antecedents in numerical linear algebra and geometric matrix computation. A rank-one modification $g$8 can be translated into a rank-one update of an orthogonal factor $g$9 while preserving the updated column space, with a closed-form Grassmann-geodesic interpretation and $F(x_l)$00 complexity for $F(x_l)$01 (Zimmermann, 2017). A Kaczmarz-inspired random pairwise orthogonalization process converges almost surely to an orthonormal basis of the original span, with $F(x_l)$02 iterations sufficient to bring $F(x_l)$03 above $F(x_l)$04 with constant probability (Shah et al., 2024). Structured orthogonal iterations on unitary-plus-rank-$F(x_l)$05 pencils achieve $F(x_l)$06 complexity per update by maintaining data-sparse orthogonal Hessenberg factors (Bevilacqua et al., 2021). Reorthogonalized Pythagorean block classical Gram–Schmidt restores an $F(x_l)$07 loss-of-orthogonality bound with approximately two global synchronizations per block column (Carson et al., 2024).

These related results clarify two recurrent misconceptions. First, orthogonalization is not synonymous with exact SVD-based projection; the literature includes feature-wise projections, polar decompositions, finite-step Newton–Schulz maps, Chebyshev-optimized polynomials, Gaussian-smoothed matrix signs, Kaczmarz-style pairwise updates, and trust-region surrogates (Oh et al., 17 May 2025, Grishina et al., 12 Jun 2025, Chang et al., 1 Jun 2026, Shah et al., 2024). Second, orthogonality is not always Euclidean: FISMO is orthogonal under a Fisher metric, while orthogonal statistical learning concerns score orthogonality to nuisance perturbations rather than matrix spectra (Xu et al., 29 Jan 2026, Yu et al., 28 Aug 2025).

The present limitations are also explicit in the recent literature. Orthogonal Residual Update is validated only on models up to ViT-B/ImageNet-1k and ResNetV2 up to depth $F(x_l)$08 on classification tasks, with larger networks and other modalities such as LLMs and diffusion left open (Oh et al., 17 May 2025). Exact matrix sign requires spectral separation or smoothing in stability proofs (Chang et al., 1 Jun 2026). Approximation quality, Lipschitz constants, and computational overhead remain coupled design variables: increasing fidelity to $F(x_l)$09 often improves geometry but may worsen cost or regularity constants, while cheaper surrogates may retain only part of the intended isotropization effect. A plausible implication is that “approximately orthogonalized updates” should be understood less as a single algorithm and more as a geometric design principle: isolate a useful direction, suppress an interfering component, and then recover stability through an operator whose approximation error is small enough for the ambient learning dynamics.