Geometry-Preserving Orthonormal Init

• Geometry-preserving orthonormal initialization is a framework that enforces exact orthonormality while maintaining the intrinsic geometry of data transformations.
• It employs methods like QR decomposition and Givens rotations to minimize geometric distortion and control singular value spectra, ensuring stable conditioning.
• The approach enhances training dynamics by aligning layerwise maps and preserving signal norms across deep networks, low-rank adaptations, and sparse architectures.

Geometry-preserving orthonormal initialization denotes a family of initialization and orthonormalization procedures that enforce orthogonality or orthonormality while attempting to preserve a relevant geometry of the object being initialized. In the cited literature, that geometry may be the Euclidean structure of signals and gradients in deep networks, the span and least-squares location of an existing vector family, the singular-vector structure of a pretrained operator, or the norm-preserving structure of a sparse linear map. Typical invariants are exact or scaled relations such as $Q^\top Q = I$, $(W^l)^\top W^l = \sigma_w^2 I$, or $A_0A_0^\top = I_r$, together with constructions that either minimize distortion within a fixed subspace, align a low-rank adapter with pretrained geometry, or maintain dynamical-isometry-style conditioning during optimization (Hu, 2012, Hu et al., 2020, Büyükakyüz, 2024, Nowak et al., 2024, Zhang et al., 30 Jun 2026).

1. Formal concept and mathematical criteria

The term covers several distinct but related objectives. In Hilbert-space orthonormalization, the target is an orthonormal basis of the same span that is as close as possible to a given ordered tuple in the least-squares sense. In deep networks, the target is typically a layerwise or end-to-end map whose singular values remain controlled, so that forward signals and backward gradients neither explode nor vanish. In low-rank adaptation, the target is a trainable subspace that is orthonormal and aligned with the pretrained operator rather than chosen from unrelated random noise. In sparse models, the target is an exactly orthogonal or orthonormal sparse operator rather than a pruned approximation to one (Hu, 2012, Hu et al., 2020, Büyükakyüz, 2024, Nowak et al., 2024, Zhang et al., 30 Jun 2026).

Setting	Canonical construction	Geometry preserved
Same-span orthonormalization	$Q = A(A^*A)^{-1/2}$	Same span; minimal $\|Q-A\|_F^2$
Deep networks	$(W^l)^\top W^l = \sigma_w^2 I$	Norms, singular values, Jacobian conditioning
LoRA-style adaptation	$\Delta W = BA$ with orthonormal factor	Pretrained subspace alignment
Sparse training	Product of Givens rotations	Exact sparse orthogonality

A recurrent distinction in this literature is between orthogonality of individual layer maps and preservation of training dynamics. Orthogonality of $W^l$ controls the singular values of the linear maps themselves, but the relevant end-to-end object may instead be a nonlinear Jacobian, a tangent kernel, or a projected low-rank update. Consequently, geometry preservation at initialization does not automatically imply faster optimization in every regime. This distinction is explicit in both the NTK analysis of wide nonlinear networks and the RLVR analysis of LoRA initialization (Huang et al., 2020, Zhang et al., 30 Jun 2026).

2. Same-span orthonormalization and minimal geometric distortion

A foundational formulation appears in the least-squares orthonormalization problem: given linearly independent vectors $(\alpha_1,\dots,\alpha_n)$ in a Hilbert space $H$, one seeks the orthonormal basis of $(W^l)^\top W^l = \sigma_w^2 I$0 minimizing

$(W^l)^\top W^l = \sigma_w^2 I$1

The unique minimizer is the symmetric, or Löwdin, orthonormalization

$(W^l)^\top W^l = \sigma_w^2 I$2

equivalently

$(W^l)^\top W^l = \sigma_w^2 I$3

where $(W^l)^\top W^l = \sigma_w^2 I$4. The construction preserves the span exactly, enforces orthonormality exactly, and minimizes total displacement exactly. The paper also gives the exact optimal value

$(W^l)^\top W^l = \sigma_w^2 I$5

and proves uniqueness of the minimizer (Hu, 2012).

This formulation makes “geometry-preserving” precise in a same-subspace sense. It does not preserve the original Gram matrix unless the input is already orthonormal, but it preserves the represented subspace and the ordering $(W^l)^\top W^l = \sigma_w^2 I$6, while minimizing Frobenius or least-squares distortion among all orthonormal bases of that span. The paper also proves perturbation bounds showing that stability depends explicitly on the conditioning of the Gram matrix through $(W^l)^\top W^l = \sigma_w^2 I$7, which identifies near-linear dependence as the principal failure mode (Hu, 2012).

A stochastic alternative is given by the Kaczmarz-inspired orthogonalization procedure that repeatedly samples an ordered pair $(W^l)^\top W^l = \sigma_w^2 I$8 of unit vectors and updates

$(W^l)^\top W^l = \sigma_w^2 I$9

At every iteration, the span of all columns is preserved exactly and every column remains unit norm. The process converges almost surely to an orthonormal basis of the original span. Its analysis uses the distances

$A_0A_0^\top = I_r$0

and the potential

$A_0A_0^\top = I_r$1

together with the determinant relation

$A_0A_0^\top = I_r$2

This supplies a second geometry-preserving notion: exact subspace preservation under local pairwise orthogonalization, rather than global least-squares optimality (Shah et al., 2024).

3. Deep-network isometry and provable benefits in linear models

In deep learning, geometry-preserving orthonormal initialization is usually motivated by norm preservation and singular-value control. For square orthogonal $A_0A_0^\top = I_r$3, one has $A_0A_0^\top = I_r$4, so norms and inner products are preserved exactly. In deep linear networks, the relevant phenomenon is that products of scaled orthogonal matrices remain well-conditioned. Under the initialization studied in deep linear networks, every non-end-to-end partial product satisfies

$A_0A_0^\top = I_r$5

This makes each partial product an exact scaled isometry at initialization, and the paper proves that these products stay close to isometry during gradient descent. The resulting theorem states that the width sufficient for efficient convergence under orthogonal initialization has no dependence on depth, whereas the Gaussian result requires width that scales nearly linearly with depth (Hu et al., 2020).

The same paper formalizes the optimization consequence through the output-dynamics matrix

$A_0A_0^\top = I_r$6

Orthogonal initialization keeps the spectra of the partial products controlled, which in turn keeps $A_0A_0^\top = I_r$7 well-conditioned. Gaussian initialization instead makes sufficiently long products exponentially small in operator norm, producing exponentially small gradients near initialization and an exponential-time stagnation result when width is sublinear in depth (Hu et al., 2020).

A related constrained viewpoint appears in orthonormal deep linear neural networks, where layers $A_0A_0^\top = I_r$8 are constrained to the Stiefel manifold and one layer is left unconstrained. The optimization is performed by Riemannian gradient descent using the tangent projection

$A_0A_0^\top = I_r$9

and the polar retraction

$Q = A(A^*A)^{-1/2}$0

With an initialization satisfying

$Q = A(A^*A)^{-1/2}$1

the iterates obey the linear convergence bound

$Q = A(A^*A)^{-1/2}$2

The paper emphasizes that leaving one layer unconstrained is crucial: if every layer were orthonormal, the end-to-end map would itself be orthonormal and too restrictive for general targets (Qin et al., 2023).

4. The NTK caveat: when orthogonality becomes invisible

The principal misconception addressed in the modern theory is that orthogonal initialization universally speeds training. In ultra-wide nonlinear networks trained in the lazy regime, the relevant object is the neural tangent kernel rather than the finite-width Jacobian spectrum. For fully connected and convolutional architectures, the NTK analysis proves that orthogonal and Gaussian initialization induce the same infinite-width Gaussian-process limit and the same deterministic limiting NTK. For fully connected networks, the limiting kernel satisfies

$Q = A(A^*A)^{-1/2}$3

and this recursion is the same under Gaussian and orthogonal initialization when the architecture and hyperparameters are matched (Huang et al., 2020).

The same work extends the standard NTK constancy result to orthogonal initialization. Under the assumption $Q = A(A^*A)^{-1/2}$4 and a learning rate below

$Q = A(A^*A)^{-1/2}$5

the empirical NTK and parameters move only $Q = A(A^*A)^{-1/2}$6 over training. In that regime, orthogonality does not alter the kernel regression dynamics, so the geometric advantages that motivate dynamical isometry are theoretically invisible at the kernel level (Huang et al., 2020).

This does not make the dynamical-isometry intuition false in general. The same paper reports that outside the NTK regime, orthogonal initialization can improve learning speed when hyperparameters place $Q = A(A^*A)^{-1/2}$7 near a linear regime,

$Q = A(A^*A)^{-1/2}$8

and when learning rates and depths are large. In the representative $Q = A(A^*A)^{-1/2}$9, $\|Q-A\|_F^2$0, $\|Q-A\|_F^2$1 experiment on CIFAR-10, averaged over 30 random initializations, the orthogonally initialized network trained faster on the training set and also reached better test accuracy than the Gaussian-initialized one. The paper’s central distinction is therefore threefold: norm preservation of layerwise maps, dynamical isometry of the full Jacobian, and preservation of training dynamics are not the same notion (Huang et al., 2020).

5. Geometry-aware low-rank adaptation in pretrained models

Low-rank adaptation recasts initialization around a frozen pretrained weight $\|Q-A\|_F^2$2 and a trainable low-rank perturbation

$\|Q-A\|_F^2$3

Standard LoRA typically initializes $\|Q-A\|_F^2$4 with Kaiming-uniform and $\|Q-A\|_F^2$5 to zero. OLoRA changes only the initialization: it computes a QR decomposition of the pretrained weight,

$\|Q-A\|_F^2$6

forms the rank-$\|Q-A\|_F^2$7 truncation

$\|Q-A\|_F^2$8

and assigns

$\|Q-A\|_F^2$9

The guaranteed orthonormality is column-wise orthonormality of the left factor,

$(W^l)^\top W^l = \sigma_w^2 I$0

The paper presents this as a drop-in LoRA modification that preserves the same low-rank parameterization, the same order of parameter count, and the same GPU memory footprint up to negligible initialization-time overhead; the extra computation is a one-time QR decomposition with stated complexity $(W^l)^\top W^l = \sigma_w^2 I$1 (Büyükakyüz, 2024).

The geometric claim in OLoRA is that the update subspace is extracted from the column-space structure of the pretrained transformation rather than initialized randomly. The paper repeatedly frames better conditioning, improved gradient flow, preservation of spectral properties, and reduced interference with the pretrained model as hypotheses or intuitions rather than theorem-level results. Its empirical support is nonetheless broad: OLoRA is reported to converge faster than standard LoRA and to outperform it in 53 out of 60 model-task-rank combinations across OPT-1.3B, Tiny-Llama-1.1B, Gemma-2B, Mistral-7B, and LLaMA-2-7B on Arc-Challenge, Arc-Easy, BoolQ, HellaSwag, OpenBookQA, and PIQA (Büyükakyüz, 2024).

The RLVR setting sharpens the role of orthonormality. For standard LoRA with $(W^l)^\top W^l = \sigma_w^2 I$2, the first-step identity is

$(W^l)^\top W^l = \sigma_w^2 I$3

The resulting approximation theorem states that, for sufficiently small $(W^l)^\top W^l = \sigma_w^2 I$4,

$(W^l)^\top W^l = \sigma_w^2 I$5

and that for $(W^l)^\top W^l = \sigma_w^2 I$6 with $(W^l)^\top W^l = \sigma_w^2 I$7,

$(W^l)^\top W^l = \sigma_w^2 I$8

with equality if $(W^l)^\top W^l = \sigma_w^2 I$9. The paper therefore proposes row-orthonormal initialization of $\Delta W = BA$0 as theoretically optimal under the standard LoRA asymmetry $\Delta W = BA$1, together with geometry-preserving choices

$\Delta W = BA$2

where $\Delta W = BA$3 and $\Delta W = BA$4 are top and bottom right singular vectors of the pretrained weight (Zhang et al., 30 Jun 2026).

This analysis also explains why PiSSA and MiLoRA can underperform in RLVR. The paper isolates singular-value scaling as a source of update amplification, proving for the first step that

$\Delta W = BA$5

Empirically, on the 1.5B DAPO mathematical-reasoning setting, the average across five benchmarks is reported as $\Delta W = BA$6 for RLPO, $\Delta W = BA$7 for RLMO, $\Delta W = BA$8 for LoRA, $\Delta W = BA$9 for MiLoRA, and $W^l$0 for PiSSA. The KL trajectories of RLPO and RLMO are described as lower than those of PiSSA and MiLoRA and often lower than standard LoRA, tying orthonormal initialization directly to RLVR stability (Zhang et al., 30 Jun 2026).

6. Exact sparse constructions, exact parameterizations, and scope limits

Sparse training raises a separate question: whether orthogonality can be preserved exactly after sparsification. Exact Orthogonal Initialization addresses this by constructing sparse orthogonal matrices as products of random Givens rotations. Starting from

$W^l$1

the method repeatedly applies

$W^l$2

where each $W^l$3 is a Givens rotation acting on a sampled coordinate pair. Because each Givens matrix is orthogonal and products of orthogonal matrices are orthogonal, the resulting sparse matrix is exactly orthogonal. The method extends to rectangular layers in the appropriate row-orthogonal or column-orthogonal sense, and to convolution through a sparse delta-orthogonal construction that satisfies

$W^l$4

The paper emphasizes two properties that are unusual in sparse initialization: exact orthogonality rather than approximate orthogonality, and arbitrary target densities rather than a discrete admissible set (Nowak et al., 2024).

The experimental evidence for exact sparse orthogonality is strongest in very deep settings. The paper reports that EOI enables training highly sparse 1000-layer MLP and CNN networks without residual connections or normalization techniques. In the 1000-layer MLP on MNIST at $W^l$5 sparsity, Uniform-EOI reaches $W^l$6, SAO-EOI reaches $W^l$7, and the dense orthogonal baseline reaches $W^l$8. In the 1000-layer CNN on CIFAR-10 at the same sparsity, ERK-EOI reaches $W^l$9, SAO-EOI reaches $(\alpha_1,\dots,\alpha_n)$0, and the dense orthogonal baseline reaches $(\alpha_1,\dots,\alpha_n)$1. The same paper shows that EOI preserves Jacobian singular values much better than approximate orthogonality methods beyond sparsity $(\alpha_1,\dots,\alpha_n)$2, which directly links exact sparse orthogonality to the geometry-preserving interpretation (Nowak et al., 2024).

A different line of work replaces initialization-by-construction with exact orthogonal parameterization. The PLR decomposition represents an orthogonal matrix as

$(\alpha_1,\dots,\alpha_n)$3

where $(\alpha_1,\dots,\alpha_n)$4 is a permutation matrix, $(\alpha_1,\dots,\alpha_n)$5 is unit lower triangular, and $(\alpha_1,\dots,\alpha_n)$6 comes from the QR decomposition of $(\alpha_1,\dots,\alpha_n)$7. This gives a minimal unconstrained parameterization with $(\alpha_1,\dots,\alpha_n)$8 free parameters, preserves exact orthogonality for every parameter value, and is attractive when one wants ordinary Euclidean optimization over orthonormal weights. Its limitations are equally structural: the representation depends on a fixed permutation and sign convention, the determinant component is fixed by $(\alpha_1,\dots,\alpha_n)$9, and the paper does not characterize the induced sampling geometry on $H$0. It is therefore better viewed as a chart-based exact parameterization than as a principled random orthonormal initializer (Bagnato et al., 2019).

Across these works, the main limitations are consistent. Geometry-preserving orthonormal initialization is not a universal guarantee of faster learning: in the infinite-width NTK regime it may be irrelevant at the level of function-space dynamics, in LoRA-like systems some theoretical benefits remain heuristic unless tied to a concrete projector or stability bound, and exact sparse or exact manifold-preserving constructions often introduce one-time QR, SVD, or Givens-rotation costs. The common conclusion is narrower and more precise: orthonormal initialization is most consequential when optimization is sensitive to conditioning, when the initialized subspace itself matters, or when exact orthogonality is otherwise destroyed by pruning, low-rank factorization, or unconstrained parameter drift (Huang et al., 2020, Büyükakyüz, 2024, Nowak et al., 2024, Zhang et al., 30 Jun 2026).