Gram–Schmidt Orthogonalization

• Gram–Schmidt orthogonalization is a method that converts a set of linearly independent vectors into an orthogonal or orthonormal system using recursive projections.
• Its variants extend to p-adic norms, Clifford modules, and non-standard inner products, improving computational efficiency and numerical stability.
• Recent advancements include block, low synchronization, mixed precision, and randomized methods that address high-dimensional and ill-conditioned problems.

Gram–Schmidt orthogonalization is a foundational procedure in linear algebra and numerical analysis, providing explicit algorithms to generate orthogonal (or orthonormal) systems from linearly independent vectors in inner-product spaces, with deep implications in numerical linear algebra, functional analysis, regression modeling, high-performance computing, and quantum information. Recent research extends and generalizes the classical process for high-dimensional, structured, block, non-commutative, and non-standard inner-product settings, and rigorously analyzes its numerical stability under finite precision and communication constraints.

1. Classical Gram–Schmidt and Its Algebraic Structure

The classical Gram–Schmidt (GS) process constructs, for a list of linearly independent vectors $\{v_1,\dots,v_n\}$ in an inner-product space (typically $\mathbb{R}^m$ or $\mathbb{C}^m$), a sequence of orthogonal vectors $\{u_1,\dots,u_n\}$ spanning the same subspaces. The recursive projections are given by

$$u_1 = v_1,\qquad u_k = v_k - \sum_{j=1}^{k-1} \frac{\langle v_k, u_j\rangle}{\langle u_j, u_j\rangle}u_j$$

and orthonormalization is achieved by $q_k = u_k/\|u_k\|$. This yields a QR factorization $V=QR$, where $Q$ is column-orthonormal and $R$ upper-triangular (Wang, 2021).

A determinant-based reformulation expresses the $k$-th orthogonal vector as a ratio of Gram matrix minors: $\mathbb{R}^m$0 where $\mathbb{R}^m$1 is the Gram matrix and $\mathbb{R}^m$2 the lower-right cofactor (Wang, 2021). This algebraic viewpoint connects GS directly to the generation of classical orthogonal polynomials and links the process to exterior algebra via the Hodge star, where the orthogonal complement is interpreted as dualization of wedge products.

2. Variants in Extended Structures and Non-Standard Inner Products

p-adic and Clifford Hilbert Modules

Generalizations of GS to $\mathbb{R}^m$3-adic normed spaces and Clifford Hilbert modules address orthogonalization in highly non-Archimedean and non-commutative environments.

• p-adic GS replaces the Euclidean inner product by a $\mathbb{R}^m$4-adic norm $\mathbb{R}^m$5 with ultrametric properties. Orthogonality is characterized by $\mathbb{R}^m$6. The algorithmic construction iteratively solves closest vector problems (CVP) in $\mathbb{R}^m$7-adic lattices, producing $\mathbb{R}^m$8-orthogonal bases with termination guaranteed by the finite values of the norm on compact sets. Simultaneous orthogonalization with respect to two norms extends the process for cryptographic applications (Deng, 2023).
• Clifford Hilbert modules admit GS in right modules over Clifford algebras $\mathbb{R}^m$9. Here, challenges include non-commutativity and non-invertibility of Clifford numbers. The projections involve inverses of Gram-diagonal terms $\mathbb{C}^m$0, with the process producing orthonormal families $\mathbb{C}^m$1 satisfying $\mathbb{C}^m$2. The existence of orthogonal projectors and diagonal invertibility is rigorously proven; the process reduces to the commutative (real/complex) case for $\mathbb{C}^m$3 or $\mathbb{C}^m$4 (Wang et al., 2021).

Non-Standard Inner Products

For weighted inner products $\mathbb{C}^m$5 with $\mathbb{C}^m$6 symmetric positive-definite, MGS (Modified Gram–Schmidt) and its efficient variants (MGS–HA and MGS–HP) minimize required matrix-vector products (MVs). Numerical error bounds demonstrate that both efficient variants achieve loss of $\mathbb{C}^m$7-orthogonality at $\mathbb{C}^m$8, matching the naive method's accuracy but with halved or batched MVs and improved computational performance (Imakura et al., 2017).

3. Numerical Stability, Block Variants, and Low Synchronization

Reduction of synchronizations is crucial for scalability in high-performance environments. Block Classical Gram–Schmidt (BCGS) divides vectors into blocks, processing them in batches to reduce communication and synchronization costs.

Stability analysis demonstrates a trade-off between synchronization count and loss of orthogonality (LOO):

Variant	Syncs/Block	LOO Bound	Conditioning Threshold
Four-sync BCGS–I+A	4	$\mathbb{C}^m$9	$\{u_1,\dots,u_n\}$0
Three-sync	3	$\{u_1,\dots,u_n\}$1	$\{u_1,\dots,u_n\}$2
Two/one-sync	2/1	$\{u_1,\dots,u_n\}$3	$\{u_1,\dots,u_n\}$4

A key result is that only the first block needs a strong intra-block QR (e.g., Householder), and subsequent intraorthogonalization may be cheaper. Removing synchronizations beyond three per block incurs a $\{u_1,\dots,u_n\}$5 or $\{u_1,\dots,u_n\}$6 penalty in orthogonality (Carson et al., 2024).

Low synchronization variants (e.g., BCGSI+LS) that achieve just one global reduction per block can easily degrade to $\{u_1,\dots,u_n\}$7 loss, which is inadmissible for ill-conditioned problems (Oktay et al., 2022, Carson et al., 2024).

4. Remedies: Mixed Precision and Reorthogonalization

For ill-conditioned blocks, mixed precision variants (e.g., BCGSI+LS–MP) restore backward stability by computing critical reductions and Cholesky/triangular solves in quadruple (high) precision, while maintaining other steps in standard precision. This innovation empirically restores the $\{u_1,\dots,u_n\}$8 bound for loss of orthogonality (Oktay et al., 2022).

Reorthogonalization (double-pass variants such as BCGS-PIP+, interleaved approaches such as PIP I+) further suppresses LOO. Theoretical and empirical evidence show that, up to $\{u_1,\dots,u_n\}$9, such methods attain machine-level orthogonality. For higher condition numbers, mixed-precision approaches remain robust, provided high-precision is allocated to the inner product and Cholesky steps (Carson et al., 2024, Oktay et al., 2022).

5. High-Dimensional and Randomized Methods

In ultra-high dimensional settings, Gram–Schmidt is integrated into variable selection (e.g., GSFR) and randomized sketching algorithms.

• Forward Regression based on Gram–Schmidt orthogonalization (GSFR) orthogonalizes candidate predictors against those already selected, efficiently measuring each variable's unique contribution to residuals with direct theoretical guarantees (convergence rate, sure screening, model selection consistency), and practical computational advantages ($$u_1 = v_1,\qquad u_k = v_k - \sum_{j=1}^{k-1} \frac{\langle v_k, u_j\rangle}{\langle u_j, u_j\rangle}u_j$$0 total for $$u_1 = v_1,\qquad u_k = v_k - \sum_{j=1}^{k-1} \frac{\langle v_k, u_j\rangle}{\langle u_j, u_j\rangle}u_j$$1) (Chen et al., 7 Jul 2025).
• Randomized Gram–Schmidt (RGS) uses random sketching: inner products are estimated via projections onto lower-dimensional random subspaces. Orthogonality is enforced only approximately at the sketch level, but with high-probability subspace embedding guarantees. In multi-precision contexts, backward stability is obtained without explicit dependence on ambient dimension. RGS enables efficient orthogonalization and integrates seamlessly with Krylov solvers such as randomized GMRES (Balabanov et al., 2020).
• Approximate Orthonormalization: For nearly orthogonal data (e.g., Gaussian matrices with i.i.d. columns), column-normalization alone yields a basis close to true orthogonality, allowing computational savings and error control $$u_1 = v_1,\qquad u_k = v_k - \sum_{j=1}^{k-1} \frac{\langle v_k, u_j\rangle}{\langle u_j, u_j\rangle}u_j$$2 in projection energy estimates, directly relevant for compressed subspace embedding applications (Li et al., 2016).

6. Generalizations and Applications

Dimensional Lifting

The generalized Gram–Schmidt process via dimensional lifting constructs exact orthogonalization for any finite set (even linearly dependent or repeated) by lifting vectors to a higher-dimensional space ($$u_1 = v_1,\qquad u_k = v_k - \sum_{j=1}^{k-1} \frac{\langle v_k, u_j\rangle}{\langle u_j, u_j\rangle}u_j$$3 for $$u_1 = v_1,\qquad u_k = v_k - \sum_{j=1}^{k-1} \frac{\langle v_k, u_j\rangle}{\langle u_j, u_j\rangle}u_j$$4 vectors in $$u_1 = v_1,\qquad u_k = v_k - \sum_{j=1}^{k-1} \frac{\langle v_k, u_j\rangle}{\langle u_j, u_j\rangle}u_j$$5) in which orthogonal projections recover the originals, and the lifted set is exactly orthogonal. The construction yields an explicit, lower-unit-triangular correction in the auxiliary dimensions. Applications include quantum state discrimination, projective measurements, and quantum speedup algorithms, where exact orthogonality of non-orthogonal states is crucial (Havlicek et al., 2016).

Non-Normalized and Inversion-Free OLS

Non-normalized Gram–Schmidt (SGSO) establishes a direct correspondence with pivot-free LU factorizations of the Gram matrix, facilitating closed-form, inversion-free computation of OLS regression coefficients. The process produces dual vectors $$u_1 = v_1,\qquad u_k = v_k - \sum_{j=1}^{k-1} \frac{\langle v_k, u_j\rangle}{\langle u_j, u_j\rangle}u_j$$6 and avoids matrix inversion or normalization, relying instead on simple backward or forward recursions applied to the unnormalized orthogonal system. This perspective generalizes to weighted least squares and fundamental regression identities (Madar et al., 2023).

---

References:

• (Havlicek et al., 2016) Dimensional lifting through generalized Gram-Schmidt process
• (Li et al., 2016) Approximation of Gram-Schmidt Orthogonalization by Data Matrix
• (Imakura et al., 2017) Efficient implementations of the modified Gram-Schmidt orthogonalization with a non-standard inner product
• (Balabanov et al., 2020) Randomized Gram-Schmidt process with application to GMRES
• (Wang et al., 2021) Orthogonalization in Clifford Hilbert modules and applications
• (Wang, 2021) A Derivation of Classical Orthogonal Polynomials using Generalized Vandermonde Determinants
• (Oktay et al., 2022) Using Mixed Precision in Low-Synchronization Reorthogonalized Block Classical Gram-Schmidt
• (Madar et al., 2023) Solving The Ordinary Least Squares in Closed Form, Without Inversion or Normalization
• (Deng, 2023) On $$u_1 = v_1,\qquad u_k = v_k - \sum_{j=1}^{k-1} \frac{\langle v_k, u_j\rangle}{\langle u_j, u_j\rangle}u_j$$7-adic Gram-Schmidt Orthogonalization Process
• (Carson et al., 2024) Reorthogonalized Pythagorean variants of block classical Gram-Schmidt
• (Carson et al., 2024) On the loss of orthogonality in low-synchronization variants of reorthogonalized block classical Gram-Schmidt
• (Chen et al., 7 Jul 2025) Forward Variable Selection in Ultra-High Dimensional Linear Regression Using Gram-Schmidt Orthogonalization