Gram-Schmidt Orthogonalisation

• Gram-Schmidt orthogonalisation is a process that transforms a set of linearly independent vectors into an orthonormal basis, preserving the original subspace.
• The method underpins various implementations—from classical and modified approaches to block, reorthogonalized, and randomized variants—to address numerical stability and computational efficiency.
• Recent research extends the technique to non-Euclidean settings and applications in Krylov methods, ultra-high dimensional regression, and p-adic lattices.

Gram–Schmidt orthogonalisation is a procedure for transforming a linearly independent family of vectors into an orthogonal or orthonormal family spanning the same subspace and, in matrix form, for producing QR factorizations such as $A=QR$ or $X=QR$, where $Q$ has orthonormal columns and $R$ is upper triangular (Barlow et al., 2011). In current research, the topic extends well beyond the classical columnwise construction to block, reorthogonalized, low-synchronization, randomized, mixed-precision, and non-Euclidean variants, as well as applications in Krylov methods, ultra-high dimensional regression, signal processing, and $p$-adic lattices (Carson et al., 2024, Balabanov et al., 2021, Chen et al., 7 Jul 2025, Deng, 2023).

1. Classical construction and QR factorization

For a matrix $A=[a_1,\dots,a_m]\in\mathbb{R}^{n\times m}$ with linearly independent columns, Gram–Schmidt constructs orthonormal vectors $Q=[q_1,\dots,q_m]\in\mathbb{R}^{n\times m}$ and an upper triangular matrix $R\in\mathbb{R}^{m\times m}$ such that

$A=QR,\qquad Q^TQ=I.$

The geometric core is the orthogonal projection of a new column $a_j$ onto the orthogonal complement of the span of the previous basis vectors: $X=QR$0 with

$X=QR$1

Equivalently,

$X=QR$2

These formulas appear in classical, block, and randomized treatments alike (Balabanov et al., 2021).

The columnwise recurrence is commonly written as

$X=QR$3

$X=QR$4

$X=QR$5

In modified Gram–Schmidt (MGS), the same orthogonalization is performed sequentially on the progressively deflated vector: $X=QR$6 which changes the floating-point behaviour while preserving equivalence in exact arithmetic (Carson et al., 2024).

A block formulation replaces vectors by blocks $X=QR$7 in a partition

$X=QR$8

At block step $X=QR$9, one computes

$Q$0

where $Q$1 is an intraorthogonalization routine such as HouseQR, TSQR, or CholQR (Carson et al., 2024). A related project-and-normalize formulation writes

$Q$2

which recasts orthogonalization as a QR problem on an augmented matrix and underlies several communication-aware algorithms (Dreier et al., 2022).

2. Loss of orthogonality and the role of reorthogonalization

In floating-point arithmetic, Gram–Schmidt is assessed not only by the residual of the QR factorization but also by the quality of orthogonality. For block methods, the loss of orthogonality is measured by

$Q$3

or equivalently $Q$4. Two further stability measures are

$Q$5

the latter being the relative Cholesky residual (Carson et al., 2024).

The standard numerical distinction is that classical Gram–Schmidt computes all inner products against the original vector $Q$6, whereas modified Gram–Schmidt interleaves projection and deflation. The cited analyses state that CGS can suffer severe loss of orthogonality, especially for ill-conditioned matrices, while MGS is typically more stable; reorthogonalization, by running the inner loop twice, often yields still better orthogonality and leads to variants such as CGS2 or MGS2 (Balabanov et al., 2021).

The reorthogonalized block classical Gram–Schmidt algorithm proposed by Barlow and Smoktunowicz factorizes a full column rank matrix $Q$7 into $Q$8 and, with appropriate assumptions on the diagonal blocks of $Q$9, produces factors satisfying

$R$0

and

$R$1

where $R$2 is machine unit roundoff (Barlow et al., 2011). The same source emphasizes that the second projection pass removes the remaining numerical contamination from previously computed subspaces.

A related analysis for symmetric saddle point problems states that $R$3, implemented with Householder QR, is backward stable under a mild assumption on the matrix $R$4: the computed vector $R$5 is the exact solution to a slightly perturbed linear system $R$6 (Okulicka-Dłużewska et al., 2013). In the single-column case $R$7, later block analyses simplify dramatically because $R$8 for the intraorthogonalization subproblems; the resulting corollary proves that DCGS2 and CGS-2 are as stable as Householder QR in the sense of unit-roundoff orthogonality (Carson et al., 2024).

3. Block orthogonalisation, synchronization, and high-performance variants

Modern high-performance work treats Gram–Schmidt as a communication problem as much as an arithmetic one. The abstract framework used for block classical Gram–Schmidt separates each block step into a projection stage and an intraorthogonalization stage: $R$9 The computed projection is modeled as

$p$0

and the computed intraorthogonalization as

$p$1

together with an orthogonality guarantee

$p$2

This reduction allows the stability of an entire block method to be expressed in terms of projection error, QR backward error, and conditioning of the projected block (Carson et al., 2024).

A central result of that analysis is that a “strong” intrablock orthogonalization routine is only needed for the very first block. In the variant called BCGS-A,

$p$3

and for $p$4,

$p$5

If the first block is orthogonalized by a Householder-like routine and later blocks use a weaker but still backward stable routine, then the entire basis achieves orthogonality at the level of roundoff under the stated conditioning assumptions (Carson et al., 2024).

The same paper studies low-synchronization variants obtained from a 4-sync reorthogonalized baseline, BCGSI+A, by removing synchronization points one at a time. The resulting 3-sync, 2-sync, and 1-sync variants have progressively weaker stability guarantees, and the analysis shows that “stability begins to degrade with the first reduction of synchronization points.” The variant requiring only one synchronization per block column “cannot be guaranteed to be stable in practice,” whereas the single-column analogues remain unit-roundoff stable (Carson et al., 2024).

Variant	Synchronizations per block	Bound on $p$6
BCGSI+A	4	$p$7
BCGSI+A-3S	3	$p$8
BCGSI+A-2S	2	$p$9
BCGSI+A-1S	1	$A=[a_1,\dots,a_m]\in\mathbb{R}^{n\times m}$0

A parallel development concerns the “Pythagorean” block variant BCGS-PIP, based on the identity

$A=[a_1,\dots,a_m]\in\mathbb{R}^{n\times m}$1

with

$A=[a_1,\dots,a_m]\in\mathbb{R}^{n\times m}$2

For BCGS-PIP, the loss of orthogonality is bounded by $A=[a_1,\dots,a_m]\in\mathbb{R}^{n\times m}$3 provided $A=[a_1,\dots,a_m]\in\mathbb{R}^{n\times m}$4. Two reorthogonalized variants, BCGS-PIP+ and BCGS-PIPI+, improve this to an $A=[a_1,\dots,a_m]\in\mathbb{R}^{n\times m}$5 bound while retaining asymptotically two synchronization points per block column (Carson et al., 2024).

A hardware-aware formulation integrates these ideas into the project-and-normalize QR view. It leads to BCGS-PIP, BCGS-PIP+, and TSPQR, and the performance model combines a Roofline model for intra-node work with a LogP-style model for inter-node communication. The framework’s practical principle is to combine different orthogonalization algorithms across hardware levels, for example Householder or TSPQR locally and BCGS-PIP+ for reductions (Dreier et al., 2022).

4. Randomized and sketched Gram–Schmidt processes

Randomized Gram–Schmidt replaces exact Euclidean inner products by sketched inner products. With a sketching matrix

$A=[a_1,\dots,a_m]\in\mathbb{R}^{n\times m}$6

or $A=[a_1,\dots,a_m]\in\mathbb{R}^{n\times m}$7, one uses

$A=[a_1,\dots,a_m]\in\mathbb{R}^{n\times m}$8

as an approximate surrogate for $A=[a_1,\dots,a_m]\in\mathbb{R}^{n\times m}$9. If the sketch is an $Q=[q_1,\dots,q_m]\in\mathbb{R}^{n\times m}$0-embedding for a subspace $Q=[q_1,\dots,q_m]\in\mathbb{R}^{n\times m}$1, then for all $Q=[q_1,\dots,q_m]\in\mathbb{R}^{n\times m}$2,

$Q=[q_1,\dots,q_m]\in\mathbb{R}^{n\times m}$3

The single-vector randomized Gram–Schmidt (RGS) orthogonalizes a new vector $Q=[q_1,\dots,q_m]\in\mathbb{R}^{n\times m}$4 by solving a small sketched least-squares problem: $Q=[q_1,\dots,q_m]\in\mathbb{R}^{n\times m}$5

$Q=[q_1,\dots,q_m]\in\mathbb{R}^{n\times m}$6

$Q=[q_1,\dots,q_m]\in\mathbb{R}^{n\times m}$7

The stated motivation is that the expensive operations remain in the original $Q=[q_1,\dots,q_m]\in\mathbb{R}^{n\times m}$8-space, while the coefficient computation is performed in a much smaller $Q=[q_1,\dots,q_m]\in\mathbb{R}^{n\times m}$9-space (Balabanov et al., 2020).

The block generalization, RBGS, partitions

$R\in\mathbb{R}^{m\times m}$0

and computes

$R\in\mathbb{R}^{m\times m}$1

$R\in\mathbb{R}^{m\times m}$2

followed by a block QR factorization

$R\in\mathbb{R}^{m\times m}$3

A low-synchronization RBGS variant can be implemented in parallel “using only one global reduction operation between processors per block,” with the sketched update

$R\in\mathbb{R}^{m\times m}$4

The same work applies RBGS to block Arnoldi, randomized GMRES, randomized FOM, and randomized Rayleigh–Ritz methods (Balabanov et al., 2021).

These randomized formulations are presented as numerically robust alternatives to deterministic Gram–Schmidt. The cited analyses state that RGS can be less computationally expensive than the classical Gram–Schmidt process while being at least as numerically stable as modified Gram–Schmidt, and that stability conditions depend on the reduced dimension $R\in\mathbb{R}^{m\times m}$5 and $R\in\mathbb{R}^{m\times m}$6, but not directly on the ambient dimension $R\in\mathbb{R}^{m\times m}$7 (Balabanov et al., 2020, Balabanov et al., 2021).

5. Generalized inner products and alternative algebraic settings

A substantial generalization concerns non-standard inner products induced by a symmetric positive definite matrix $R\in\mathbb{R}^{m\times m}$8. In that setting, the thin QR factorization is

$R\in\mathbb{R}^{m\times m}$9

with inner product

$A=QR,\qquad Q^TQ=I.$0

Straightforward MGS implementations require $A=QR,\qquad Q^TQ=I.$1 matrix-vector multiplications with respect to $A=QR,\qquad Q^TQ=I.$2, but the paper “Efficient implementations of the modified Gram-Schmidt orthogonalization with a non-standard inner product” proposes high-accuracy and high-performance $A=QR,\qquad Q^TQ=I.$3-MV implementations. The key observation is that every $A=QR,\qquad Q^TQ=I.$4 can be formed using vectors in the ranges of $A=QR,\qquad Q^TQ=I.$5 and $A=QR,\qquad Q^TQ=I.$6, so the expensive action of $A=QR,\qquad Q^TQ=I.$7 can be reduced to a single pass over the columns (Imakura et al., 2017).

Beyond real and complex Euclidean spaces, the paper “On $A=QR,\qquad Q^TQ=I.$8-adic Gram-Schmidt Orthogonalization Process” formulates orthogonality using a norm $A=QR,\qquad Q^TQ=I.$9 on a finite-dimensional vector space over $a_j$0. A decomposition

$a_j$1

into one-dimensional subspaces is $a_j$2-orthogonal if

$a_j$3

The construction is algorithmic rather than projection-based: finding a vector orthogonal to a hyperplane is reduced to a closest vector problem in a $a_j$4-adic lattice, and the resulting algorithms are linked to cryptography based on $a_j$5-adic lattices (Deng, 2023).

Another nonstandard construction is “dimensional lifting through generalized Gram-Schmidt process.” Here an ordered family $a_j$6 is embedded into a higher-dimensional space $a_j$7 by appending carefully chosen extra coordinates so that the lifted vectors become mutually orthogonal. In one convenient representation,

$a_j$8

with the coefficients determined recursively from the orthogonality conditions $a_j$9. The construction is order-dependent and non-unitary, and it allows orthogonalization even for linearly dependent vectors, repeated vectors, and zero vectors (Havlicek et al., 2016).

6. Applications and related constructions

In ultra-high dimensional linear regression, Gram–Schmidt orthogonalization appears as the mechanism behind Gram-Schmidt Forward Regression (GSFR). With centered data, the method orthogonalizes each candidate predictor against the span of the already selected variables and evaluates the standardized unique-contribution score

$X=QR$00

At each stage,

$X=QR$01

The paper states that GSFR is theoretically equivalent to forward regression except for the stopping rule, improves computational efficiency, and in ultra-high dimensional settings achieves a convergence rate comparable to OGA while ensuring variable selection consistency under mild conditions (Chen et al., 7 Jul 2025).

In the theory of orthogonal polynomials, Gram–Schmidt applied to monomials with weighted $X=QR$02 inner products yields classical Hermite, Laguerre, and Jacobi systems. For example, with

$X=QR$03

and starting basis $X=QR$04, the paper derives determinant formulas for the orthogonalized polynomials through generalized Vandermonde determinants with Gamma and Beta function entries. It also presents a geometric formulation via the Hodge star operator, where the orthogonal complement can be expressed through exterior algebra (Wang, 2021).

A different application concerns approximation of orthogonalization itself. For a full column rank matrix

$X=QR$05

with column-normalized version

$X=QR$06

the orthonormalized matrix $X=QR$07 obtained by Gram–Schmidt satisfies

$X=QR$08

The cited analysis shows that if $X=QR$09 is close to orthonormal, then $X=QR$10 is small; this is used to explain why a high dimensional Gaussian matrix can approximate the corresponding truncated Haar matrix, and it is connected to compressed subspace clustering and random projections (Li et al., 2016).

Across these settings, the persistent theme is unchanged: Gram–Schmidt orthogonalisation is the mechanism by which a spanning family is replaced by one whose geometry is structured by orthogonality. What varies is the ambient algebra, the error model, the communication pattern, and the downstream task—whether QR factorization in floating-point arithmetic, Krylov basis generation, variable selection, polynomial construction, or lattice computation.