Orthogonal Subspace Projection (OSP)

• Orthogonal Subspace Projection (OSP) is a technique that represents a subspace via an orthogonal projector, allowing for retention of structured signals or removal of nuisance components.
• OSP underpins classical subspace geometry, Krylov methods, and modern neural architectures by enabling efficient denoising, perturbation analysis, and adaptation in various scientific applications.
• Although OSP provides precise subspace manipulation, its effectiveness depends on accurate rank estimation and careful separation between signal and noise to avoid unwanted contamination.

Orthogonal Subspace Projection (OSP) denotes a class of methods that represent a linear subspace by its orthogonal projector and then operate either on the projected component or on the orthogonal residual. For a full-column-rank basis $B$, the canonical projector is $P_B = B(B^\top B)^{-1}B^\top$, while for an orthonormal basis $U$ it reduces to $P_U = UU^\top$; the complementary projector $I-P_U$ removes the component lying in the span of $U$ (Liski et al., 2012). In contemporary literature, this core construction appears in classical subspace geometry, Krylov methods, subspace averaging, perturbation analysis, and a wide range of modern ML systems that treat nuisance, capability, semantic, or adaptation directions as subspaces to be preserved, rotated, or excised (Wang et al., 17 Jan 2026).

1. Linear-algebraic basis

An orthogonal projector is characterized by the identities $P=P^\top=P^2$, and it provides a unique representation of a subspace through $\mathcal{S}_P=\{Px:x\in\mathbb{R}^p\}$ (Liski et al., 2012). In matrix terms, the projector onto the column space of $A$ can be written as $P_A=AA^\dagger$, with $P_B = B(B^\top B)^{-1}B^\top$0 the Moore–Penrose pseudoinverse; when $P_B = B(B^\top B)^{-1}B^\top$1 has full column rank, this reduces to the familiar $P_B = B(B^\top B)^{-1}B^\top$2 form (Xu, 2018). The complementary operator $P_B = B(B^\top B)^{-1}B^\top$3 is the standard OSP object for nulling a protected or nuisance subspace.

This projector formalism supports two symmetric viewpoints. One may project onto a desired subspace, retaining only structured signal components, or project onto the orthogonal complement of an undesired subspace, suppressing interference while preserving whatever is orthogonal to it. Both views appear explicitly in later applications: NBNet projects noisy features onto a learned signal subspace, whereas SeLop removes a learned nuisance span via $P_B = B(B^\top B)^{-1}B^\top$4 (Cheng et al., 2020, Wang et al., 17 Jan 2026).

A related formulation appears in Krylov solvers. For the linear system $P_B = B(B^\top B)^{-1}B^\top$5, an orthogonal projection method enforces the Galerkin condition $P_B = B(B^\top B)^{-1}B^\top$6, where $P_B = B(B^\top B)^{-1}B^\top$7 is the $P_B = B(B^\top B)^{-1}B^\top$8-step Krylov subspace. For SPD $P_B = B(B^\top B)^{-1}B^\top$9, this is equivalent to best approximation in the $U$0-norm over that subspace, so OSP here is not a nuisance-removal heuristic but the defining variational principle of the solver (Timsit et al., 2023).

2. Subspace geometry, comparison, and estimation

Because a projector uniquely encodes a subspace, projector geometry provides a natural language for comparing and combining subspace estimates. The weighted Crone–Crosby distance

$U$1

extends subspace comparison to different ranks and leads to the average orthogonal projector, obtained from the eigendecomposition of a weighted mean projector $U$2 (Liski et al., 2012). This gives a principled consensus construction when multiple projector estimates are available.

Projector perturbation theory makes explicit how OSP inherits conditioning from its basis matrix. For $U$3 and $U$4, the Frobenius error $U$5 admits exact identities and sharp upper and lower bounds in terms of $U$6, $U$7, $U$8, and the pseudoinverse perturbation $U$9 (Xu, 2018). A particularly important structural fact is that rank changes impose a nonzero lower bound: $P_U = UU^\top$0 where $P_U = UU^\top$1 and $P_U = UU^\top$2. This makes rank preservation a necessary condition for continuity of projector estimates.

The same geometric language supports subspace selection. Under matroid constraints, the objective

$P_U = UU^\top$3

seeks a subset whose span captures maximal projected energy of a target vector $P_U = UU^\top$4. Forward regression greedily maximizes one-step projected-energy gain, and OMP greedily maximizes residual correlation, with guarantees controlled by elemental curvatures and principal angles (Zhang et al., 2015). In the mutually orthogonal case, these greedy rules become optimal under a uniform matroid and achieve a $P_U = UU^\top$5-approximation under a non-uniform matroid (Zhang et al., 2015).

A further generalization appears in wideband array processing, where the signal-subspace projector $P_U = UU^\top$6 is treated as a smooth matrix-valued function of frequency and approximated by a polynomial $P_U = UU^\top$7 (Selva, 2017). This moves OSP from a static matrix to a structured projector field and improves IC-MUSIC and MTOPS by regularizing subspace estimates across frequency (Selva, 2017).

3. Explicit OSP inside learned representations

Several recent neural architectures implement OSP in a mathematically direct sense. NBNet’s SSA module learns basis vectors $P_U = UU^\top$8 from feature maps and then applies the exact orthogonal projector

$P_U = UU^\top$9

so denoising is realized as projection of noisy features onto a learned signal subspace (Cheng et al., 2020). The basis is not orthonormalized during learning; orthogonality is handled analytically through the projector formula itself.

SeLop makes the OSP structure even more explicit in a frozen CLIP visual encoder. At selected middle and deep transformer layers, it learns a skinny matrix $I-P_U$0, orthogonalizes it by QR to obtain $I-P_U$1, forms $I-P_U$2, and replaces visual tokens $I-P_U$3 by

$I-P_U$4

The intended nuisance span is a low-rank subspace of forgery-irrelevant factors, justified by a local linearization $I-P_U$5 with $I-P_U$6 and, with domain variables, $I-P_U$7 (Wang et al., 17 Jan 2026). The method uses only cross-entropy on the final $I-P_U$8 token, with no explicit supervision for $I-P_U$9 or $U$0, and only $U$1M trainable parameters (Wang et al., 17 Jan 2026).

Its strongest diagnostic evidence is the counterfactual validation. Using the learned nuisance-only representation $U$2 yields near-random AUCs around $U$3 to $U$4, whereas the orthogonal-complement representation $U$5 yields frame-level AUCs of $U$6 on CDF-v1, $U$7 on CDF-v2, $U$8 on DFDC, $U$9 on DFDCP, and $P=P^\top=P^2$0 on DFD (Wang et al., 17 Jan 2026). This is one of the clearest recent examples of OSP as learned nuisance-subspace removal rather than fixed preprocessing.

4. OSP in PEFT, continual learning, and safety alignment

Modern PEFT and CL work has extended OSP from feature space to weight, adapter, gradient, and activity spaces, but the exact relation to classical OSP varies substantially.

OPLoRA is a strict projector-based construction. Given the SVD $P=P^\top=P^2$1, it defines

$P=P^\top=P^2$2

and constrains the LoRA update to $P=P^\top=P^2$3 (Xiong et al., 14 Oct 2025). This double-sided projection exactly preserves the top-$P=P^\top=P^2$4 singular triples: $P=P^\top=P^2$5 making OSP a hard knowledge-preservation mechanism rather than a regularizer (Xiong et al., 14 Oct 2025).

Continual machine unlearning uses an explicitly cumulative complement projector. After each task, the top left singular vectors $P=P^\top=P^2$6 of the learned LoRA update define newly occupied input-side directions, and the residual projector is updated as

$P=P^\top=P^2$7

The next task is trained with $P=P^\top=P^2$8, so optimization is constrained to the orthogonal complement of earlier tasks throughout training (Rahulamathavan et al., 14 Apr 2026). In the reported CIFAR-100 setting, this preserves retained accuracy near baseline across long unlearning sequences while static fusion collapses from $P=P^\top=P^2$9 to $\mathcal{S}_P=\{Px:x\in\mathbb{R}^p\}$0 at $\mathcal{S}_P=\{Px:x\in\mathbb{R}^p\}$1 and $\mathcal{S}_P=\{Px:x\in\mathbb{R}^p\}$2 (Rahulamathavan et al., 14 Apr 2026).

OGPSA applies the same idea in gradient space. A low-rank capability subspace $\mathcal{S}_P=\{Px:x\in\mathbb{R}^p\}$3 is estimated from small reference gradients, orthonormalized by Gram–Schmidt, and each safety gradient is replaced by

$\mathcal{S}_P=\{Px:x\in\mathbb{R}^p\}$4

This is justified by the first-order preservation condition $\mathcal{S}_P=\{Px:x\in\mathbb{R}^p\}$5, and on Qwen2.5-7B-Instruct under sequential SFT$\mathcal{S}_P=\{Px:x\in\mathbb{R}^p\}$6DPO it improves SimpleQA from $\mathcal{S}_P=\{Px:x\in\mathbb{R}^p\}$7 to $\mathcal{S}_P=\{Px:x\in\mathbb{R}^p\}$8 and IFEval from $\mathcal{S}_P=\{Px:x\in\mathbb{R}^p\}$9 to $A$0 while preserving strong safety (Sun et al., 8 Feb 2026).

HLOP realizes the same complement projection with lateral circuits and Hebbian/anti-Hebbian learning. Instead of explicitly projecting the full gradient matrix, it projects the presynaptic trace: $A$1 On PMNIST, this yields ACC/BWT of $A$2 for DSR+HLOP versus $A$3 for the DSR baseline, and analogous gains for BPTT+SG and OTTT (Xiao et al., 2024).

Not every PEFT paper using the label “OSP” implements a classical projector. OrthoTryOn inserts task-specific orthogonal bottleneck rotations $A$4 inside shared LoRA modules,

$A$5

and proves decorrelation of task-specific weight increments in expectation, with expected gradient interference decaying as $A$6 (Yang et al., 26 Jun 2026). OoPk, despite its title, does not construct $A$7 or $A$8; it uses a low-rank update $A$9 with soft orthogonality loss $P_A=AA^\dagger$0 and is therefore better described as an orthogonally regularized adaptation space than as classical OSP (Li et al., 23 Jun 2025).

5. Semantic and physiological projections

OSP has also been specialized to domains where the protected or removed subspace is semantic or physiological rather than purely algebraic.

In robust interpretability for VLMs, Orthogonal Semantic Projection constructs a distractor dictionary $P_A=AA^\dagger$1 of text embeddings and uses the OMP residual

$P_A=AA^\dagger$2

as the purified query (Bilgiç et al., 8 Jun 2026). This is an orthogonal-complement projection onto the residual semantic direction after greedy distractor selection. The paper reports consistent AUROC gains on ImageNet-Segmentation, including $P_A=AA^\dagger$3 for LeGrad+CLIP, $P_A=AA^\dagger$4 for CheferCAM+CLIP, $P_A=AA^\dagger$5 for GradCAM+SigLIP, and $P_A=AA^\dagger$6 for DAAM+Stable Diffusion 2 (Bilgiç et al., 8 Jun 2026).

In electrodermal activity decomposition, ospEDA builds a lag matrix $P_A=AA^\dagger$7 from delayed versions of an initial tonic estimate and re-estimates tonic by projecting the measured signal onto the tonic subspace: $P_A=AA^\dagger$8 or $P_A=AA^\dagger$9 when regularization is needed, with $P_B = B(B^\top B)^{-1}B^\top$00 selected by MDL (Lee et al., 8 Apr 2026). The phasic component is then the residual. On simulated data, ospEDA achieves tonic/phasic RMSE of $P_B = B(B^\top B)^{-1}B^\top$01 at $P_B = B(B^\top B)^{-1}B^\top$02 dB SNR, and at $P_B = B(B^\top B)^{-1}B^\top$03 dB SNR it attains phasic RMSE $P_B = B(B^\top B)^{-1}B^\top$04, Pearson correlation $P_B = B(B^\top B)^{-1}B^\top$05, and $P_B = B(B^\top B)^{-1}B^\top$06; on real-world datasets it reaches AUROC $P_B = B(B^\top B)^{-1}B^\top$07 and maintains $P_B = B(B^\top B)^{-1}B^\top$08 across all five datasets (Lee et al., 8 Apr 2026).

These examples show that OSP is not confined to nuisance nulling. It can act as semantic residualization, tonic-background extraction, or any operation in which a structured span is estimated and either retained or discarded.

6. Terminology, scope, and recurring limitations

The current literature uses “OSP” with at least four distinct levels of strictness. Some works implement the textbook operator $P_B = B(B^\top B)^{-1}B^\top$09 or its orthonormal specialization $P_B = B(B^\top B)^{-1}B^\top$10; others embed $P_B = B(B^\top B)^{-1}B^\top$11 inside a trainable network; others replace projection by orthogonal rotation; and still others use only soft orthogonality penalties.

Usage	Core operator	Relation to classical OSP
SeLop (Wang et al., 17 Jan 2026)	$P_B = B(B^\top B)^{-1}B^\top$12 on token features	Direct learned complement projection
OPLoRA (Xiong et al., 14 Oct 2025)	$P_B = B(B^\top B)^{-1}B^\top$13 with $P_B = B(B^\top B)^{-1}B^\top$14, $P_B = B(B^\top B)^{-1}B^\top$15	Direct double-sided complement projection
OrthoTryOn (Yang et al., 26 Jun 2026)	$P_B = B(B^\top B)^{-1}B^\top$16, $P_B = B(B^\top B)^{-1}B^\top$17	Orthogonal rotation, not complement projection
OoPk (Li et al., 23 Jun 2025)	$P_B = B(B^\top B)^{-1}B^\top$18	Soft orthogonality, not an explicit projector
Orthogonal Semantic Projection (Bilgiç et al., 8 Jun 2026)	OMP residual $P_B = B(B^\top B)^{-1}B^\top$19	Complement projection after greedy support selection

A recurrent misconception is that any use of orthogonality is OSP. The cited works show otherwise. Classical OSP requires an actual orthogonal projector or orthogonal-complement projector; orthogonality-preserving rotations and soft penalties are related constructions, but they are not identical to projection. A second misconception is that OSP is automatically safe. The recent ML papers repeatedly expose its failure modes: if nuisance rank is underestimated, contamination remains; if it is overestimated, signal is erased; and if the available orthogonal complement saturates, later tasks lose capacity (Wang et al., 17 Jan 2026, Rahulamathavan et al., 14 Apr 2026). This suggests that OSP is most reliable when the unwanted variation is genuinely low-rank, the retained and removed directions are sufficiently separable, and the protected subspace can be estimated stably.

Across these settings, the common invariant is geometric: encode the structured directions by a projector, then restrict downstream computation to either the range of that projector or its orthogonal complement. The contemporary literature has greatly expanded the spaces in which this is done—signal, feature, semantic, weight, gradient, and activity spaces—but the underlying mechanism remains the same.