Null-Space Projection

• Null-space projection is a mathematical operation that projects vectors onto the orthogonal complement of a specified subspace, preserving invariants and preventing interference.
• It is widely applied in continual learning, signal recovery, and robotics to enforce constraints and mitigate issues like catastrophic forgetting.
• The method leverages algebraic formulations via the Moore–Penrose pseudoinverse and SVD, offering scalable and precise solutions in high-dimensional settings.

Null-space projection is a foundational mathematical operation that constrains vectors or transformations so their effects are annihilated by a specified linear operator, or more generally, so they lie in the orthogonal complement of a given subspace. In applied mathematics, signal processing, machine learning, robotics, optimization, and high-dimensional statistics, null-space projections are employed to enforce constraints, avoid interference, preserve invariants, and explicitly separate information or control across subspaces defined by prior knowledge, task structure, or data-driven features.

1. Algebraic Formulation and Construction

Let $A\in\mathbb{R}^{m\times n}$ be a real matrix of full row rank. The (right) null space $N(A)$ is defined as

$N(A) = \{x\in\mathbb{R}^n : A x = 0 \}$

The orthogonal projector onto $N(A)$ is given by

$P_{\mathrm{null}} = I_n - A^\dagger A$

with $A^\dagger = A^T (A A^T)^{-1}$ the Moore–Penrose pseudoinverse (assuming $A$ has full row rank), so that $A P_{\mathrm{null}} = 0$, $P_{\mathrm{null}}^2 = P_{\mathrm{null}}$.

If $A$ is not full rank, one generally uses the SVD $A = U \Sigma V^T$, in which case the projector can be constructed as $P_{\mathrm{null}} = V_2 V_2^T$ where $V_2$ are the columns of $V$ corresponding to zero singular values. Thresholding on singular values is routinely employed to tolerate numerical rank deficiency or to adapt to practical noise (Peng et al., 26 Jul 2025).

For general sets of directions $\{a_j\}_{j=1}^m$, one stacks them row-wise as $A$, and $P_{\mathrm{null}}$ then projects any vector $x$ onto the subspace orthogonal to all $a_j$.

2. Null-Space Projection in Continual Learning

Null-space projection has become central in continual and multi-task learning, where the major challenge is to prevent interference (catastrophic forgetting) between sequentially presented tasks. The paradigm is: after absorbing task $t$, encode accumulated knowledge (typically as a set of task gradients or feature directions) in $G_{\mathrm{prev}}$, then constrain subsequent update directions to lie in $N(G_{\mathrm{prev}})$ (Peng et al., 26 Jul 2025, Cheng et al., 2024, Lu et al., 2024).

In gradient-based continual learning, the update at each step is replaced by its projection: $$\tilde{g} = P_{\mathrm{null}}\,g,\quad P_{\mathrm{null}} = I_n - G_{\mathrm{prev}}^T (G_{\mathrm{prev}} G_{\mathrm{prev}}^T)^{-1} G_{\mathrm{prev}}$$ This mechanism ensures that the loss on all previous tasks is preserved to first order, as parameter moves have no component along directions that modify previously learned knowledge (Peng et al., 26 Jul 2025).

Visual prompt-tuning in transformers (Lu et al., 2024) and Mamba-CL’s state space continual learning (Cheng et al., 2024) both extend null-space projections to enforce multi-dimensional orthogonality constraints, projecting parameter or prompt updates orthogonally to accumulated subspaces defined by the self-attention or recurrent feature statistics of prior tasks.

Empirically, these projections sharply decrease forgetting without rehearsal or architecture changes, and can even be shown (under certain consistency conditions) to make the output on all old-task inputs mathematically invariant across future learning steps (Peng et al., 26 Jul 2025, Cheng et al., 2024).

3. Null-Space Projection in Model Editing, Quantization, and Fairness

Null-space projection affords exact preservation of certain behaviors or invariants during model adaptation:

• Model Editing: When performing locate-then-edit interventions to alter LLM output, null-space projection guarantees that the edited weight perturbations affect only the desired new facts, with no impact on preserved knowledge. Formally, parameter updates $\Delta W$ are projected as $\Delta W\,P_{\mathrm{null}}$ where $P_{\mathrm{null}}$ is built from the covariance of all key vectors whose associations must remain unchanged. This yields exact recall-invariance for all preserved key-value pairs (Fang et al., 2024).
• Adapter-Based Fine-Tuning (LoRA): LoRA-Null (Tang et al., 4 Mar 2025) initializes adapters in the null space of pre-trained activations to guarantee that fine-tuning-induced $\Delta W$ do not disturb the original world-knowledge directions, achieving both improved preservation and fine-tuning quality.
• Post-Training Quantization: The Q2N method (Zhao et al., 21 May 2025) constrains the quantization-induced perturbation of weights to lie within the null space of input activations, thereby ensuring that quantization noise does not impact the model’s outputs on calibration (or important) samples, with a scalable spectral approximation for high-dimensional LLM layers.
• Fairness and Representation Purification: Iterative Null-space Projection (INLP) (Ravfogel et al., 2020) repeatedly projects out directions predictive of protected attributes (e.g., gender in embeddings) so that subsequent linear classifiers cannot recover them. Each iteration nulls the row-space spanned by successive classifiers, and the procedure halts once protected-attribute information is empirically absent.

4. Signal Recovery, Inverse Problems, and Constraint Enforcement

The null space of a forward (sensing) operator $A$ in underdetermined systems encodes the set of indistinguishable solutions, and its projection provides an analytic route to inject prior or constraint-satisfying corrections.

• Imaging Inverse Problems: NPN regularizers (Jacome et al., 2 Oct 2025) restrict recovery algorithms to add only a low-dimensional, learned, nonlinear component in $N(A)$, preserving data fidelity while using a learned prior to select among infinite null-space-compatible candidates. The projection $P_{\mathrm{null}} = I - A^\dagger A$ or more generally $P_S = S^\dagger S$ for a learned subspace $S$ is used to enforce that regularization acts only on the invisible (unobservable) components.
• Sparse Signal Recovery: Iterative Null-space Projection with Adaptive Thresholding (Esmaeili et al., 2016) alternates support-thresholded least-squares with explicit null-space projection to the affine constraint set $\{x : \Phi x = y\}$, yielding provable recovery down to the algebraic sample bound.
• Physics-Informed Learning: In the context of Extreme Learning Machines, exact enforcement of boundary or initial conditions is achieved by formulating all admissible parameters as $\beta_* + W\,\eta$ with $W$ a basis for $N(B_{\mathrm{bc}})$, where $B_{\mathrm{bc}}$ encodes boundary constraints. This turns the constrained learning problem into unconstrained optimization within the null space (Mishra et al., 16 Jan 2026).

5. Control Theory and Robotics: Hierarchical and Constrained Control

Null-space projections underpin operational-space robotic control, constrained motion planning, and hierarchical task execution:

• Operational-Space Control: The classic null-space formulation partitions the action $\dot{q}$ into a task-performing component $J^\dagger b$ and a null-space component $N\,\pi$, where $N=I-J^\dagger J$ ensures $\dot{q}$ does not interfere with the primary task (Lin et al., 2016).
• Hierarchical Control: In multiple-priority task execution, each lower-priority control input is projected by a cumulative null-space projector, recursively built to guarantee that all higher-priority tasks are unaffected. For two tasks:

$u = u_1 + (I - J_1^\# J_1) u_2$

For $K$ tasks, $u = \sum_k N_k u_k$ with $N_k$ inductively defined (Patil et al., 28 Mar 2025).

• Constrained Learning from Data: In cases where the constraint geometry is unknown, null-space projection matrices can be inferred directly from observed state-action data by optimizing for projectors that preserve observed null-space components and annihilate task-space components (Lin et al., 2016).

6. Generalizations and Theoretical Extensions

Null-space projection can be formulated with respect to generalized notions of orthogonality and convexity. For a strongly quasiconvex function $\varphi$, the $\varphi$-projection onto a closed convex cone $C$ admits a null-space equal to the set $\{x : P_C(x)=0\}$, generalizing the Euclidean orthogonal projector to nonlinear settings. Key properties (such as when this kernel is itself a convex cone) are fully characterized in terms of the geometry of $\varphi$'s unit sphere—specifically, coherence of meridian arcs—and dual cone/polar mapping (Németh et al., 29 Apr 2025).

A similar perspective informs adaptive and soft variants of null-space projection (e.g., soft NP-LoRA (Chen et al., 14 Nov 2025)), where the projection is relaxed to trade off strict orthogonality against preservation of multiple competing subspaces.

7. Computational Methods, Approximate Projection, and Scalability

While the analytic form $P_{\mathrm{null}} = I - A^\dagger A$ is theoretically clean, high-dimensional applications (e.g., LLMs with hidden size $d\approx 10^4$) necessitate scalable approximations:

• SVD/QR Decomposition: Null spaces are routinely constructed from right/left singular vectors with singular values below a practical threshold, or via QR decomposition for efficiency (Peng et al., 26 Jul 2025, Zhao et al., 21 May 2025).
• Prefix–Suffix Sum Ratio/Adaptive Thresholding: To set the effective dimension of the null space, methods select a threshold on eigenvalue decay so that only directions with negligible energy in the data are considered; this balances preservation with usable parameter flexibility (Peng et al., 26 Jul 2025, Zhao et al., 21 May 2025).
• Online and Incremental Updates: Projectors can be updated with growing feature sets by re-computing or incrementally growing the null-space basis (Zimmer, 2013).

Below is a summary table of the most common null-space projector forms appearing in the literature:

Null-space Context	Projector Formula	Reference
General matrix $A$	$P=I - A^\dagger A$	(Jacome et al., 2 Oct 2025, Zhao et al., 21 May 2025)
Row/col-space $G_{\rm prev}$	$$P_{\rm null}=I - G_{\rm prev}^T (G_{\rm prev} G_{\rm prev}^T)^{-1} G_{\rm prev}$$	(Peng et al., 26 Jul 2025)
Learned subspace $V_k$	$P=I - V_kV_k^T$	(Chen et al., 14 Nov 2025)
Multiple classifiers $W$	$P = I - W^T (W W^T)^{-1} W$	(Ravfogel et al., 2020)
Quasiconvex $\varphi$	$P_f(x) = \arg\min_{y\in C} \varphi(x-y)$, null$=\{x:P_f(x)=0\}$	(Németh et al., 29 Apr 2025)

The above forms underpin the domain-specific adaptations and applications throughout optimization, control, learning, and signal recovery.

---

References:

• "GNSP: Gradient Null Space Projection for Preserving Cross-Modal Alignment in VLMs Continual Learning" (Peng et al., 26 Jul 2025)
• "NPN: Non-Linear Projections of the Null-Space for Imaging Inverse Problems" (Jacome et al., 2 Oct 2025)
• "LoRA-Null: Low-Rank Adaptation via Null Space for LLMs" (Tang et al., 4 Mar 2025)
• "Boost Post-Training Quantization via Null Space Optimization for LLMs" (Zhao et al., 21 May 2025)
• "NPAT Null-Space Projected Adversarial Training Towards Zero Deterioration" (Hu et al., 2024)
• "Visual Prompt Tuning in Null Space for Continual Learning" (Lu et al., 2024)
• "NP-LoRA: Null Space Projection Unifies Subject and Style in LoRA Fusion" (Chen et al., 14 Nov 2025)
• "About the kernel of the strongly quasiconvex function generated projection" (Németh et al., 29 Apr 2025)
• "AlphaEdit: Null-Space Constrained Knowledge Editing for LLMs" (Fang et al., 2024)
• "Outlier Detection through Null Space Analysis of Neural Networks" (Cook et al., 2020)
• "Exact Constraint Enforcement in Physics-Informed Extreme Learning Machines using Null-Space Projection Framework" (Mishra et al., 16 Jan 2026)
• "Mamba-CL: Optimizing Selective State Space Model in Null Space for Continual Learning" (Cheng et al., 2024)
• "Task Hierarchical Control via Null-Space Projection and Path Integral Approach" (Patil et al., 28 Mar 2025)
• "Improving Prediction Certainty Estimation for Reliable Early Exiting via Null Space Projection" (He et al., 8 Jun 2025)
• "Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection" (Ravfogel et al., 2020)
• "Learning Null Space Projections in Operational Space Formulation" (Lin et al., 2016)
• "Iterative Null-space Projection Method with Adaptive Thresholding in Sparse Signal Recovery and Matrix Completion" (Esmaeili et al., 2016)
• "A row space method for solving a system of linear equations" (Zimmer, 2013)
• "Teleportation With Null Space Gradient Projection for Optimization Acceleration" (Wu et al., 17 Feb 2025)