Activation-based Null-Space Initialization (LoRA-Null)

• LoRA-Null is a principled method that leverages the null space of activations to preserve pre-trained model outputs while enabling efficient parameter adaptation.
• It employs SVD-based null space estimation and low-rank adapter initialization to control interference and maintain learned representations.
• Empirical results in continual learning and LLM adaptation validate its effectiveness, offering theoretical guarantees and robust knowledge retention.

Activation-based Null-Space Initialization (LoRA-Null) is a principled method for initializing low-rank adapters in neural network layers by leveraging the null space of input activations associated with pre-existing knowledge. This approach is designed to preserve the original behavior of a pre-trained model—particularly its world knowledge or previously acquired representations—while enabling parameter-efficient fine-tuning and continual learning. LoRA-Null has been independently developed in both LLM adaptation (Tang et al., 4 Mar 2025) and continual learning settings (Pham et al., 25 Feb 2026), offering strong empirical performance and theoretical guarantees on knowledge retention and minimal interference.

1. Mathematical Foundation and Null Space Construction

At the core of LoRA-Null is the exploitation of the null space of activations associated with a given neural network layer. For a layer with pre-trained weight matrix $$W_0 \in \mathbb{R}^{d_\mathrm{out} \times d_\mathrm{in}}$$ and a matrix of input activations $X \in \mathbb{R}^{d_\mathrm{in} \times N}$ (with $N$ sampled tokens or data points), the method proceeds by:

• Computing the singular value decomposition (SVD) $X = U \Sigma V^\top$, where $U$ and $V$ are orthogonal and $\Sigma$ is diagonal.
• Identifying the null space of $X$, which is the subspace corresponding to singular values below a threshold or exactly zero.
  • In standard settings, columns of $V$ corresponding to zero singular values span the null space $\mathcal{N} = \{v \mid Xv = 0\}$.
  • In continual learning, small (not necessarily zero) singular values are used to approximate a near-null subspace, controlling for a threshold $X \in \mathbb{R}^{d_\mathrm{in} \times N}$0 relative to the Frobenius norm of $X \in \mathbb{R}^{d_\mathrm{in} \times N}$1 such that $X \in \mathbb{R}^{d_\mathrm{in} \times N}$2 for indices $X \in \mathbb{R}^{d_\mathrm{in} \times N}$3.
• Forming a basis $X \in \mathbb{R}^{d_\mathrm{in} \times N}$4 for the (approximate) null space from the relevant right singular vectors, ensuring $X \in \mathbb{R}^{d_\mathrm{in} \times N}$5 (Pham et al., 25 Feb 2026).

This process isolates directions in the parameter space where changes will have minimal or no effect on the outputs associated with the sampled activations, directly enabling the preservation of pre-existing model behavior.

2. Adapter Parameterization and Initialization

The adapter parametrization in LoRA-Null proceeds as follows:

Component	Notation	Description
Pre-trained weight	$X \in \mathbb{R}^{d_\mathrm{in} \times N}$6	Frozen base matrix after null-space projection
Null-space projector	$X \in \mathbb{R}^{d_\mathrm{in} \times N}$7	Orthogonal projector onto the null space
Adapter initialization	$X \in \mathbb{R}^{d_\mathrm{in} \times N}$8	Adapter weight restricted to null space
Low-rank factorization	$X \in \mathbb{R}^{d_\mathrm{in} \times N}$9	SVD of $N$0 with $N$1

The initialization ensures that at the start of fine-tuning, the layer output for the sampled activations is unchanged, i.e., $N$2 (Tang et al., 4 Mar 2025). In continual learning (NESS), task-specific updates are restricted to lie within the (approximate) null space as $N$3, with new matrices $N$4 of dimension $N$5 for each task. Initialization with $N$6 ensures the adapter is neutral at the outset (Pham et al., 25 Feb 2026).

3. Training Protocols and Regularization

Adapter fine-tuning with LoRA-Null employs objective functions and regularization tailored to minimize forgetting and interference:

• Loss function for task $N$7:

$N$8

where $N$9 indexes layers, $X = U \Sigma V^\top$0 is cross-entropy loss, and $X = U \Sigma V^\top$1 is the weight-decay coefficient (Pham et al., 25 Feb 2026).

• Regularization: Spectral-norm or Frobenius-norm penalties are enforced to bound the influence of adapted parameters on prior activations.
• Freezing strategies:
  • LoRA-Null-v1: Both $X = U \Sigma V^\top$2 and $X = U \Sigma V^\top$3 are trainable.
  • LoRA-Null-v2: $X = U \Sigma V^\top$4 is frozen, only $X = U \Sigma V^\top$5 is trainable, tightly controlling changes in the preserved activation subspace (Tang et al., 4 Mar 2025).
• In continual learning, after optimizing adapters for each task, the weight update is $X = U \Sigma V^\top$6.

This structure allows strictly localized learning capacity for the downstream or new task while maintaining the model’s behavior on previously seen input directions.

4. Theoretical Guarantees and Stability Analyses

LoRA-Null features robust theoretical underpinnings:

• For any prior activation $X = U \Sigma V^\top$7 in the row-space of $X = U \Sigma V^\top$8, the induced norm after adaptation is bounded via the maximal small singular value:

$X = U \Sigma V^\top$9

satisfying stability constraints for all previously encountered data (Pham et al., 25 Feb 2026).

• Knowledge preservation: For LoRA-Null-v2, the output on sampled activations remains exactly or approximately invariant during fine-tuning, since $U$0 by construction; updates to $U$1 do not interfere with these outputs (Tang et al., 4 Mar 2025).
• Column-space alignment: For full-rank $U$2, the column space of the initialized adapter matches the null space, ensuring adapters operate only in invariant directions relative to $U$3.

A plausible implication is that the expressivity of task adaptation versus retention can be directly tuned by adjusting the rank $U$4 of the adapter.

5. Empirical Results Across Domains

Experimental validation confirms LoRA-Null’s efficacy in both continual learning and parameter-efficient LLM fine-tuning:

• Continual learning (NESS) (Pham et al., 25 Feb 2026):
  • CIFAR-100 (10-task): ACC ≈ 72.46% ± 0.26, BWT = +0.03% ± 0.40
  • 5-datasets: ACC ≈ 90.20% ± 0.47, BWT = −0.58% ± 0.15
  • MiniImageNet (20-task): ACC ≈ 63.72% ± 0.46, BWT = +0.41% ± 0.58
  • Backward Transfer (BWT) consistently greater than −1%, outperforming SGP and GPM baselines in forgetting minimization.
• LLM adaptation (Tang et al., 4 Mar 2025):
  • Retention: LoRA-Null matches or surpasses standard LoRA in knowledge benchmarks (TriviaQA, NQ-Open, WebQS), often within 1–2 points of the frozen backbone.
  • Downstream tasks: Comparable or superior performance in math (MetaMathQA → GSM8k/MATH), code (Magicoder → HumanEval/MBPP), and instruction following (WizardLM → MTBench).
  • Freezing $U$5 (LoRA-Null-v2) provides optimal world-knowledge retention; freeing $U$6 and $U$7 (v1) yields higher downstream accuracy with a slight retention trade-off.

Performance is robust to SVD rank selection ($U$8), enabling explicit retention/adaptation trade-off via adapter subspace dimension.

6. Practical Implementation Considerations

Deployment of LoRA-Null is efficient and amenable to standard ML pipelines:

• Activation sampling: Use 200–300 representative tokens per layer; additional samples refine null-space estimation with limited incremental benefit.
• Rank selection: Default $U$9 (LLM); lower $V$0 for retention, higher $V$1 for adaptation.
• Computational cost: Null-space SVD on $V$2 matrices is a one-time, negligible overhead per layer.
• Integration: LoRA-Null is a drop-in module for HuggingFace/PEFT workflows and continual learning libraries. It requires only a single SVD-based projection and adapter replacement per fine-tuned layer.
• Freezing policy: To maximize retention, freeze $V$3 (LoRA-Null-v2); to maximize adaptation capacity, train both $V$4 and $V$5.

These properties ensure LoRA-Null’s applicability for large-scale LLMs and continual learning systems, with minimal impact on training and inference efficiency.

7. Relation to Prior Work and Research Impact

Activation-based Null-Space Initialization distinguishes itself from alternative PEFT and continual learning techniques by directly leveraging the subspace structure of activation data, rather than relying solely on weight or gradient orthogonalization. It formalizes and extends theoretical connections between singular value spectra of layer input representations and catastrophic forgetting, providing unified linear algebraic treatment across task domains.

Compared to methods such as SGP, GPM, PiSSA, CorDA, and MiLoRA, LoRA-Null achieves the strongest balance of pre-trained knowledge retention and adaptive capacity (Tang et al., 4 Mar 2025, Pham et al., 25 Feb 2026). Its SVD-based basis selection, interpretability, and empirical reliability contribute to its adoption in both academic research and practical downstream fine-tuning pipelines.