Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation

Abstract: Catastrophic forgetting in continual adaptation is usually studied through parameter drift, replay, or distillation, but these views do not identify which output-space directions are vulnerable. We give a function-space account in the NTK regime: new-task training induces old-task prediction drift through the cross-task kernel, yielding a closed-form predictor for the forgetting vector before any new-task gradient step. In frozen-backbone linear-head PEFT-CL, where the model is linear in the trainable parameters, the predictor is exact up to numerical precision; for nonlinear adapters/full fine-tuning, it is a local NTK approximation. The same expression reveals that forgetting concentrates in a small number of old-task NTK eigenmodes and under frozen linear heads gives a Kronecker scaling rule for the vulnerable rank. These results clarify the relation to prior NTK-overlap theory, explain why parameter-space regularizers can miss output-space interference, and motivate a targeted spectral regularizer.

• The paper introduces a closed-form predictor for forgetting using NTK linearization, achieving over 0.99 cosine similarity against observed task drift.
• It reveals that catastrophic forgetting is confined to low-dimensional vulnerable subspaces formed by dominant NTK eigenmodes, guiding precise spectral regularization.
• The method outperforms traditional parameter-space approaches, offering actionable insights for continual learning and neural adaptation.

Catastrophic Forgetting as Low-Rank Function-Space Interference: A Theoretical and Empirical Framework

Introduction

Catastrophic forgetting continues to present a critical obstacle in continual and incremental learning, particularly in fine-tuning large pretrained architectures. This paper, "Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation" (2606.18024), establishes a rigorous theoretical framework for analyzing forgetting through the lens of function-space geometry in the Neural Tangent Kernel (NTK) regime. The work departs from parameter-space regularization and proposes precise analytical tools to predict and control the direction and magnitude of forgetting before adaptation occurs, based on kernel-theoretic projections in output space.

Closed-Form Forgetting Prediction in Function Space

A central contribution is a closed-form predictor for the forgetting vector $\Delta f_A$, representing the drift in Task-A predictions induced by adaptation to a new task, B. Leveraging NTK linearization, the predictor utilizes only information available before any gradient step on Task B. In the regime of a frozen backbone and a linear, task-shared head (parameter-efficient fine-tuning with linear readout), the prediction becomes exact modulo numerical precision; for nonlinear adapters or full fine-tuning, it acts as a local approximation. Empirically, this analytic prediction attains cosine similarities with realized drift exceeding $0.99$ on benchmark tasks, thereby confirming the precision of the theoretical model.

Figure 1: Left: Predicted vs.\ realized $\Delta f_A$ demonstrates $\cos>0.99$ on Split-MNIST/CIFAR-10; Center: 50–90% of forgetting energy concentrates in 1–6 eigenmodes; Right: Drift decomposition—spectral regularization targets the vulnerable subspace at high selectivity ratios.

Structural Insights: Low-Rank Vulnerable Subspaces

The analysis exposes that catastrophic forgetting does not diffuse homogeneously but is confined to a low-dimensional vulnerable subspace of the output space; this subspace is the span of the dominant eigenvectors of the task-specific NTK matrix $K_{AA}$. In the linear-head regime, forgetting is provably low-rank, with the effective dimension scaling as $k^* \approx C \cdot k_G$, where $C$ is the number of outputs and $k_G$ reflects the effective rank of the feature Gram matrix. Spectral bias and cross-task kernel alignment together select which directions are susceptible to drift.

This finding resolves a persistent question: function-space forgetting is sharply structured, and parameterized regularization that is not output-selective (e.g., diagonal Fisher or orthogonal parameter constraints) fails to provide effective protection in the presence of head sharing.

Figure 2: Forgetting $\Delta f_A$ is confined to the column space of $K_{AA}$, and energy localizes within the high-eigenvalue "vulnerable" subspace; only targeted spectral regularization efficiently constrains drift along these critical directions.

Methodological Implications: Spectral Regularization

Exploiting the discovered low-rank structure, the paper designs a spectral regularization approach that penalizes drift specifically within the empirically measured top-$0.99$0 NTK eigenmodes. Unlike broad functional regularization (e.g., LwF) or standard parameter-space regularizers (EWC, SI), the spectral regularizer precisely targets the directions of maximum risk, preserving plasticity elsewhere. Empirical evaluation confirms that, especially in shared-head continual learning, spectral regularization strongly suppresses forgetting in the relevant eigenspace while leaving complementary directions unconstrained, consistent with the theoretical prescription.

Empirical Results and Comparative Analysis

On a sweep of continual learning benchmarks, including shared-head Split-MNIST and Split-CIFAR-10, parameter-space methods (EWC, SI, GPM) underperform dramatically, matching naive baselines and failing to mitigate output drift. Function-space methods (LwF, spectral) are substantially more effective, with spectral regularization yielding significant reductions in measured forgetting and inter-task confusion rates. On Split-CIFAR-100 with a frozen pretrained backbone and large output space, spectral and LwF converge in performance, as predicted by the scaling law $0.99$1: when the vulnerable subspace covers most output directions, broad and targeted function-space regularization have overlapping support.

Theoretical and Practical Implications

The research challenges the sufficiency of parameter-space constraints in realistic large-scale continual adaptation. The identification of a data-dependent, low-rank output subspace as the locus of forgetting has immediate implications for both diagnostic tools and regularizer design. The spectral probe provides direct, computable feedback about the geometric vulnerability of task representations. For practitioners, the theory motivates a shift towards methods that respect this eigenspace structure, either through explicit regularization or algorithmic integration with NTK-informed projections. The Kronecker factorization and rank-scaling heuristics support principled hyperparameter selection, especially in high-output regimes. Furthermore, the predictive accuracy of the framework opens the door to proactive model selection based on pre-adaptation kernel diagnostics.

Limitations and Future Directions

The analytic predictor is exact in parameter-efficient, linear-head adaptation but serves as a local approximation with nonlinear heads or full backbone tuning, where NTK eigenstructure and cross-task dynamics become time-varying. The present analysis is scoped to MSE loss; extension to cross-entropy introduces coupling that breaks strict Kronecker structure but retains the broader design intuition. Long task sequences with compounding representational drift are not directly modeled—online or adaptive variants of the vulnerable subspace estimate may address multi-task regimes.

Potential directions for future work include:

• Extending precise output-space geometry predictions to evolving feature representations and end-to-end fine-tuning,
• Deriving softmax-aware or cross-entropy analogs of the closed-form forgetting equation,
• Investigating online kernel tracking for sequences of tasks and memory-efficient approximations for ultra-large-scale settings,
• Applying spectral analysis to instruction tuning and continual adaptation of LLMs, where task coupling and head-sharing are endemic.

Conclusion

The presented framework reframes continual adaptation and catastrophic forgetting as fundamentally function-space phenomena, governed by the interaction of spectral bias, NTK eigenstructure, and cross-task kernel alignment. By delivering analytic predictors, empirical validation, and a theoretically grounded regularization strategy, the paper provides both mechanistic insight and actionable guidance for improving the reliability of continual learning in large, pretrained neural architectures.