From Order to Distribution: A Spectral Characterization of Forgetting in Continual Learning

Abstract: A central challenge in continual learning is forgetting, the loss of performance on previously learned tasks induced by sequential adaptation to new ones. While forgetting has been extensively studied empirically, rigorous theoretical characterizations remain limited. A notable step in this direction is \citet{evron2022catastrophic}, which analyzes forgetting under random orderings of a fixed task collection in overparameterized linear regression. We shift the perspective from order to distribution. Rather than asking how a fixed task collection behaves under random orderings, we study an exact-fit linear regime in which tasks are sampled i.i.d.\ from a task distribution~$Π$, and ask how the generating distribution itself governs forgetting. In this setting, we derive an exact operator identity for the forgetting quantity, revealing a recursive spectral structure. Building on this identity, we establish an unconditional upper bound, identify the leading asymptotic term, and, in generic nondegenerate cases, characterize the convergence rate up to constants. We further relate this rate to geometric properties of the task distribution, clarifying what drives slow or fast forgetting in this model.

• The paper introduces a spectral operator theory for continual linear regression by linking loss-level forgetting to the eigenspectra of the overlap operator.
• It rigorously characterizes forgetting dynamics via exponential decay rates and polynomial tails, revealing the role of task covariance and geometric complementarity.
• Synthetic experiments validate the theoretical predictions, demonstrating that richer task families accelerate loss decay.

Spectral Analysis of Forgetting in Continual Linear Regression

Problem Formulation and Shift in Perspective

The paper "From Order to Distribution: A Spectral Characterization of Forgetting in Continual Learning" (2604.13460) develops a rigorous operator-theoretic analysis of forgetting in the exact-fit overparameterized linear regression regime. Moving beyond previous order-centric frameworks, notably Evron et al. (2022), which focus on forgetting dynamics for a fixed collection of tasks under random orderings, this work reorients the analysis toward the statistical properties of the task-generating distribution $\Pi$. The primary object is the loss-level forgetting quantity

$$F^\Pi(k) := \mathbb{E}\left[\frac{1}{k} \sum_{t=1}^k \ell_t(w_k)\right], \quad \ell_t(w) = \frac{1}{n_t}\|X_t w - y_t\|_2^2$$

where each $(X_t, y_t)$ is a rank-deficient regression task drawn i.i.d.\ from $\Pi$, consistent with a common parameter $w^\star$, and each update is the minimum-norm exact fit closest to the previous iterate. The consecutive iterates $w_1,w_2,\ldots$ induce projection dynamics through orthogonal projectors $P_t = I - X_t^+X_t$, but the central issue is to characterize forgetting in terms of the actual loss, not merely state evolution.

Exact Operator Identity and Spectral Expansion

A principal contribution is the derivation of an exact operator-level identity relating $F^\Pi(k)$ to the spectral properties of the so-called overlap operator $S_\Pi(A) = \mathbb{E}[P_tAP_t]$. This identity captures the full loss-level geometry, explicitly retaining the task covariance $C_t$ alongside projection dynamics:

$$F^\Pi(k) := \mathbb{E}\left[\frac{1}{k} \sum_{t=1}^k \ell_t(w_k)\right], \quad \ell_t(w) = \frac{1}{n_t}\|X_t w - y_t\|_2^2$$0

where $$F^\Pi(k) := \mathbb{E}\left[\frac{1}{k} \sum_{t=1}^k \ell_t(w_k)\right], \quad \ell_t(w) = \frac{1}{n_t}\|X_t w - y_t\|_2^2$$1.

This relation is further analyzed via spectral decomposition: the expansion splits $$F^\Pi(k) := \mathbb{E}\left[\frac{1}{k} \sum_{t=1}^k \ell_t(w_k)\right], \quad \ell_t(w) = \frac{1}{n_t}\|X_t w - y_t\|_2^2$$2 into contributions from different eigenspaces of $$F^\Pi(k) := \mathbb{E}\left[\frac{1}{k} \sum_{t=1}^k \ell_t(w_k)\right], \quad \ell_t(w) = \frac{1}{n_t}\|X_t w - y_t\|_2^2$$3,

$$F^\Pi(k) := \mathbb{E}\left[\frac{1}{k} \sum_{t=1}^k \ell_t(w_k)\right], \quad \ell_t(w) = \frac{1}{n_t}\|X_t w - y_t\|_2^2$$4

where $$F^\Pi(k) := \mathbb{E}\left[\frac{1}{k} \sum_{t=1}^k \ell_t(w_k)\right], \quad \ell_t(w) = \frac{1}{n_t}\|X_t w - y_t\|_2^2$$5 couples spectral modes of the task covariance and the shared target, and $$F^\Pi(k) := \mathbb{E}\left[\frac{1}{k} \sum_{t=1}^k \ell_t(w_k)\right], \quad \ell_t(w) = \frac{1}{n_t}\|X_t w - y_t\|_2^2$$6 determines decay scales, with diagonal modes decaying exponentially at rate $$F^\Pi(k) := \mathbb{E}\left[\frac{1}{k} \sum_{t=1}^k \ell_t(w_k)\right], \quad \ell_t(w) = \frac{1}{n_t}\|X_t w - y_t\|_2^2$$7 and mixed modes exhibiting slower behavior, including polynomial $$F^\Pi(k) := \mathbb{E}\left[\frac{1}{k} \sum_{t=1}^k \ell_t(w_k)\right], \quad \ell_t(w) = \frac{1}{n_t}\|X_t w - y_t\|_2^2$$8 tails in degenerate cases.

Convergence Rates, Geometric Interpretation, and Bounds

The analysis identifies the leading asymptotics governed by the top spectral value $$F^\Pi(k) := \mathbb{E}\left[\frac{1}{k} \sum_{t=1}^k \ell_t(w_k)\right], \quad \ell_t(w) = \frac{1}{n_t}\|X_t w - y_t\|_2^2$$9:

$(X_t, y_t)$0

with subdominant terms exponentially suppressed unless coefficients vanish due to geometric degeneracy. The data-dependent exponential upper bound is shown to match the leading asymptotics up to constants, confirming the tightness and dominance of the spectral decay mechanism:

$(X_t, y_t)$1

Geometric interpretation is achieved by connecting $(X_t, y_t)$2 to principal angles between error directions and task null spaces, where slow forgetting is attributable to directions persistently invisible to the task family. Explicit formulas are provided in terms of normalized second-order invisibility scores, and it is established that commuting projector families (i.e., tasks with mutually compatible null spaces) may exhibit exactly zero forgetting, marking a fundamental obstruction to uniform positive lower-bound theories.

Experimental Validation

Synthetic experiments in high-dimensional regimes systematically substantiate the theoretical predictions. The empirical forgetting curves are shown to track the explicit spectral bounds on the correct scale, far outperforming coarse $(X_t, y_t)$3 projector-based baselines. The analytic spectral rate $(X_t, y_t)$4 matches empirical local decay rates across regimes, and varying task-family richness directly controls $(X_t, y_t)$5 and long-horizon forgetting, confirming the geometric dependence predicted by theory.

Figure 1: Synthetic experiments show empirical forgetting closely matching the explicit spectral upper bound and analytic rate $(X_t, y_t)$6, with richer task families yielding accelerated decay.

Implications and Connections

This spectral operator characterization reframes continual learning forgetting in terms of distribution-level geometry and spectral contractivity. Practically, the results elucidate how task diversity, complexity, and geometric complementarity mitigate forgetting through accelerated spectral decay; conversely, high-dimensional regimes with weak task richness experience protracted forgetting due to slow identification of the common solution. Theoretically, these findings subsume and generalize order-based results, including random-order and cyclic bounds, identifying precisely when and how slower decay laws emerge.

Future work can extend these spectral techniques to more general regimes: underparameterized, noisy, replay-based, regularized, or nonlinear continual learning models. Furthermore, this framework encourages the design of task sequences and learning algorithms that optimize distribution-level spectral properties, potentially fostering more robust continual adaptation with minimized forgetting.

Conclusion

A distribution-centric spectral operator theory of forgetting is established for exact-fit continual linear regression. The main forgetting quantity admits a sharp spectral expansion, unconditional exponential upper bounds, and asymptotic decay rates governed by the overlap operator's leading eigenvalue. Task geometry and richness are revealed as the pivotal determinants of forgetting speed, and the theoretical predictions are corroborated by synthetic experiments. These results provide an authoritative loss-level perspective that clarifies the scale, mechanism, and geometric dependence of forgetting in continual learning (2604.13460).