Papers
Topics
Authors
Recent
Search
2000 character limit reached

Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning

Published 26 Jan 2026 in cs.LG and cs.CL | (2601.18699v1)

Abstract: LLMs exhibit remarkable performance across diverse tasks through pre-training and fine-tuning paradigms. However, continual fine-tuning on sequential tasks induces catastrophic forgetting, where newly acquired knowledge interferes with previously learned capabilities. Despite widespread observations of this phenomenon, the mechanistic understanding remains limited. Here, we present a comprehensive mechanistic analysis of catastrophic forgetting in transformer-based LLMs during sequential fine-tuning. Through systematic experiments across multiple model scales (109B to 400B total parameters) and task sequences, we identify three primary mechanisms driving forgetting: gradient interference in attention weights, representational drift in intermediate layers, and loss landscape flattening. We demonstrate that forgetting severity correlates strongly with task similarity (Pearson r = 0.87) and gradient alignment metrics. Our analysis reveals that approximately 15 to 23 percent of attention heads undergo severe disruption during fine-tuning, with lower layers showing greater susceptibility. These findings establish mechanistic foundations for developing targeted mitigation strategies in continual learning systems.

Summary

  • The paper analyses the mechanistic causes of catastrophic forgetting in LLMs during continual fine-tuning, identifying three key factors: gradient interference, representational drift, and phase flattening. With 35,000 GPU-hours of experimentation, this study's insights centre on these mechanism details, such as early epoch attention disruption, intermediate layer drift, and the inversion of gradients due to conflicts.
  • The decimation of 4 lifiscale retrention models due to increased intertask similarity was found to be a crucial driver of forgetting.
  • Forgetting was correlated with loss landscape flattening preceding its effects on long-term memory management.

Overview and scope

This paper presents a mechanistic analysis of catastrophic forgetting in LLMs subjected to sequential fine-tuning, moving beyond behavioral characterization to identify parameter-level causes (2601.18699). The study examines six models—Llama 4 Scout (109B total/17B active parameters), Llama 4 Maverick (400B total/17B active), DeepSeek-V3.1 (671B), GPT-5.1 (~1.5T estimated), Claude Opus 4.5, and Gemini 2.5 Pro (~1T estimated)—across twelve task sequences built from twenty-four NLP tasks spanning seven domains. The central claim is that forgetting decomposes into three interacting mechanisms: gradient interference disrupting attention weights, representational drift in intermediate layers, and loss landscape flattening around prior task minima.

Experimental framework

The authors constructed high-, medium-, and low-similarity task sequences (four to six tasks each, 3–5 epochs per task) and collected checkpoints every 500 steps, preserving optimizer states and gradient statistics. Behavioral forgetting was measured as accuracy degradation on held-out test sets of prior tasks; mechanistic measurements included attention weight distances, attention entropy, CKA similarity of hidden representations, per-parameter gradient cosine similarities, and Hessian eigenvalue spectra via stochastic Lanczos quadrature. Experiments consumed roughly 35,000 GPU-hours on H100/H200 hardware.

Behavioral findings

Forgetting was substantial and systematically modulated by task similarity and model scale. On Llama 4 Maverick, first-task accuracy on high-similarity sequences fell from 89.3% to 67.2% after three subsequent tasks (24.8 points absolute); low-similarity sequences produced worse forgetting (31.7 points) than medium-similarity ones (18.3 points). Counterintuitively, higher inter-task similarity correlated with more severe forgetting (Pearson r=0.87r = 0.87, p<0.001p < 0.001). Temporally, forgetting was non-linear: minimal during the first 1–2 epochs, then accelerating sharply as the new task converged. Mixture-of-experts architectures showed heterogeneous patterns, with some experts retaining task knowledge while others were overwritten.

Attention disruption as the dominant early mechanism

Component-freezing experiments identified attention as the primary locus of early forgetting: freezing attention layers reduced forgetting by 64%, versus 23% for feedforward layers and 8% for output layers. Approximately 15–23% of attention heads underwent severe reorganization (weight-space displacement exceeding 2.5 standard deviations), disproportionately concentrated in lower layers. Disrupted heads exhibited entropy increases of 1.8–2.4 bits and specialization-index decreases averaging 0.43. Notably, only 31% of severely disrupted heads were highly activated on the new task, indicating that disruption is largely non-selective—a strong claim implying current fine-tuning wastes capacity by overwriting heads irrelevant to the target objective. Causally, ablating the 20% most disrupted heads post-hoc restored 47% of lost original-task performance at only 8% cost on the new task.

Representational drift in intermediate layers

Intermediate layers showed the largest drift, with CKA decreasing 0.32–0.47 compared to 0.15–0.23 in lower layers and 0.08–0.14 in upper layers. PCA revealed rotation of leading principal components by 35–52 degrees while higher-order components rotated only 8–15 degrees, indicating preferential disruption of dominant representational structure. Drift magnitude was uncorrelated with task relevance (r=0.12r = 0.12, p=0.24p = 0.24): both task-relevant and irrelevant dimensions drifted comparably. Learned affine realignment of intermediate representations recovered 38% of lost performance; combined with head restoration, interventions recovered 71% of original-task performance, establishing causal contributions of both mechanisms.

Gradient interference as a predictive signature

Gradient cosine similarity between current-task and previous-task gradients strongly predicted retention: when similarity dropped below −0.3, forgetting rates increased 2.4-fold relative to positively aligned conditions. Interference localized to attention query/key projections (67% conflict rate) versus value projections (34%) and feedforward weights (29%). First-epoch gradient alignment predicted final forgetting magnitude at r=−0.79r = -0.79, providing an actionable early-warning signal. Severely disrupted parameters also exhibited 2.8× larger gradient magnitudes; aggressive gradient clipping (norm 0.5) reduced forgetting by only 18% while slowing new-task learning by 27%.

Loss landscape flattening

Hessian analysis showed maximum eigenvalues for the first task's loss falling from 147.3 at convergence to 34.2 after three subsequent tasks, while curvature along recent-task directions remained high (112.7). Linearity indices rose from 0.28 to 0.71, indicating erosion of well-defined minima. Critically, curvature reduction preceded behavioral forgetting by 1–2 epochs, making it a leading mechanistic indicator. A curvature regularization penalty reduced forgetting by 34% with 12% slower convergence—a markedly better trade-off than L2 regularization (14% reduction, 31% slower convergence).

Integrated mechanistic model

The three mechanisms operate on distinct timescales and interact synergistically: gradient interference disrupts lower-layer attention during epochs 1–2; representational drift accumulates over epochs 3–5 as intermediate layers adapt; landscape flattening emerges after epoch 4+, removing restoring forces toward prior minima and rendering forgetting progressively irreversible. Larger models exhibit less gradient interference (attributed to higher-dimensional parameter spaces) but comparable representational drift, suggesting drift reflects intrinsic properties of transformer representations rather than capacity limits. The counterintuitive similarity–forgetting relationship is explained by localized query/key conflicts masked by globally aligned gradients.

Limitations and open questions

The paper concedes several constraints explicitly. Mechanisms were characterized under supervised fine-tuning with discrete task boundaries; continual adaptation under RLHF or instruction tuning, and gradual distribution shift, may involve additional mechanisms. Task sequences contained only 4–6 tasks, far short of realistic deployment horizons involving hundreds of tasks. Although proprietary models up to ~1.5T parameters were included via API, the deepest mechanistic analyses (full gradients, Hessians) were feasible mainly up to the 400B open-weight scale, leaving open whether mechanisms change qualitatively beyond a trillion parameters. Alternative architectures (state-space models, encoder-decoder variants) were not analyzed in depth, so architecture-generality remains unverified. Finally, results describe average behavior across runs; individual variability and rare failure modes are uncharacterized.

Conclusion

The paper provides a systematic decomposition of catastrophic forgetting in transformer LLMs into attention disruption, representational drift, and loss landscape flattening, supported by causal intervention experiments and predictive early-training signatures. Its most consequential findings—that freezing attention reduces forgetting by 64%, that first-epoch gradient alignment forecasts final forgetting at r=−0.79r = -0.79, and that curvature regularization outperforms generic regularizers—suggest mechanism-targeted continual learning strategies are feasible. Whether these mechanisms hold across longer task horizons, non-supervised adaptation regimes, and architectures beyond decoder-only transformers remains unresolved.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.