Low-Rank Adaptation Reduces Catastrophic Forgetting in Sequential Transformer Encoder Fine-Tuning: Controlled Empirical Evidence and Frozen-Backbone Representation Probes

Abstract: Sequential fine-tuning of pretrained language encoders often overwrites previously acquired capabilities, but the forgetting behavior of parameter-efficient updates remains under-characterized. We present a controlled empirical study of Low-Rank Adaptation (LoRA) in sequential transformer encoder fine-tuning with companion representation probes that test a frozen-backbone explanation of its robustness. In five full-validation BERT-base reruns on an RTE->MRPC->CoLA->SST-2 sequence, full fine-tuning yields 19.9%+/-4.8% average forgetting, whereas standard LoRA (r=8, query/value modules) yields 0.6%+/-1.4% (paired t-test, p=0.002, Cohen's d_s=3.12). Task-level analyses confirm this reduction is not merely an aggregate effect. Secondary experiments on RoBERTa-base show the same pattern, and the strongest EWC baseline remains at 15.5%+/-1.4% forgetting. A six-task extension reveals that low average forgetting can hide strong task-level heterogeneity. Fine-grained freezing ablations show a marked forgetting drop once frozen parameters exceed roughly 95%, with classifier-only and shallow-adapter baselines approaching LoRA. Companion task-similarity probes in GPT-2 and RoBERTa show the same directional story: frozen-backbone regimes preserve higher inter-task similarity than full fine-tuning, gradual unfreezing weakens stability, and full fine-tuning exhibits its clearest divergence at the final transformer layer. These results support a restrained mechanistic interpretation: LoRA helps largely because backbone freezing preserves a more stable shared feature scaffold. We position standard LoRA as both a strong empirical baseline for sequential encoder adaptation and a useful probe of how selective plasticity shapes interference in transformer continual learning.

• The paper finds that LoRA reduces average forgetting from 19.9% to 0.6% across five BERT-base runs, a 97.2% relative reduction with a large paired effect, but lowers final average accuracy from 64.5% to 59.5%.
• The paper shows that freezing at least about 95% of the backbone produces nearly the same protection as LoRA, indicating that restricted backbone updates—not low-rank parameterization alone—primarily preserve prior-task performance.
• The paper demonstrates that aggregate forgetting can hide severe task-level failures, as MRPC forgetting reached roughly 36 points in the six-task extension, underscoring the need for task-specific evaluation and matched continual-learning baselines.

Overview and motivation

This paper presents a controlled empirical study of whether standard Low-Rank Adaptation (LoRA) reduces catastrophic forgetting in sequential fine-tuning of pretrained transformer encoders, without any continual-learning-specific machinery. The author's central question is deliberately narrow: does constraining the trainable subspace alone—by freezing the backbone and learning low-rank updates—materially reduce forgetting relative to full fine-tuning? The paper's contribution is methodological and interpretive rather than algorithmic. It provides a tightly audited estimate of the LoRA forgetting reduction under a single consistent protocol, then uses freezing ablations and representation-space probes to narrow the best-supported explanation, concluding that strong backbone freezing—not low-rank parameterization per se—is the dominant driver.

A distinctive feature of the work is its explicit evidence-tier taxonomy. The headline result derives exclusively from five full-validation BERT-base reruns (seeds 45–49); earlier screening sweeps that used validation[:200] slices are reported transparently as exploratory and are never pooled with confirmatory statistics. This protocol separation is itself offered as a reporting template to prevent the mixed-protocol overclaiming common in sequential fine-tuning studies.

Experimental design

The main task sequence is RTE → MRPC → CoLA → SST-2 from GLUE, using BERT-base-uncased. Forgetting for task $T_i$ is defined as the drop between post-training peak accuracy and accuracy after all subsequent tasks; average forgetting $\overline{F}$ is the mean over previously seen tasks, with negative values indicating backward transfer. Training uses three epochs, learning rate $2 \times 10^{-5}$, batch size 16, AdamW, 100 warmup steps, and weight decay 0.01, with SST-2 capped at 2,000 examples. The confirmatory comparison uses paired statistics over seeds; secondary experiments (RoBERTa replication, EWC baseline, six-task extension) use a different seed set (42–44) and are explicitly not treated as confirmatory.

Confirmatory result

The headline finding is large and statistically robust. Across five full-validation BERT-base reruns, full fine-tuning yields $19.9\% \pm 4.8\%$ average forgetting, while standard LoRA ($r=8$, query/value modules) yields $0.6\% \pm 1.4\%$—a 97.2% relative reduction (paired $t$-test, $t(4)=6.99$, $p=0.002$, Cohen's $d_z = 3.12$, an unusually large effect size). Task-level breakdowns show the reduction holds on every previously learned task: under full fine-tuning, forgetting is largest on MRPC (23.7 points), CoLA (21.4), and RTE (14.4), whereas under LoRA MRPC forgetting is approximately zero and CoLA shows slight backward transfer. The gap widens progressively as more tasks are added.

Two qualifications accompany this result. First, there is an accuracy trade-off: final average accuracy is 64.5% for full fine-tuning versus 59.5% for LoRA, so forgetting reduction is not free. Second, the claim is scoped to BERT-base encoder classification over four GLUE tasks with a single LoRA configuration; it does not establish that forgetting is solved generally or uniformly protected across tasks.

Secondary evidence

The RoBERTa-base replication ($\overline{F}$0 seeds) reproduces the qualitative pattern: 24.2% ± 2.2% forgetting for full fine-tuning versus 1.7% ± 1.0% for LoRA (92.8% relative reduction). The strongest EWC baseline ($\overline{F}$1) reaches $\overline{F}$2 forgetting with higher final accuracy (67.0%), but this comparison is contextual rather than paired due to differing seed sets. Notably, even the best regularization-based baseline leaves roughly an order of magnitude more forgetting than LoRA, suggesting implicit structural constraints can outperform explicit anti-forgetting objectives—at least in this setting.

The six-task extension (adding QNLI and WNLI) reveals important heterogeneity: average forgetting rises only modestly to 5.6% ± 0.8%, but MRPC forgetting jumps to 35.8–37.5 points once QNLI is introduced. The paper correctly declines to treat this as clean scalability evidence, and it demonstrates concretely that low average forgetting can mask severe task-level degradation—a caution against relying on aggregate forgetting scores alone.

Freezing ablation and the frozen-backbone hypothesis

A fine-grained freezing ablation (exploratory tier, but full-validation, $\overline{F}$3 seeds) provides the key mechanistic link. Partially unfrozen configurations behave like full fine-tuning: freezing layers 0–9 (86.5% frozen) yields 14.8% forgetting, and layers 0–10 (93.0%) yields 13.2%. Once the frozen fraction crosses roughly 95%, forgetting drops sharply—to 5.2% when layer-11 attention is also frozen, 3.1% for classifier-only training, and 2.6–2.9% for shallow-adapter and standard LoRA configurations. Because classifier-only and last-layer-adapter baselines approach standard LoRA, the pattern favors a frozen-backbone explanation over any LoRA-specific inductive bias. The author frames this as suggestive rather than causal: the ablation does not prove attention weights are the unique locus of interference.

Companion representation probes strengthen this interpretation. In GPT-2 probes, a fully frozen backbone preserves inter-task centroid cosine similarity of 0.9988 versus 0.8935 for full fine-tuning, while retaining competitive mean accuracy (0.671 vs. 0.701). Gradual unfreezing produces monotonic erosion of similarity (0.9982 → 0.9943 → 0.9911 → 0.8542 under full fine-tuning). Layer-wise analysis localizes full fine-tuning's clearest divergence at the final transformer layer (0.9902 vs. 0.9973 for LoRA). Orthogonal-regularization sweeps leave task similarity essentially flat (0.9775–0.9778), weakening a generic update-decorrelation explanation. An eight-seed GPT-2 validation confirms the direction statistically ($\overline{F}$4). These probes are exploratory and architecture-mismatched to the BERT benchmark, so they inform interpretation without extending the confirmatory result.

Limitations

The paper identifies five principal limitations candidly. The confirmatory claim rests on one model family, four GLUE tasks, and a single LoRA configuration ($\overline{F}$5, query/value); rank, module-targeting, and variant choices (AdaLoRA, DoRA) may shift the trade-off. The mechanism probes use GPT-2 and RoBERTa rather than the confirmatory BERT setup, leaving open whether the representation-stability account replicates within the same architecture. Baselines exclude replay-augmented and recent PEFT-CL methods such as O-LoRA and InfLoRA under matched seeds. Finally, the existence of earlier subset-validation sweeps creates a disclosed leakage risk that requires careful archival hygiene. The author also stops short of claiming a unique causal mechanism, noting that the probes do not isolate every alternative explanation.

Conclusion

This paper delivers a carefully audited empirical result—standard LoRA reduces average forgetting from 19.9% to 0.6% in sequential BERT-base fine-tuning—and pairs it with converging evidence that the reduction stems primarily from strong backbone freezing preserving a stable shared feature scaffold, rather than from low-rank parameterization itself. The findings position standard LoRA as both a strong baseline for sequential encoder adaptation and a probe of how selective plasticity shapes interference. Key open questions include whether the ~95% freezing transition generalizes across architectures and longer task sequences, and whether the representation-probe account holds within the confirmatory model itself.