Parameter-Efficient Continual Learning for Automatic Speech Recognition

Abstract: Speech foundation models enable strong general-purpose ASR and are attractive for downstream adaptation. However, their size and the catastrophic forgetting induced by sequential fine-tuning demand parameter-efficient and regularized training methods, motivating parameter-efficient continual learning (PECL). While PECL has been widely studied in NLP and vision, it has received less attention in ASR. In this paper, we propose a simple yet effective PECL method based on recent advances in parameter-efficient fine-tuning for ASR. We partition pretrained weight matrices into head and tail subspaces according to singular values and restrict adaptation to approximate rotations within the low-energy tail subspace, preserving dominant components and reducing forgetting. For subsequent tasks, rotations are combined via weight averaging to further improve retention. Experiments on two benchmarks demonstrate reduced forgetting and superior overall performance compared to recent PECL baselines.

• The paper proposes CSSVD, a method that restricts adaptation to low-energy tail singular directions, reducing catastrophic forgetting.
• It employs structured SVD decomposition and a novel rotation mechanism to balance adaptation and retention across sequential tasks.
• Experimental results show CSSVD achieves lower average WER and minimal forgetting compared to state-of-the-art PECL baselines.

Parameter-Efficient Continual Learning for Automatic Speech Recognition

Introduction and Problem Context

The scaling of automatic speech recognition (ASR) to massive, multilingual foundation models has enabled significant gains in generalization and downstream task adaptation. However, continually adapting these large ASR models to new downstream tasks introduces severe practical challenges. Foremost among these are (1) the computational burden of full fine-tuning for models with hundreds of millions of parameters, and (2) catastrophic forgetting (CF), which degrades prior-task performance after sequential adaptation. While parameter-efficient continual learning (PECL) has been extensively explored in NLP and vision, its deployment in ASR remains relatively peripheral, primarily tackled via data rehearsal or ad hoc regularization rather than explicit parameter-efficiency constraints.

This work proposes a dedicated PECL approach for ASR, called Continual Structured Singular Value Decomposition (CSSVD), which leverages subspace partitioning in pretrained weight matrices to restrict adaptation, and formulates an averaging mechanism for sequential task updates that empirically reduces forgetting and improves aggregate ASR performance.

Continual SSVD: Subspace-Restricted Adaptation

CSSVD builds on the structured SVD (SSVD) framework, originally advanced for parameter-efficient fine-tuning (PEFT) in ASR. In conventional SSVD, adaptation is performed within the subspace spanned by the top singular vectors (maximal singular values) via learned scaling and rotation, which can interfere with the essential representations acquired during pretraining and thereby induce forgetting.

CSSVD inverts this strategy. For each linear layer, the weight matrix is decomposed via SVD into a "head" (the top $d-k$ singular directions reflecting high-importance knowledge) and a "tail" (the bottom $k$ singular directions associated with low-importance information). Adaptation parameters are strictly localized to the tail subspace, implemented as an approximate rotation matrix $\bm G$ (parameterized via a skew-symmetric generator), with the head left unchanged. This selective adaptation minimizes interference with the core learned representations of the original and prior tasks.

The process for learning the first new task comprises: SVD decomposition of $\bm W$ into head/tail; restricted optimization of a rotation $\bm G_1$ within the tail subspace to match new-task data; and update of the task-specific weight as $$\bm W_1 = \bm W_0 + \bm U_T \bm \Sigma_T \bm G_1 \bm V_T^\top$$, where only the tail (underlined part) is trainable.

Figure 1: Overview schematic of the CSSVD mechanism for a single linear layer, showing head/tail decomposition and restricted adaptation via tail subspace rotation.

For subsequent tasks, CSSVD recalculates the SVD of the adapted weight, redefines a new head/tail split (to reflect any changes in singular value importance), and again adapts with a new rotation within the fresh tail. The composition of successive tasks is managed via averaging of task-specific updates, which acts as a convex combination across the sequence to balance adaptation and retention and further encourages mitigation of catastrophic forgetting.

Experimental Evaluation: Benchmarks, Baselines, and Metrics

Experiments are conducted using the OWSM v3.2 small ASR model (nine E-Branchformer encoders and nine Transformer decoders, CTC branch for training only, total 366.7M params, 50k vocabulary, pretrained on 180k hours of speech from 151 languages).

Two continual adaptation scenarios are benchmarked:

1. Experiment 1: Initial tasks—English (ENG), German (DEU), Spanish (ESP); new tasks—Dutch (NL) and Flemish (VL), both from the CGN corpus.
2. Experiment 2: Initial tasks as above; new tasks—VL and dialectal Flemish (DVL) from the GCND corpus.

Compared PECL baselines include LoRA, LoRA+FTA, SSVD, MiLoRA, OPLoRA, BiLoRA, and EWC-LoRA, with adaptation restricted to linear weight matrices. Reference conditions include Full Fine-Tuning (FFT) and Separate Models (oracle access to task identity).

Metrics are Average Word Error Rate (WER) and Backward Transfer (BWT), the latter quantifying mean WER change for prior tasks (negative values indicating forgetting).

Results and Analysis

Comparative Performance and Forgetting Mitigation

CSSVD consistently achieves the lowest Average WER of all evaluated methods across both experiments. For instance, in Experiment 1, CSSVD delivers Average WER of 18.33%, surpassing all PECL baselines and even outperforming FFT. Catastrophic forgetting is essentially eliminated, with BWT of –1.9 compared to –18.2 (FFT) and –35.7 (LoRA).

• LoRA, SSVD, and OPLoRA exhibit substantial forgetting, with prior-task WER degradation of over 30 points.
• MiLoRA demonstrates that initialization in the minor singular directions alone fails to retain prior knowledge.
• LoRA+FTA and BiLoRA deliver the strongest baseline retention, but still lag behind CSSVD in both adaptation and forgetting.
• EWC-LoRA (using Fisher information computed only on ENG) reduces forgetting, but adaptation to new tasks is slower and overall retention is inferior to CSSVD.

In the more demanding Experiment 2, where DVL proves challenging for OWSM, CSSVD again yields the best trade-off, with 97% reduction in LoRA's forgetting and 7% improvement over the best baseline.

Ablation Findings

Ablation studies isolate key elements underlying CSSVD's efficacy:

• Restriction to the lowest-$k$ singular directions is the primary factor in minimizing interference; skipping convex averaging increases WER and forgetting, but even then, the method outperforms baselines.
• Recomputing head/tail partitions for each task (rather than fixing the original split) is neutral in impact, but more practical.
• Explicit rescaling (as in SSVD) is unnecessary; approximate rotation suffices.
• Weight averaging with LoRA or SSVD alone is less effective than CSSVD's subspace-oriented strategy.

These results demonstrate that protecting high-energy (head) subspaces and confining adaptation to the low-energy tail is essential for scalable PECL in speech foundation models.

Implications and Future Research Directions

CSSVD introduces a rigorous SVD-based framework for parameter-efficient continual learning in ASR, offering clear advantages in knowledge retention and controlled capacity allocation across sequential tasks. This paradigm is especially attractive for large-scale deployment scenarios, where retraining or duplicating entire models per task is impractical and access to prior-task data is typically constrained.

The results suggest theoretical and practical implications beyond ASR:

• Generalization to other domains: The approach is naturally extensible to vision and NLP models, particularly as SVD-based subspace strategies for PEFT gain wider adoption.
• Selective adaptation across layers: Future refinements could include adaptive allocation of adaptation budget to layers with the most relevance for new tasks or least vulnerability to interference.
• Compositional continual learning: The averaging mechanism for task updates in CSSVD hints at broader models for synthesizing task solutions in a shared subspace, relevant for both multitask and meta-continual learning.
• Long-horizon continual learning: While current results are robust over two or three task additions, further investigation is warranted for longer sequences and more diversified task streams.

Conclusion

This study provides an extensive evaluation of PECL methods for ASR, introducing CSSVD as an effective subspace-restricted approach that minimizes catastrophic forgetting and improves average ASR performance under strict parameter budgets. By constraining adaptation to the tail of the singular spectrum and leveraging averaged subspace transformations, CSSVD sets a new standard for scalable, data-efficient continual adaptation in foundation-model-based speech recognition (2606.09342).