SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning

Abstract: Continual Learning (CL) with foundation models has recently emerged as a promising paradigm to exploit abundant knowledge acquired during pre-training for tackling sequential tasks. However, existing prompt-based and Low-Rank Adaptation-based (LoRA-based) methods often require expanding a prompt/LoRA pool or retaining samples of previous tasks, which poses significant scalability challenges as the number of tasks grows. To address these limitations, we propose Scalable Decoupled LoRA (SD-LoRA) for class incremental learning, which continually separates the learning of the magnitude and direction of LoRA components without rehearsal. Our empirical and theoretical analysis reveals that SD-LoRA tends to follow a low-loss trajectory and converges to an overlapping low-loss region for all learned tasks, resulting in an excellent stability-plasticity trade-off. Building upon these insights, we introduce two variants of SD-LoRA with further improved parameter efficiency. All parameters of SD-LoRAs can be end-to-end optimized for CL objectives. Meanwhile, they support efficient inference by allowing direct evaluation with the finally trained model, obviating the need for component selection. Extensive experiments across multiple CL benchmarks and foundation models consistently validate the effectiveness of SD-LoRA. The code is available at https://github.com/WuYichen-97/SD-Lora-CL.

• The paper presents a decoupled strategy that separates direction and magnitude learning in LoRA to mitigate catastrophic forgetting.
• S-LoRA improves computational efficiency by eliminating prompt selection and enabling direct inference using the last-stage model.
• Empirical results on benchmarks like ImageNet-R, ImageNet-A, and DomainNet demonstrate S-LoRA’s superior balance of stability and adaptability.

"SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning" (2501.13198)

Abstract

The paper introduces Scalable Decoupled Low-Rank Adaptation (S-LoRA), a novel method designed to enhance continual learning (CL) by decoupling the learning process of the direction and magnitude of LoRA parameters. This method is developed to mitigate the scalability issues and computational overhead observed in traditional prompt-based approaches for class incremental learning. S-LoRA achieves efficient inference by utilizing the last-stage trained model for direct testing, without the requirement of a prompt selection process.

Introduction

Continual Learning aims to enable models to learn from non-i.i.d data distributions presented sequentially, a scenario where models traditionally suffer from catastrophic forgetting. The paper discusses the potential of foundation models, particularly within the field of vision transformers (ViT), in addressing these challenges due to their ability to transfer knowledge effectively and resist forgetting. The existing methods employ prompt-based strategies, which, despite their effectiveness, face challenges in scalability and computational efficiency. The novel S-LoRA methodology is developed to counter these limitations, providing a scalable solution that disrupts the existing norms of task-specific prompt selection by incrementally decoupling the direction and magnitude in low-rank adaptations.

Methodology

S-LoRA is predicated on the principle of decoupling parameter learning into direction and magnitude components within LoRA structures. By utilizing previously learned directions and calibrating them through learnable scaling factors, the method seeks a low-loss trajectory during the training of sequential tasks, thereby converging to a region of minimal loss common among diverse tasks. Additionally, S-LoRA eliminates the need for computationally intensive prompt selection during inference, which traditionally demanded significant resources. This is achieved through end-to-end optimization, resulting in enhanced computational efficiency and making it rehearsal-free, unlike existing techniques that necessitate the storage of sample features.

Figure 1: Illustration of weight updates in vanilla LoRA and the proposed S-LoRA within continual learning scenarios.

Theoretical Insights and Empirical Evaluation

The theoretical framework underlines that S-LoRA inherently discovers a shared low-loss region across tasks by modifying the magnitude of LoRA updates while preserving the directionality of prior tasks. Theoretical analyses are supported by empirical evidence demonstrating superior performance across multiple CL benchmarks, including ImageNet-R, ImageNet-A, and DomainNet. These experiments highlight S-LoRA's effectiveness in maintaining a balance between stability and adaptability without the necessity for storage or significant memory demands, unlike methodologies such as HidePrompt and InfLoRA which require extensive sample feature retention.

Additionally, the paper introduces enhanced variants of S-LoRA, dubbed ES-LoRA, incorporating dynamic rank adjustment and knowledge distillation techniques to further streamline parameter efficiency. These variants showcase high performance with reduced parameter overhead, reinforcing the core principles of S-LoRA.

Figure 2: (a) Illustration of the gradually learned $\boldsymbol{\alpha}$ values on different tasks.

Conclusion

The proposed S-LoRA offers a substantial advancement in the field of CL by addressing core limitations of existing methods related to scalability and computational demand. By innovatively separating the learning of direction and magnitude, S-LoRA finds a low-loss path and reaches a collectively optimal region across sequentially learned tasks. This is achieved while enhancing inference efficiency, thus contributing significantly to both practical implementations and theoretical understanding of continual learning dynamics. Future exploration could explore further refining parameter efficiencies and expanding this approach across diverse neural network architectures beyond vision transformers.