SR$^2$-LoRA: Self-Rectifying Inter-layer Relations in Low-Rank Adaptation for Class-Incremental Learning

Abstract: Pre-trained models with parameter-efficient fine-tuning (PEFT) have demonstrated promising potential for class-incremental learning (CIL), yet catastrophic forgetting still persists when adapting models to new tasks. In this paper, we present a novel perspective on catastrophic forgetting through the analysis of inter-layer relation drift, i.e., the progressive disruption of relationships among layer-wise representations during the learning of new tasks. We theoretically show that the increase of such drift reduces the classification margins of previously learned tasks, thereby degrading overall model performance. To address this issue, we propose \underline{S}elf-\underline{R}ectifying inter-layer \underline{R}elation Low-Rank Adaptation~(SR$^2$-LoRA), a simple yet effective method that mitigates catastrophic forgetting by constraining inter-layer relation drift. Specifically, SR$^2$-LoRA constructs the relation matrices induced by the previous and current models on current-task samples, and aligns the corresponding singular values. We further theoretically show that this alignment exhibits greater robustness to estimation perturbations than direct entry-wise alignment. Extensive experiments on standard CIL benchmarks demonstrate that SR$^2$-LoRA effectively mitigates catastrophic forgetting, with its advantages becoming more pronounced as the number of tasks increases. Code is available in the \href{https://github.com/FqWan24/SR-2-LoRA}{repository}.

• The paper introduces SR²-LoRA, a novel framework that aligns inter-layer singular values to mitigate catastrophic forgetting in class-incremental learning.
• It leverages low-rank adaptation modules while freezing previous weights to preserve deep representational structures across sequential tasks.
• Empirical results on CIFAR-100, CUB-200, and ImageNet variants show significant accuracy gains and improved stability over existing PEFT methods.

Self-Rectifying Inter-Layer Relations in LoRA for Class-Incremental Learning

Introduction and Motivation

Class-Incremental Learning (CIL) continues to confront the persistent challenge of catastrophic forgetting when adapting Pre-Trained Models (PTMs) to sequential tasks. Despite advances in Parameter-Efficient Fine-Tuning (PEFT) techniques such as Low-Rank Adaptation (LoRA), model adaptation on new tasks degrades performance on earlier tasks due to disruption in inherent representational structures. This work introduces a novel theoretical and practical framework—SR$^2$-LoRA—which identifies and controls catastrophic forgetting by targeting inter-layer relation drift: the progressive disruption of the relational structure among representations across model layers during task sequences.

The critical assertion is that inter-layer relations encode task knowledge beyond individual layer activations. The empirical and theoretical analysis establishes that constraining inter-layer relation drift can mitigate margin collapse and, thereby, performance degradation across task sequences.

Theoretical Foundations

The paper provides a rigorous analytical framework connecting forgetting to the distortion of inter-layer dependencies during the learning of new tasks. Key theoretical contributions are:

• Connection to Classification Margins: Forgetting is bounded by the reduction in classification margins of previous tasks, with changes in these margins governed by deviations in aggregated residuals across all layers.
• Role of Inter-Layer Relations: The cumulative deviation in these residuals can be decomposed into terms capturing both the magnitude of layer-wise changes and the degree of relation drift across layers. Theoretical results (Theorem 2) directly bound task forgetting by the magnitude of inter-layer relation drift.
• Stability of Singular Value Alignment: The analysis shows that the singular values of inter-layer relation matrices are robust to estimation noise, providing a stable target for alignment-objectives during training.

This foundation motivates constraining not merely the features or activations, but the deeper relational structure between layers.

SR$^2$-LoRA: Model and Method

SR$^2$-LoRA introduces a regularization protocol integrated with LoRA-based PEFT for ViT backbones in CIL:

• LoRA Modules for Adaptation: For each new task, only a new set of low-rank modules (i.e., matrices $(B_t, A_t)$ per attention layer) are learned, while previous weights and modules remain frozen.
• Relation Matrix Construction: For each task sample, at each task boundary, the inter-layer relation matrices are computed for both the pre-update (task $t-1$) and current (task $t$) model, using cosine similarity between all pairs of layer representations.
• Singular Value Alignment: Instead of direct elementwise matching of relation matrices (which is noise-sensitive), SR$^2$-LoRA aligns their singular values, exploiting the theoretical stability guarantee. The alignment loss augments the cross-entropy objective for the current task.

Formally, the total objective is a weighted sum of the classification loss and the singular value alignment loss, the latter computed in a sample-wise manner for robustness across the training set.

Empirical Evaluation

Datasets, Backbones, and Baselines

SR$^2$-LoRA is evaluated on CIFAR-100, CUB-200, ImageNet-R, and ImageNet-A under 5-, 10-, 20-, and 50-task settings. All models use ViT-B/16 pre-trained on ImageNet-21K. Baselines span selection-based (e.g., CODA-Prompt), prototype-based (e.g., LoRA-DRS), covariance-based (e.g., MACIL), and orthogonality-based (e.g., InfLoRA) methods.

Main Results

SR$^2$-LoRA consistently outperforms vanilla fine-tuning and all PEFT baselines, with the accuracy gain increasing as the number of tasks grows. For instance, on CUB-200, SR$^2$-LoRA achieves average accuracy exceeding 94% under the 20-task split, where vanilla baseline falls below 90%. On more challenging datasets (ImageNet-A, ImageNet-R), the method closes large absolute gaps in both average and last-task accuracy, especially for long task sequences.

Ablation Studies

Ablations confirm the superiority of inter-layer relation alignment over conventional last-layer feature distillation and direct matrix entrywise alignment. Sample-level singular value alignment yields better stability and less variance than batch-averaged or eigenvalue-based alternatives.

Impact of Alignment Depth

Deeper-layer alignment (deep-to-shallow) has a significantly higher effect on retention and forgetting, empirically validating the theoretical predictions about the impact of disruptions in late-layer relations.

LoRA Hyperparameters

SR$^2$0-LoRA performs robustly across a range of adaptation ranks, with moderate ranks (e.g., 10) providing the best trade-off.

Scheduling and Overhead

The method is robust to the choice of regularization coefficient schedules; a fixed value suffices, and training/inference overheads are negligible, usually only involving SVD computations over small relation matrices per sample.

Generality

Performance improvements hold across different PTM depths and pre-training corpora (e.g., ViT-S/B/L, ImageNet-1K vs. 21K).

Implications and Future Impact

Theoretical Implications

This work establishes a new axis of analysis for representation stability in CIL, foregrounding the importance of structural relationships between layers, rather than just within-layer or output-level constraints. The singular value-based alignment approach is demonstrated to be both principled (via robustness) and practically effective.

Practical Implications

SR$^2$1-LoRA offers a directly applicable, scalable drop-in replacement for LoRA-based CIL strategies, introducing no significant compute or parameter cost. The mechanism is compatible with deep ViT variants, making it suitable for resource-constrained continual learning scenarios.

Extensions and Future Directions

• Application to other domains (e.g., NLP transformers, multimodal encoders) where catastrophic forgetting under PEFT is problematic.
• Automated adaptation of alignment depths per dataset/domain or developing meta-learning schedules for the alignment constraint.
• Combining inter-layer relation alignment with advanced experience replay or generative replay for further robustness.

Conclusion

SR$^2$2-LoRA provides a compelling and well-theorized solution to catastrophic forgetting in class-incremental learning with low-rank adaptation. By directly constraining inter-layer relation drift through singular value alignment, the framework achieves state-of-the-art continual learning accuracy while maintaining memory and computational efficiency. This introduces a promising direction for scalable, robust adaptation of large pre-trained models in continual learning regimes (2605.07420).