A Simple Efficiency Incremental Learning Framework via Vision-Language Model with Nonlinear Multi-Adapters

Abstract: Incremental Learning (IL) aims to learn new tasks while preserving previously acquired knowledge. Integrating the zero-shot learning capabilities of pre-trained vision-LLMs into IL methods has marked a significant advancement. However, these methods face three primary challenges: (1) the need for improved training efficiency; (2) reliance on a memory bank to store previous data; and (3) the necessity of a strong backbone to augment the model's capabilities. In this paper, we propose SimE, a Simple and Efficient framework that employs a vision-LLM with adapters designed specifically for the IL task. We report a remarkable phenomenon: there is a nonlinear correlation between the number of adaptive adapter connections and the model's IL capabilities. While increasing adapter connections between transformer blocks improves model performance, adding more adaptive connections within transformer blocks during smaller incremental steps does not enhance, and may even degrade the model's IL ability. Extensive experimental results show that SimE surpasses traditional methods by 9.6% on TinyImageNet and outperforms other CLIP-based methods by 5.3% on CIFAR-100. Furthermore, we conduct a systematic study to enhance the utilization of the zero-shot capabilities of CLIP. We suggest replacing SimE's encoder with a CLIP model trained on larger datasets (e.g., LAION2B) and stronger architectures (e.g., ViT-L/14).

• The paper introduces SimE, which fine-tunes lightweight adapters on the first task, freezes the CLIP encoder, and expands a prototype-based classifier for later classes without replay data or further training.
• SimE achieves up to 91.66% average accuracy on CIFAR-100 and 86.47% on TinyImageNet while using far fewer trainable parameters and roughly one-third the GPU memory of competing approaches.
• The study shows adapter density has nonlinear effects: more adapters help across transformer blocks but can hurt within blocks for small task counts, while larger pre-training datasets and backbones consistently improve results.

Overview and motivation

This paper addresses class-incremental learning (CIL) with pre-trained vision-LLMs, targeting three persistent limitations of existing CLIP-based continual learning approaches: high training cost, dependence on replay memory banks, and sensitivity to backbone strength. The authors propose SimE, a framework that freezes a CLIP image encoder after finetuning only lightweight adapters on the first task, then performs prototype-based classification for all subsequent tasks without any weight updates or stored exemplars. Beyond the framework itself, the paper makes two empirical contributions: a systematic characterization of how adapter placement affects incremental performance, and an ablation of CLIP pre-training datasets and ViT backbone sizes in the incremental setting.

The SimE framework

SimE operates in three phases. First, the frozen CLIP image encoder is finetuned via adapters on the first task only; the encoder is then permanently frozen. Second, class prototypes are computed as the mean feature vectors of each class under the adapted encoder, and a fully connected classifier is extended by appending new prototype columns ($W_t = W_{t-1} + W_{\text{pro}}$), with cosine similarity used for classification. Third, for every subsequent task $i > 1$, no parameters are trained—only prototypes are computed and the classifier updated.

A notable design choice is the composite encoder: because adapter finetuning on the first task inevitably erodes some zero-shot capability, SimE concatenates features from both the finetuned encoder $E^*$ and the original pre-trained encoder $E$, i.e., $E^c(\bm{c}) = \{E^*(\bm{c}); E(\bm{c})\}$, before classification. This preserves the pre-trained representation while adding task-adapted features, at the cost of doubling feature dimensionality.

Multi-Adapter design and theoretical analysis

The paper generalizes AdaptFormer into a Multi-Adapter with three sub-module types per transformer block: AdaptMLP (parallel to the MLP branch), AdaptAtten (parallel to self-attention input), and AdaptAll (spanning both). Each sub-module uses the standard down-projection–ReLU–up-projection bottleneck.

Two propositions frame the empirical findings. Theorem 1 establishes a monotonic bound: since inserting adapters nests parameter spaces ($$\Theta_0 \subseteq \Theta_{N,\text{loc}} \subseteq \Theta_{M,\text{loc}}$$), the supremum of achievable performance cannot decrease with more adapters. Theorem 2 states that actual solutions need not be monotonic—the optimizer may find a worse $\theta^*$ in a larger space. This gap between capacity and realized performance formalizes the paper's central empirical observation, though the proof of Theorem 2 is essentially a restatement that the training procedure does not return the argmax; it offers no mechanism explaining when degradation occurs.

Main results

On CIFAR-100 (10 steps) with ViT-B/16, SimE achieves 85.94% average accuracy versus 82.15% for ZSCL, 75.17% for Continual-CLIP, and 79.35% for iCaRL. With ViT-L/14 pre-trained on LAION-2B, SimE reaches 91.66% average / 86.03% last accuracy on CIFAR-100—roughly 9.6 points above traditional methods on TinyImageNet and 5.3 points above other CLIP-based methods on CIFAR-100, per the abstract's claims. On TinyImageNet (100 base classes, 100 incremental classes), SimE attains 86.47% average accuracy in the strongest configuration, notably stable across 5/10/20 steps (79.35/79.32/79.29%), reflecting the fact that nothing is trained after task one.

Against parameter-efficient CIL methods using the same ViT-B/16 backbone, SimE leads on most datasets: 84.98% average on CUB200 (vs. 78.80% for APER), 89.77% on ImageNet-100, 63.19% on ImageNet-A, and 83.29% on OmniBenchmark, while remaining competitive on ImageNet-R and ObjectNet. An appendix comparison confirms these gains persist when all methods share the identical backbone, indicating the improvement stems from the learning strategy rather than backbone capacity alone.

Efficiency comparisons show SimE trains only thousands-to-millions of adapter parameters (~1.19M for a single sub-module type across 12 blocks), uses roughly one-third of the GPU memory of competing methods, and requires no replay buffer—a structural advantage over CoOp-style approaches whose memory banks grow unboundedly.

The nonlinear correlation of adapter connections

The paper's most distinctive finding concerns adapter density. Between transformer blocks, more adapters monotonically help: inserting adapters into all 12 blocks outperforms partial placements (e.g., blocks 1–3 or 10–12), and early-block placement dominates late-block placement, suggesting primary feature adaptation matters most. Within blocks, however, the picture reverses at small incremental steps: combining AdaptAtten + AdaptMLP + AdaptAll (3.57M parameters) yields 85.54% average accuracy at 10 steps, below the single AdaptAtten configuration (85.94%) despite triple the trainable parameters. Only at 50 steps does the full Multi-Adapter configuration become optimal (85.00% vs. 84.16%). The authors attribute this to overfitting and representational redundancy under mild distribution shift, with extra capacity becoming useful only under larger shifts. Bottleneck dimension, by contrast, has negligible effect (accuracy varies by less than 0.5 points from dimension 1 to 256).

Influence of CLIP components

Pre-training dataset scale and diversity matter substantially: LAION-2B pre-training improves CIFAR-100 average accuracy to 88.34% versus 85.60% for OpenAI's WIT-400M, with t-SNE visualizations showing tighter intra-class clusters and clearer inter-class boundaries for larger datasets. Backbone scaling also helps: ViT-L/14 reaches 88.79% average accuracy versus 85.94% for ViT-B/16, while patch size matters more than input resolution (ViT-L/14-336px slightly underperforms ViT-L/14). Under severe class imbalance ($imb\_factor = 0.01$), accuracy drops by up to 7.06 points relative to balanced settings, indicating SimE remains vulnerable to distributional skew even though it avoids forgetting by construction.

Limitations and open questions

Several caveats bear directly on the reported results. First, the headline gains depend on strong backbones and large pre-training corpora; with ViT-B/16 on WIT-400M, SimE's advantage over ZSCL narrows considerably, so practitioners without access to LAION-scale models will see smaller margins. Second, the theoretical analysis explains capacity monotonicity but not the observed within-block degradation—Theorem 2 is existential rather than predictive, leaving open which insertion strategies will fail a priori. Third, the "nonlinear correlation" claim rests primarily on CIFAR-100 experiments; whether it holds for other datasets and backbone scales is untested. Fourth, finetuning exclusively on the first task assumes the base task adequately represents the downstream domain—an assumption that may fail when later tasks diverge sharply from the base distribution. Finally, the composite encoder doubles feature dimensionality, and the paper does not analyze the compute cost of this concatenation at inference time.

Conclusion

SimE demonstrates that freezing a CLIP image encoder after a single round of adapter finetuning, combined with prototype-based classifier expansion, achieves state-of-the-art CIL accuracy with minimal trainable parameters and no replay memory. Its systematic ablations establish that adapter placement interacts nontrivially with incremental step size—more connections help between blocks but can hurt within blocks at small steps—and that pre-training data scale and backbone size are dominant levers for incremental performance. The main open questions concern predicting adapter-placement outcomes theoretically and validating the step-size-dependent behavior beyond CIFAR-100.