GR4CIL: Gap-compensated Routing for CLIP-based Class Incremental Learning

Abstract: Class-Incremental Learning (CIL) aims to continuously acquire new categories while preserving previously learned knowledge. Recently, Contrastive Language-Image Pre-trained (CLIP) models have shown strong potential for CIL due to their powerful generalization ability. However, existing methods still face two key challenges: shared-parameter adaptation tends to cause old-knowledge drift, and task-specific knowledge organization often leads to poorly calibrated cross-task responses, making reliable routing difficult. To address these issues, we propose GR4CIL, a framework combining task discrimination and knowledge routing for CLIP-based CIL. GR4CIL preserves task-specific visual knowledge while maintaining an incrementally stable shared textual semantic space, thereby reducing interference across tasks. Moreover, we introduce an orthogonal compensation mechanism to mitigate modality-gap-induced bias, enhance within-task discrimination, and enlarge the score margin between the ground-truth task and competing tasks. As a result, GR4CIL enables more reliable task-aware routing over learned knowledge while retaining the zero-shot generalization capability. Experiments on multiple benchmarks show that GR4CIL consistently outperforms strong baselines.

• The paper introduces GR4CIL, which combines task-specific visual LoRA modules, anchored text adaptation, orthogonal compensation heads, and prototype-driven routing to reduce modality-gap errors in CLIP-based class-incremental learning.
• Across CIFAR-100, ImageNet-R, ImageNet100, and ImageNet-1K, GR4CIL achieves the best replay-free results, improving average accuracy by about 1.50 points and last-task accuracy by 1.51 points over the strongest baselines.
• Orthogonal compensation improves image-text subspace alignment, raises routing accuracy from 81.37% to 82.85%, preserves zero-shot transfer, and supports open-scenario detection, though per-task modules and threshold selection limit scalability.

Motivation and problem setting

GR4CIL addresses class-incremental learning (CIL) built on pre-trained CLIP models, where the model must absorb a sequence of disjoint class sets without replay buffers, task identifiers at test time, or loss of zero-shot transfer. The authors identify two failure modes in existing CLIP-based CIL pipelines. First, methods that continually update shared parameters—whether through full fine-tuning or parameter-efficient adapters—inevitably perturb the representation structures that earlier tasks depend on, blurring knowledge boundaries across tasks. Second, once knowledge is organized into task-specific modules (as in mixture-of-experts approaches such as MOE4CL), inference becomes a routing problem: the model must decide which task branch should handle an input, and prior work has underexplored how to make competing task branches sufficiently separable in score space.

The paper's central empirical observation concerns the CLIP modality gap. Following prior measurements of inter-modality similarity, the authors show that downstream adaptation enlarges the gap within each task—driven primarily by negative image-text pairs whose similarity drops sharply during training—and, more importantly, that the modality gaps formed after task-specific adaptation are inconsistent across tasks. Because classification still relies on text features as classifiers, this task-wise inconsistency causes different branches to respond unevenly to the same sample during unified inference, degrading both intra-task discrimination and cross-task routing.

Method

GR4CIL combines three components: decoupled knowledge learning, orthogonal compensation, and prototype-driven routing.

Decoupled incremental learning. The visual branch receives one LoRA module per task (applied to key/value projections), frozen after its task completes; the text branch uses a single shared LoRA updated incrementally. Two regularizers stabilize the shared text space: an anchor loss that pins each previously learned class's text feature to its cached anchor via cosine similarity, and a bounded separation loss that penalizes cosine similarity above a threshold $\tau$ between new-class text features and all seen classes, preventing unbounded repulsion while keeping classes distinguishable. Training minimizes a CLIP-style cross-entropy over current-task classes plus these two terms.

Orthogonal compensation. The core theoretical contribution frames text-subspace classification as an approximation problem. Writing the task's text feature matrix $\mathbf{T}^t$ via SVD and letting $\mathbf{P}_t = \mathbf{U}_t\mathbf{U}_t^\top$ be the projector onto the text subspace, Proposition 1 shows the best classifier constrained to the text subspace is $\mathbf{P}_t W_t^\star$, with error $\|(\mathbf{I}-\mathbf{P}_t)W_t^\star\|_F^2$; Lemma 1 lower-bounds this error by the singular-value tail $\sum_{j>r_t}\sigma_j^2(W_t^\star)$ of an ideal full-space classifier $W_t^\star$. GR4CIL therefore learns a per-task compensation head projected into the orthogonal complement, $$\widehat{W}_{\mathrm{comp}}^t = \mathbf{P}_t^\perp W_{\mathrm{comp}}^t$$, trained with cross-entropy on compensation logits alone. Corollary 1 establishes that approximation error is monotonically non-increasing under this direct-sum construction, with reduction equal to $$\|\mathbf{P}_{R,t}(\mathbf{I}-\mathbf{P}_t)W_t^\star\|_F^2$$. The head is initialized with visual class prototypes to aid optimization in the high-dimensional, non-convex complement space. The final score adds the compensation logits with weight $\beta$: $\mathbf{T}^t$0.

The authors are explicit that the theory is a geometric justification rather than a description of practice: $\mathbf{T}^t$1 is never learned, and no claim is made that the compensation head recovers the optimal residual subspace or attains the singular-value bound.

Routing and open-scenario interface. Inference performs unified score competition over all seen classes, with a prototype term added ($\mathbf{T}^t$2). Beyond standard CIL, each task carries an acceptance threshold on its maximum softmax probability; samples rejected by all tasks can be routed either to a confidence-weighted fusion of task-conditioned features against candidate zero-shot labels, or directly back to the original frozen CLIP. This interface reuses scores already computed during standard inference rather than training separate autoencoders as in DDAS.

Empirical results

Across four benchmarks split into 10 tasks, using CLIP ViT-B/16 and comparing exclusively against replay-free baselines, GR4CIL achieves the best results on every dataset:

Method	CIFAR-100 Avg/Last	ImageNet-R Avg/Last	ImageNet100 Avg/Last	ImageNet-1K Avg/Last
MOE4CL	85.36 / 78.37	85.28 / 80.77	86.39 / 76.66	81.29 / 72.73
MagMax	85.63 / 79.00	87.13 / 80.85	86.33 / 75.92	80.74 / 71.31
MG-CLIP	87.00 / 80.57	87.58 / 82.67	87.31 / 78.38	81.88 / 73.68
AdapterVLM	87.98 / 81.65	88.25 / 82.51	86.03 / 77.05	81.60 / 73.47
GR4CIL	89.35 / 83.22	89.50 / 84.15	87.65 / 78.64	83.36 / 75.33

The average improvement over the strongest baseline is +1.50 Avg-Acc and +1.51 Last-Acc, with low variance over three runs (standard deviations below 0.33). Results hold under 5-task and 20-task partitions, though the margin narrows in the 20-task ImageNet-R setting. On OOD detection—where future-task classes serve as OOD samples—GR4CIL reaches 89.13/87.15 AUROC on CIFAR-100 and 88.75/83.09 on ImageNet-R, modestly exceeding AdapterVLM, supporting the feasibility of the confidence-based trigger.

Zero-shot evaluation after CIL on CIFAR-100 or ImageNet-R shows preserved or slightly improved generalization relative to original CLIP on Pets, Food101, and ImageNet-1K (e.g., 86.23 vs. 84.96 on Pets), indicating the incremental adaptation does not destroy the pre-trained transfer capability—a property many fine-tuning-based CIL methods sacrifice.

Component analyses substantiate the mechanism. Subspace-distance measurements show the joint text-plus-compensation space aligns far better with the image discriminative subspace than the text space alone (e.g., I-TC distance 0.224 vs. I-T 0.814 on CIFAR-100). Compensation increases intra-task GT-class softmax confidence and enlarges the GT-versus-competing-task score margin by an average of 20.34%, raising average routing accuracy from 81.37% to 82.85%. Ablations attribute gains to each component: on CIFAR-100, adding the anchor/separation losses lifts Last-Acc from 79.40 to 80.88, the compensation term to 82.88, and the prototype term to 83.22. Notably, an unconstrained compensation head (no orthogonality projection) yields only 87.34 Avg-Acc versus 89.35 for the orthogonal version, confirming that the benefit comes from modeling residual directions beyond the text subspace rather than from added classifier capacity. Cached-anchor similarities above 0.993 across all datasets confirm the shared text space remains stable throughout the sequence. The method also reports fewer trainable parameters and lower peak GPU memory than other task-specific baselines.

Limitations and open questions

The authors state three limitations plainly. First, the open-scenario extension depends on threshold selection for task-level acceptance and is positioned as an entry point rather than a solution to open-world inference. Second, the framework assumes relatively clear task boundaries; behavior under ambiguous boundaries remains unaddressed, though the OOD-awareness mechanism may relate to discovering new tasks in task-agnostic settings. Third, although efficiency is favorable relative to comparable baselines, task-specific visual LoRAs and compensation heads accumulate linearly with the number of tasks, so long-term overhead reduction remains unresolved. Additionally, the separation threshold $\mathbf{T}^t$3 embodies a genuine trade-off between cross-task semantic consistency and intra-task discrimination, with performance peaking at intermediate values (0.6–0.7); whether this balance transfers to other architectures or longer sequences is not established.

Conclusion

GR4CIL offers a coherent answer to the routing problem in modular CLIP-based CIL: preserve task-exclusive visual knowledge in frozen LoRA modules, anchor a shared textual semantic space, and compensate for modality-gap-induced residual bias with heads constrained to the orthogonal complement of the text subspace, backed by a clean subspace-approximation argument. The consistent ~1.5-point improvement over strong replay-free baselines, together with evidence that compensation specifically improves score margins and routing accuracy, indicates the design targets the right bottleneck. The main unresolved issues are scalability of per-task modules, robustness to ambiguous task boundaries, and principled thresholding for the open-world trigger.