CSC: Cross-Space Clustering Loss for Incremental Learning

• Cross-Space Clustering Loss is a distillation technique that aligns new features with previous class centroids to mitigate catastrophic forgetting.
• It enforces intra-class clustering and inter-class separation by optimizing cosine similarity between current and frozen feature spaces.
• CSC is integrated into incremental learning frameworks, consistently improving performance on benchmarks like CIFAR-100 and ImageNet subsets.

Cross-Space Clustering Loss (CSC) is a class-level distillation objective introduced within the context of class-incremental learning. The primary goal of this loss is to enable continual learning of new classes while minimizing catastrophic forgetting of prior classes by leveraging the structural geometry of the feature space of a previously trained model. CSC achieves this by enforcing both intra-class clustering and inter-class separation between feature vectors in the current and previous model states, thereby providing robust class assignment and enhancing stability in class-incremental protocols (Ashok et al., 2022).

1. Formal Definition and Mathematical Formulation

Let $F^{T_{t-1}}_\phi$ represent the frozen feature extractor of the model after task $t-1$, and $F^{T_t}_\phi$ the current feature extractor during task $t$. For a data sample $x_i$ in a mini-batch $B = \{(x_i, y_i)\}_{i=1}^k$ comprised of both new-task and exemplar memory samples, define

$$z_i^{t} = F^{T_t}_\phi(x_i),\ \ \ z_j^{t-1} = F^{T_{t-1}}_\phi(x_j).$$

The Cross-Space Clustering loss is expressed as:

$$L_{CSC} = \frac{1}{k^2} \sum_{i=1}^k\sum_{j=1}^k \left[1 - \cos(z_i^t, z_j^{t-1})\right] \cdot \operatorname{sgn}(y_i, y_j),$$

where $\cos(a, b) = \frac{a^\top b}{\|a\|\|b\|}$, and $\operatorname{sgn}(y_i, y_j) = +1$ if $t-1$0, and $t-1$1 otherwise.

For $t-1$2, $t-1$3 is minimized, encouraging current features to be attracted toward the old feature subspace of the same class. For $t-1$4, the loss is maximized in magnitude but negative, thus driving repulsion between current instance features and old features of different classes. Equivalently, this can be interpreted as current features being attracted to the mean “centroid” of the old features of their class and repelled from those of other classes.

2. Integration into Incremental Learning Frameworks

CSC is typically integrated as an auxiliary cross-space consistency loss alongside existing class-incremental learning objectives in methods such as iCaRL, LUCIR, and PODNet. The total loss combines the base method’s classification/distillation loss $t-1$5 with the weighted CSC (and optionally, Controlled Transfer) losses:

$t-1$6

where $t-1$7 and $t-1$8 are tunable hyperparameters. Typical choices are $t-1$9 and $F^{T_t}_\phi$0, but these should be adjusted via cross-validation on a separate validation stream. The batch size $F^{T_t}_\phi$1 must be sufficient to sample a diverse set of old classes to accurately estimate class centroids $F^{T_t}_\phi$2.

The core steps in a training iteration are:

• Sample mini-batches from both new-class data and old-class exemplar memory.
• Extract current and old features for all mini-batch samples.
• Compute $F^{T_t}_\phi$3, $F^{T_t}_\phi$4, and optional $F^{T_t}_\phi$5.
• Backpropagate the total loss to update current model parameters (Ashok et al., 2022).

3. Geometric Intuition and Theoretical Rationale

CSC performs class-level distillation by aligning each current feature vector towards the mean region of its corresponding class cluster in the old model’s feature space, while simultaneously maximizing the separation from old feature vectors of all other classes. This joint optimization causes all current embeddings of class $F^{T_t}_\phi$6 to be pulled toward the old mean $F^{T_t}_\phi$7 and repelled from the old regions corresponding to other classes.

This structure imparts several geometric properties:

• Intra-class clustering: All class members converge towards a tight, cohesive cluster, thereby maintaining class integrity.
• Inter-class separation: Repulsion from other class regions broadens the margin and reduces class overlap.
• Herd-immunity: The collective attraction of class instances towards their mean strengthens a class boundary against representation drift and forgetting.

No formal theoretical bounds on representation drift or the quality of feature clustering are provided. The effectiveness of CSC is established empirically through its demonstrated capacity to reduce catastrophic forgetting.

4. Experimental Evaluation and Benchmark Results

The introduction of CSC, in conjunction with the Controlled Transfer (CT) objective (referred to as CSCCT), consistently improves state-of-the-art class-incremental learning baselines across multiple benchmarks:

• CIFAR-100 (50/C and C/C protocols):
  • iCaRL + CSCCT: +2.76% (C=1), +3.31% (C=2), +2.33% (C=5)
  • LUCIR + CSCCT: +2.69%, +1.13%, +2.61%
  • PODNet + CSCCT: +1.92%, +1.12%, +1.06%
  • Under the challenging C/C protocol, iCaRL+CSCCT narrows the performance gap to PODNet.
• ImageNet-Subset (50/C and C/C protocols):
  • Performance gains of ≈1.0–1.7% added to all three base methods.
• Ablation of CSC vs. CT (with LUCIR on CIFAR-100):
  • LUCIR + CSC alone: up to +2.04% (C=1) and +2.72% (hard C/C)
  • LUCIR + CT alone: up to +1.97% (C=5) and +3.03% (C=2 hard)
  • Full CSCCT yields the best average performance across settings, e.g., +2.95% on average.

Feature-space visualizations (e.g., t-SNE of old 50 classes following 100-class training) empirically confirm that CSC enhances cluster compactness and separability.

5. Effects on Stability and Plasticity

CSC primarily enhances stability, as measured by Average Accuracy on Previous Tasks, by curbing forgetting through class-level geometric preservation. In comparison, the Controlled Transfer loss optimizes plasticity by facilitating the positioning of new classes to maximize positive forward transfer and minimize negative backward transfer. The synergistic combination of both objectives leads to an optimal trade-off between stability and plasticity, a central challenge in continual learning scenarios.

6. Implementation Considerations and Best Practices

Effective application of CSC requires:

• Sufficient mini-batch diversity to sample a representative subset of old class exemplars for reliable mean estimation.
• Appropriate weighting of the CSC loss via the hyperparameter $F^{T_t}_\phi$8. Cross-validation is advised to select optimal values for both $F^{T_t}_\phi$9 and $t$0.

CSC can be used as a plug-in distillation objective on top of a broad range of class-incremental learning frameworks. The choice and size of the exemplar memory, base method, and loss blending parameters all affect empirical performance (Ashok et al., 2022).

7. Significance and Future Directions

CSC, by enforcing a class-level attraction/repulsion structure between current and old model feature spaces, demonstrably mitigates catastrophic forgetting in class-incremental learning. Its effects are most pronounced in scenarios with frequent incremental updates and limited access to prior data. Future work may explore formal theoretical guarantees, adapt CSC to non-class-incremental continual learning protocols, or investigate its effect in settings with weaker semantic label overlap, as well as further enhance the class-level geometric alignment mechanism. Possible directions also include improved old exemplar selection and memory management strategies to maximize the clustering and separation effects induced by CSC.