Class-Incremental Learning (CIL)

• Class-Incremental Learning (CIL) is a continual learning paradigm that enables models to learn new classes without revisiting all previous data.
• It employs methodologies such as exemplar replay, generative replay, and parameter regularization to combat catastrophic forgetting.
• CIL techniques optimize memory efficiency and model adaptability, making them essential for dynamic, lifelong learning applications.

Class-Incremental Learning (CIL) is a continual learning paradigm in which a model is extended to recognize new classes that arrive over time, while preserving its ability to discriminate previously learned classes without access to all historical data. Unlike task-incremental or domain-incremental settings, CIL presents the additional challenge that task identity is unavailable at inference, requiring a single classifier to distinguish among all observed classes across all phases. The chief technical hurdle in CIL is catastrophic forgetting—the overwriting of acquired knowledge when the model is updated on new class data. Developing scalable, memory-efficient, and robust CIL algorithms remains a central research objective in lifelong machine learning.

1. Foundations and Challenges

Class-Incremental Learning extends statistical risk minimization to a non-stationary class distribution, typically modeled as a sequence of tasks $\mathcal{D}_1, \mathcal{D}_2, \ldots$, each introducing disjoint sets of classes. After each phase, the model is required to classify among all seen classes with no access to the full training set of previous phases. The core challenge—catastrophic forgetting—arises because neural network parameter updates on newly introduced classes inevitably degrade performance on previously learned classes. This instability is particularly acute in CIL because the same parameter set must encode both old and new knowledge under non-i.i.d., streaming data distributions (Zhou et al., 2023).

The absence of a task identifier at test time further exacerbates the problem compared to task-incremental or domain-incremental scenarios, necessitating effective mitigation strategies for inter-class interference and classifier bias.

2. Taxonomy of CIL Methodologies

CIL methodologies are generally categorized according to the principal mechanism by which they address forgetting and knowledge integration (Zhou et al., 2023):

• Data-Centric Approaches:
  • Exemplar Replay: Maintaining a fixed-size memory buffer of representative samples (exemplars) from past classes. Upon each update, the new class data are mixed with the replay buffer for rehearsal (Zhao et al., 2020).
  • Generative Replay: Training a generative model to synthesize past class samples, thereby obviating explicit exemplar storage.
  • Data Regularization: Constraining updates by ensuring losses computed over exemplars do not worsen (e.g., gradient episodic memory).
• Model-Centric Approaches:
  • Dynamic Network Expansion: Adding parameters (e.g., neurons, modules) as new classes arrive (DER, FOSTER, DyTox).
  • Parameter Regularization: Penalizing deviation of parameters critical to past tasks (e.g., Elastic Weight Consolidation, EWC).
• Algorithm-Centric Approaches:
  • Knowledge Distillation: Regularizing the new model to mimic outputs (logits or features) of the model trained on previous classes (e.g., LwF, feature distillation, relational distillation).
  • Bias Correction/Rectification: Adjusting classifier weights or logits post-update to correct for imbalance caused by overrepresentation of new classes.

Hybrid and parameter-efficient designs (e.g., adapters, prompt-based tuning) increasingly blend these principles, aiming for balance between plasticity and stability (Wang et al., 14 Jun 2025).

3. Memory Efficiency and Exemplar Management

Under practical memory constraints, storing all previously observed samples is infeasible. Memory-efficient CIL methods seek to maximize knowledge retention per memory unit. Techniques include:

• Low-Fidelity Exemplar Storage:

Representing exemplars in a compressed form (e.g., via encoder–decoder architectures or PCA) to fit more samples within the buffer. The performance-memory trade-off is made explicit: with a compression ratio $r$, one can store $1/r$ times as many samples, but must contend with a domain shift between high- and low-fidelity domains (Zhao et al., 2020).

• Duplet Learning Schemes:

Training jointly on original and auxiliary (compressed) exemplars with a cross-domain loss that explicitly reduces the domain gap by ensuring feature and prediction consistency between both versions. The loss may take the form

$$L_{dup} = \frac{1}{|A| + |\hat{A}|} \sum_{(x, \hat{x}, y)} [\ell(x, y) + \ell(\hat{x}, y)],$$

where $\ell$ comprises both cross-entropy and distillation terms.

• Exemplar Selection in Noisy/3D Domains:

For modalities prone to corruption (e.g., point clouds), robust selection strategies (e.g., farthest point sampling) are used to preserve intra-class diversity, mitigating forgetting even on corrupted data (Ma et al., 18 Mar 2025).

These innovations ensure that under a fixed memory budget, the replay buffer preserves class separability more effectively, directly impacting sustained accuracy across phases.

4. Advances in Representation Learning and Model Regularization

CIL research strongly emphasizes the quality and stability of learned representations:

• Prototype-based Embedding:

Discriminative embeddings are encouraged by regularized prototype losses and hard triplet losses. For instance, decoupled prototypes and curriculum clustering enable accurate novel class detection and regulated model expansion with reduced confusion among embeddings (Yang et al., 2020).

• Initial Phase Representation Decorrelation:

The initial model's quality heavily affects all subsequent incremental phases. Class-wise Decorrelation (CwD) loss is introduced to spread variance uniformly across dimensions, preventing early representation collapse; the loss penalizes lack of diversity in activation patterns, promoting oracle-like scatterings (Shi et al., 2021). This is formalized in the loss:

$$L_{CwD}(\theta) = \frac{1}{Cd^2} \sum_{c=1}^C ||K^{(c)}||_F^2$$

where $K^{(c)}$ is the correlation matrix per class.

• Importance-aware Parameter Regularization:

Adapters and prompt-based finetuning can be regularized with parameter-importance metrics (e.g., $\mu/\sigma^2$ for channel activations) to protect critical channels while selectively updating others (Wang et al., 14 Jun 2025).

These designs are empirically validated by both standard accuracy and advanced measures such as representation similarity (CKA), linear probing, and transfer learning performance (Cha et al., 2022).

5. Catastrophic Forgetting and Compensation Mechanisms

Mitigation of forgetting is addressed via several axes:

• Classifier Adaptation:

Distillation labels induce classifier bias; to address this, adaptation phases retrain classifier layers on true-labeled replay data with the feature extractor fixed, substantially boosting retained accuracy (often by $6$–$7.5\%$) (Zhao et al., 2020).

• Trainable Semantic Drift Compensation:

Fine-tuning the classifier with compensated prototypes (reflecting observed semantic drift in feature space across tasks) enables alignment of decision boundaries and rectifies misclassification due to incremental drift (Wang et al., 14 Jun 2025).

• Task-Prediction-Based Routing:

When disjoint tasks/classes are learned with separate submodels, adaptive task prediction—using, for instance, likelihood-ratio tests in feature space combining Mahalanobis distances and $k$-NN to estimate in-distribution probability against a replay buffer—improves accuracy and further prunes catastrophic forgetting (Lin et al., 2023).

6. Impact of Pre-training, Self-supervision, and Data Efficiency

Recent research extends CIL beyond supervised, label-rich settings:

• Pre-trained Models and Self-supervised Features:

The use of strong pre-trained backbones (e.g., Vision Transformers, CLIP, self-supervised representations) significantly increases base accuracy and representation transferability, especially when combined with feature augmentation (branch cloning and score fusion) and careful adaptation strategies (Wu et al., 2022). Statistical analysis frameworks reveal that initial pre-training strategy is typically the single dominant factor influencing average incremental accuracy, though algorithm selection is decisive for minimizing forgetting (Petit et al., 2023).

• Exemplar-Free and Semi-supervised CIL:

Approaches utilizing contrastive learning to obtain robust feature extractors, paired with incremental prototype classifiers regularized via unsupervised consistency losses and resampling-based pseudo-replay, can outpace traditional exemplar-based baselines, even when using less than 1% labels per phase (Liu et al., 27 Mar 2024). Data-free methods based on embedding distillation and task-oriented latent feature generation further extend CIL to privacy-critical domains without storing exemplar data (Huang et al., 2023).

7. Evaluation Protocols and Fairness Considerations

The rigor of CIL evaluation now emphasizes not only average and last phase accuracy but also the trade-off between performance and memory footprint (Zhou et al., 2023). Area under performance–memory curves (AUC-A/AUC-L) is used to benchmark both static and dynamically expandable architectures under fair resource constraints. Critical insights are:

• Methods augmenting memory with extra parameters or buffer storage must align for head-to-head comparison.
• Averaged accuracy alone may obscure representation collapse or non-transferability; thus, probing via linear evaluation, $k$-NN, transfer set testing, and representation similarity are crucial (Cha et al., 2022).
• Distillation-based methods with constrained exemplars are empirically optimal under aligned memory budgets, but hybrid strategies (combining e.g., regularization, exemplar replay, and dynamic expansion) remain leading candidates for future research (Zhou et al., 2023, Wang et al., 14 Jun 2025).

8. Outlook and Future Directions

Class-Incremental Learning continues to evolve with advances in pre-training, representation learning, scalable architecture design, and robust evaluation. Open challenges include: improved methods for online and few-shot CIL, multi-modal and multi-domain continual learning, privacy-preserving frameworks (e.g., data- and exemplar-free CIL), deeper theoretical understanding of forgetting and knowledge transfer mechanisms, and the development of holistic evaluation standards that reflect both performance efficiency and practical deployment constraints.

Efforts to model feature evolution through phenomena such as neural collapse and to design architectures that approach the information-theoretic limits of representational compactness and plasticity-stability trade-off are poised to drive the next phase of developments in CIL (He et al., 25 Apr 2025).