Prototype-Based Continual Learning

• Prototype-based continual learning is a framework that represents classes as learned prototypes in an embedding space, facilitating incremental and memory-efficient model updates.
• It employs techniques like online averaging, momentum update, and contrastive losses to robustly update prototypes and prevent catastrophic forgetting.
• Empirical results show that such methods achieve superior performance in class-incremental, federated, and few-shot settings compared to traditional buffer-based approaches.

Prototype-based continual learning encompasses a family of frameworks that represent each class or concept by a vector (or distribution) in an embedding space and use these prototypes to enable incremental learning, mitigate catastrophic forgetting, and facilitate efficient model updates. Prototypes summarize class information, support efficient memory management, and offer compatibility with both supervised and unsupervised, as well as federated and decentralized learning scenarios. This article systematically details the central motivations, formal methodologies, variants, and notable empirical findings in prototype-based continual learning, with a focus on settings and advances documented in the research literature.

1. Fundamental Principles and Definitions

Prototype-based continual learning maintains for each class or concept a prototype—typically a centroid or a set of summary statistics—in the learned embedding space. The most canonical form utilizes the nearest-class-mean rule, classifying an input $x$ as

$\hat{y} = \arg\min_{c}\; \|f_\theta(x) - P^c\|_2$

where $f_\theta$ is the embedding function (often a neural network) and $P^c$ is the prototype for class $c$. Prototypes may be simple empirical means (Hendryx et al., 2021, Li et al., 2023), variational distributions (Zhang et al., 2019), or even ensembles or learnable vectors (Valerio et al., 8 Dec 2025, Doan et al., 2023).

Continual learning settings include class-incremental, domain-incremental, task-incremental, or online data-incremental regimes. Class prototypes are updated as new data (or tasks) are encountered, either by simple averaging, momentum, or more sophisticated adaptation rules including online averaging or metaplasticity (Hajizada et al., 2024, Hendryx et al., 2021). Some systems further support open-world and novelty detection via thresholding prototype-to-sample distances (Hajizada et al., 2024).

2. Prototype Maintenance, Update, and Regularization

The critical challenge in continual learning is preventing catastrophic forgetting, which prototype-based methods address via various mechanisms for prototype update and embedding stability:

• Online Averaging and Merging: Prototypical continual learning methods aggregate new examples via online mean updates or numerically stable recursive schemes (West's algorithm):

$$\mu_{j,t} = \mu_{j,t-1} + \frac{k_{j, \text{new}}}{k_{j, t}}(\bar z_j^{(\text{new})} - \mu_{j,t-1})$$

This ensures that each class prototype integrates new information without erasing older knowledge (Hendryx et al., 2021).

• Momentum and Metaplasticity: To control plasticity and stability, per-prototype adaptive learning rates are used, e.g., $\alpha_t^i = 1/g_t^i$, where the "goodness" score $g_t^i$ consolidates correct predictions (Hajizada et al., 2024). Prototypes become progressively less malleable as they prove consistent, echoing neurobiological plasticity stabilization.
• Contrastive and Equilibrium Objectives: Techniques such as Online Prototype Equilibrium (OPE) (Wei et al., 2023) employ batch-level prototype-wise contrastive losses to promote inter-class separation and maintain actively adjusted, robust cluster geometry even as tasks shift.
• Prototype-Sample Distillation: To maintain the semantic relevance of stored prototypes as feature extractors drift, distillation losses enforce the preservation of sample-to-prototype similarity ranks across tasks (Asadi et al., 2023), thereby anchoring class boundaries without replaying raw data.
• Drift Compensation: Learnable linear maps re-align prototypes into new embedding spaces across tasks, allowing prototype sets to remain up-to-date without requiring access to full historical data or assuming that backbone changes can be modeled by simple translations (Gomez-Villa et al., 2024).
• Hierarchical Structures: In structured data, prototypes may be maintained at atomic, node, or class hierarchies (e.g., for graphs), with mathematically bounded memory and provable non-forgetting guarantees under separation assumptions (Zhang et al., 2021).

3. Integration with Replay, Federated, Generative, and Prompt-based Learning

Prototype-based approaches extend naturally to scenarios fundamental to modern machine learning practice:

• Memory-Efficient and Label-Free Replay: Rather than store raw exemplars, some systems build a compact, label-free replay buffer of prototypes and support vectors. Loss terms such as cluster preservation (e.g., MMD$^2$ on stored representations) ensure that cluster topology is maintained even as new tasks distort latent geometry. "Push-away" and "pull-toward" losses decouple adaptation and retention (Aghasanli et al., 9 Apr 2025, Lange et al., 2020).
• Federated and Vertical Federated Learning: Prototype exchange is computationally and semantically efficient in federated settings. Merging of online prototypes enables group models to be swiftly updated with low communication bandwidth and strong resilience against catastrophic forgetting, avoiding the need to transmit gradients or model weights (Hendryx et al., 2021, Wang et al., 13 Feb 2025). In vertical settings, prototypes encode feature evolution across partitions and tasks, acting as knowledge carriers in privacy-preserving distributed computation.
• Generative Replay via Prototypes: Diffusion and other generative replay models can be conditioned on learnable prototypes, enabling high-fidelity sampling of previously observed domains or classes for regularization and classifier rehearsal (Doan et al., 2023).
• Few-Shot, Meta-, and Prompt-based Variants: In low-data regimes, prototypes operate as anchors for meta-learning, maximizing adaptation to new classes while leveraging a rehearsal buffer optimally via dynamically updated prototype-guided sample selection (Li et al., 2023, Zhang et al., 2019). Prompt-tuned approaches in vision and text utilize prototypes to steer prompt selection and alignment, greatly reducing catastrophic forgetting and closing the sequential-vs-joint learning gap (Li et al., 2023, Mei et al., 1 Apr 2026, Tran et al., 2023).

4. Formal Algorithms and Resource Efficiency

Prototype-based continual learning methods are characterized by straightforward, resource-conscious update rules:

Method	Storage per class	Communication	Memory growth
Online-avg ProtoNet (Hendryx et al., 2021)	$O(d)$	$\hat{y} = \arg\min_{c}\; \|f_\theta(x) - P^c\|_2$0	Linear in $\hat{y} = \arg\min_{c}\; \|f_\theta(x) - P^c\|_2$1
Label-free replay (Aghasanli et al., 9 Apr 2025)	$\hat{y} = \arg\min_{c}\; \|f_\theta(x) - P^c\|_2$2	N/A	Linear in $\hat{y} = \arg\min_{c}\; \|f_\theta(x) - P^c\|_2$3
CoPE momentum (Lange et al., 2020)	$\hat{y} = \arg\min_{c}\; \|f_\theta(x) - P^c\|_2$4	N/A	Linear in $\hat{y} = \arg\min_{c}\; \|f_\theta(x) - P^c\|_2$5
Prototypical Prompt (Li et al., 2023)	$\hat{y} = \arg\min_{c}\; \|f_\theta(x) - P^c\|_2$6	N/A	Linear in $\hat{y} = \arg\min_{c}\; \|f_\theta(x) - P^c\|_2$7
OVA prompt buffer (Tran et al., 2023)	$\hat{y} = \arg\min_{c}\; \|f_\theta(x) - P^c\|_2$8	N/A	Linear in $\hat{y} = \arg\min_{c}\; \|f_\theta(x) - P^c\|_2$9
Variational ProtoReplay (Zhang et al., 2019)	$f_\theta$0 + 1 image	N/A	Linear in $f_\theta$1
Exemplar-based (iCaRL)	$f_\theta$2	$f_\theta$3	Linear in $f_\theta$4 (with $f_\theta$5 exemplars)

Here, $f_\theta$6 is embedding dimension, $f_\theta$7 is number of classes, $f_\theta$8 is number of tasks, and $f_\theta$9 is buffer size per class or task.

Efficient merging (e.g., West’s update), label-free buffers, and generative replay minimize both memory and data privacy footprints. For streaming or robotics scenarios with single-pass, few-shot data and no replay, metaplastic update rules further guarantee bounded, adaptive model size (Hajizada et al., 2024).

5. Empirical Results and Comparative Performance

Prototype-based continual learning consistently demonstrates competitive or superior resilience to catastrophic forgetting and adaptation latency, as substantiated by published benchmarks:

• Class-Incremental and Federated Benchmarks: On Omniglot (600 classes) and mini-ImageNet (20 classes), prototype averaging networks achieve 22–33 points higher accuracy than complex regularization or buffer-based baselines, retaining >80% base-class accuracy post-sequential learning (Hendryx et al., 2021).
• Online Continual Learning: Online prototype equilibrium and adaptive feedback outperform buffer-based (e.g., iCaRL, MIR) and constraint-based (AGEM, DER++) methods by 5–15 percentage points in average accuracy across buffer sizes and benchmark datasets (Wei et al., 2023).
• Label-Free and Unsupervised Settings: Label-free cluster preservation approaches match or surpass state-of-the-art methods (offline, iCaRL, PRD, ER-AML) in average accuracy and backward transfer, even outperforming offline learning in some domains (Aghasanli et al., 9 Apr 2025).
• Few-Shot and Streaming: Continually Learning Prototypes set new state-of-the-art few-shot online continual learning (FS-OCL) results on OpenLORIS, achieving >99% base and up to 87% new-class accuracy with strictly single-pass, buffer-free protocols (Hajizada et al., 2024).
• Domain Extension and Structure: Hierarchical Prototype Networks scale continual learning to multi-million node graphs and achieve near-zero forgetting with bounded memory for node and class-level prototypes (Zhang et al., 2021).

6. Limitations, Challenges, and Future Directions

Prototype-based continual learning methods, while resource-efficient and empirically robust, face open challenges:

• Semantic Drift and Prototype Misalignment: As the embedding backbone changes, stored prototypes become semantically outdated unless updated or appropriately projected. Recent advances formalize drift compensation as a learnable linear or nonlinear map, achieving near-oracle performance without requiring access to all prior data (Gomez-Villa et al., 2024).
• Prototype Granularity and Scalability: With large numbers of classes or highly non-Gaussian/multimodal distributions per class, fixed-vector prototypes may be insufficient. Augmenting with sub-prototype mixtures or density estimation is a promising direction (Asadi et al., 2023).
• Label Scarcity and Unsupervised Settings: Fully label-free operation is achievable via unsupervised pseudo-contrastive objectives and cluster preservation; however, some accuracy gap persists versus supervised variants (Aghasanli et al., 9 Apr 2025).
• Rehearsal Buffer Constraints: For privacy- or bandwidth-constrained settings, prototype and support selection, as well as fully generative replay, provide strong trade-offs between memory usage and accuracy (Zhang et al., 2019, Doan et al., 2023).
• Task-Agnostic and Open World: Automatic novelty detection, prototype allocation with pseudo-labels, and semi-supervised learning continue to be extended; automatic adaptation to dynamic task boundaries remains an open research area (Hajizada et al., 2024, Asadi et al., 2023).

7. Broader Implications and Theoretical Guarantees

Prototype-based frameworks suggest a fundamental class of continual learning architectures that decouple fixed feature-space representations from light-weight, interpretable class summaries. They enable:

• Plug-and-play class expansion, with updates scaling linearly in new classes and requiring $P^c$0 memory/communication per class (Hendryx et al., 2021);
• Communication-efficient and privacy-preserving learning in federated or decentralized environments;
• Differential privacy via prototype noise addition rather than raw data sharing (Hendryx et al., 2021);
• Theoretical guarantees of bounded memory and zero forgetting (under separation conditions) in certain hierarchical and spherical prototype frameworks (Zhang et al., 2021);
• Seamless integration into prompt-based, generative, and meta-learning systems, including real-world continual, low-resource, or multi-modal domains.

Prototype-based continual learning thus establishes a widely applicable paradigm for scalable, robust, and efficient lifelong machine learning and distributed intelligence, actively pushing the empirical and theoretical boundaries as substantiated in the latest research corpus.