Prototype-Guided Replay in Continual Learning

• The paper demonstrates that prototype-guided replay summarizes past knowledge using low-dimensional embeddings, achieving near-zero forgetting on benchmarks.
• It employs tailored replay losses and dynamic buffer management strategies that preserve latent space geometry, enhancing stability-plasticity tradeoffs.
• Prototype-guided replay is adaptable across tasks like classification, regression, and generative modeling, utilizing label-free and unsupervised methods for memory efficiency.

Prototype-guided replay is a class of memory-efficient continual learning techniques in which representative “prototypes”—typically low-dimensional embeddings or centroids in model latent space—are used to summarize and replay past knowledge. This approach addresses catastrophic forgetting by using prototypes as anchors for replay-based rehearsal, domain alignment, or generative conditioning. Strategies range from explicit feature-space herding, variational and generative schemes, through to fully unsupervised or label-free memory buffers. Prototype-guided replay has been successfully deployed for classification, regression, and even deep diffusion models, demonstrating marked improvements in stability-plasticity tradeoff, memory efficiency, and, in some settings, outperforming standard offline learning.

1. Prototype Extraction, Update, and Memory Management

Prototype extraction generally entails computing feature-space centers or centroids, using methods such as mean-pooling, clustering, or adaptive criteria per class or cluster. For example, ProCA (Lin et al., 2022) constructs, per class $k$, a set of $T$ prototypes $p^k_m$ by identifying pseudo-labeled target samples whose feature representations (via the current $G(\cdot)$) are nearest to the per-class empirical center $f^k_t$

$$f_t^k = \frac{1}{|\mathcal D^k_t|} \sum_{x \in \mathcal D^k_t} G(x)$$

and selects prototypes iteratively by minimizing the distance to the incremental mean (see Algorithm 1 in (Lin et al., 2022)).

Memory management incorporates:

• Buffer size limits: Either fixed quota per class (e.g., $T=20$ in ProCA, $M=31$ per class in LRCP (Aghasanli et al., 9 Apr 2025)), or dynamic sizing using statistical criteria—such as the “Kaiser criterion” or intracluster variance “knee” (Brignac et al., 2023).
• Replacement/merging heuristics: Examples include discarding oldest prototypes, merging via K-means, or enforcing temporal or representational diversity.
• Label-free mechanisms: Some frameworks (e.g., iSL-LRCP) store only latent representations with input references, omitting explicit class labels (Aghasanli et al., 9 Apr 2025).

This memory-efficient encoding enables substantial reduction in storage (often <$\!2\%$ of the raw data replay requirement (García-Santaclara et al., 10 Feb 2026)), supports privacy constraints, and mitigates drift.

2. Replay Mechanisms and Loss Functions

Prototype-guided replay pipelines involve both explicit rehearsal (sampling prototypes or their original samples) and loss terms that preserve the geometric structure of latent spaces:

• Classification loss on prototypes: E.g., ProCA employs a replay classification loss $$\mathcal L_{\rm RP} = -\frac{1}{|\mathcal M|}\sum_{(p,y)\in\mathcal M} \log h(G(p))_y$$ to enforce stability of old-class representations (Lin et al., 2022).
• Prototype/domain alignment loss: Aligns source and target class centers or prototypes (e.g., $T$0).
• Cluster preservation loss: In iSL-LRCP, a maximum mean discrepancy loss ($T$1) is computed between old and new buffer embeddings to preserve intra-class and inter-class geometry (Aghasanli et al., 9 Apr 2025).
• Push-away and pull-toward losses: For class-incremental settings, a push-away term increases angular distance between new samples and old prototypes; for domain-incremental, a pull-toward term preserves domain invariance.

In generative settings, prototype vectors serve as conditional anchors for sample generation (e.g., class-prototype conditional diffusion, see Section 3).

3. Prototype-Guided Generative Replay

Recent advances combine prototype guidance with generative models:

• Diffusion-based replay: GPPDM uses learnable prototype vectors $T$2 per class as persistent conditioning channels in the denoising process of diffusion models. Prototypes are updated jointly with the denoising network and guide reverse sampling, ensuring generated replay data remain class-representative and high-fidelity (Doan et al., 2023).
• Gradient projection: To prevent interference in the generator, gradients w.r.t. cross-attention weights are projected to remain orthogonal to earlier task subspaces, thereby stabilizing old-task representations.
• Pipeline integration: At each task, generated replay samples from previous prototypes are interleaved with real data to train the classifier. Ablations confirm that prototype-guided conditioning improves both replay sample quality and forgetting metrics over prior diffusion replay approaches.

In regression, prototypes (input–output pairs associated with output-space discretization) can be used to generate synthetic rehearsal data, bridging classification and continuous prediction (García-Santaclara et al., 10 Feb 2026).

4. Unsupervised and Label-Free Prototype Replay

Label-free buffers extract and select prototypes without relying on explicit class labels:

• Latent space clustering: After each task, unsupervised K-means clustering over model embeddings produces cluster centers (“prototypes”) and supports along variance axes. No label information is stored or used in selection or rehearsal (Aghasanli et al., 9 Apr 2025).
• Cluster-preservation losses: Alignment of latent distributions across tasks is enforced without relying on class supervision, ensuring old clusters remain distinct and stable.
• Support for unsupervised continual learning: Unsupervised label-free prototype selection has demonstrated competitive, and in cases superior, performance to label-dependent baselines.

These strategies yield privacy-aware, compact replay buffers and are particularly robust to class and domain drift.

5. Prototype Selection Strategies and Buffer Optimization

Effective prototype-guided replay depends critically on instance selection, buffer allocation, and update policies:

• Herding (class-mean/prototype): Widely used, selecting buffer samples closest in feature space to class mean (prototype). Shown to outperform random (reservoir) sampling in both average accuracy and forgetting under constrained memory (Brignac et al., 2023).
• IPM (Iterative Projection and Matching): Selects representative samples that best approximate task subspace coverage.
• Dynamic buffer sizing: Data-driven buffer population (e.g., Kaiser criterion or intracluster variance “knee” finding) adapts the number of prototypes per class to class complexity, frequently enabling near-optimal forgetting reduction at 3x-5x reduced memory compared to fixed quotas (Brignac et al., 2023).

Empirical findings consistently highlight the class and structure–aware selection (herding/IPM) as superior to uniform or diversity-based strategies (e.g., GSS) for mitigating forgetting in tight memory regimes.

6. Empirical Effectiveness and Limitations

Prototype-guided replay approaches consistently surpass traditional experience replay or regularization schemes on benchmarks for class-incremental, domain-incremental, and task-free continual learning:

• Near-zero forgetting: ProCA achieves <1% drop in step-1 accuracy, with final accuracies up to 87.8% (Office-31-CI), outperforming all baselines by large margins (Lin et al., 2022).
• Label-free approaches: iSL-LRCP attains average accuracy up to 92.87% (SplitCaltech256) surpassing offline training and strong backward transfer (Aghasanli et al., 9 Apr 2025).
• Generative replay: GPPDM achieves +17.2 pp improvement in final average accuracy on CIFAR-100-CI with ablations confirming the unique contributions of prototype-guided conditioning (Doan et al., 2023).
• Regression: In non-stationary continual regression, prototype-guided replay enables stable performance with <2% prototype/sample ratio (García-Santaclara et al., 10 Feb 2026).

Limitations include buffer management sensitivity (size, merging, replacement policies), slightly lower absolute accuracy than full batch training in highly non-linear settings, and potential underrepresentation of rare modes or classes under strict memory constraints.

7. Generalizations and Prospects

Prototype-guided replay mechanisms generalize naturally beyond their original settings:

• Modality-agnostic: Effective in vision, text, and regression, and compatible with both supervised and unsupervised pipelines.
• Integration with meta-learning, generative, and clustering frameworks: Providing versatile memory-compact rehearsal compatible with parameter-space regularization and contrastive objectives.
• Future research: Includes the development of fully label-free domain-incremental pipelines, learned adaptive buffer management, and richer generative models augmenting prototype replay to further bridge plasticity and stability.

Collectively, prototype-guided replay represents a flexible, data-efficient, and theoretically grounded solution to catastrophic forgetting in continual learning, with broad applicability across domains and learning paradigms (Lin et al., 2022, Doan et al., 2023, García-Santaclara et al., 10 Feb 2026, Aghasanli et al., 9 Apr 2025, Brignac et al., 2023, Zhang et al., 2019, Ho et al., 2021).