Revisiting Prototype Rehearsal for Exemplar-Free Continual Learning: Manifold-Aware Boundary Sampling with Adaptive Class-Balanced Loss

Abstract: Exemplar-free class-incremental learning (EFCIL) aims to acquire new classes over time without storing raw data. Historically, prototype rehearsal, which samples around stored class prototypes and mixes them with current-task data, has been a popular strategy to reduce catastrophic forgetting. However, recent drift-compensation methods that explicitly realign prototypes in the evolving feature space consistently outperform prototype-based rehearsal, raising the question of whether rehearsal itself is fundamentally limited. We argue that the performance gap stems not from the idea of prototype rehearsal per se, but from how it is typically instantiated: existing approaches treat prototypes as isolated class summaries that ignore information from nearby enemy classes, and fail to correct the emerging class imbalance between a handful of synthetic old-class samples and hundreds of real instances from newly introduced classes. Building on this hypothesis, we revisit prototype rehearsal and propose a manifold-aware variant that restores its competitiveness in EFCIL. First, we introduce Constrained Expansive Over-Sampling, which interpolates each old-class prototype toward its nearest enemy features from new classes, generating boundary-aware rehearsal samples that better follow the underlying data manifold while preserving inter-class separation. Second, we design an Adaptive Class-Balanced loss that performs time-based class weighting, amplifying gradients from older prototypes when they are most informative and gradually annealing their influence as richer supervision from later tasks accumulates. Together, these components turn prototype rehearsal into a drift-resilient, imbalance-aware mechanism that closes, and often reverses, the gap to recent drift-compensation methods, achieving state-of-the-art performance across multiple EFCIL benchmarks.

• The paper introduces manifold-aware CEOS and Adaptive Class-Balanced Loss to mitigate prototype drift and class imbalance, enhancing decision boundary coverage.
• The paper employs targeted synthetic feature generation that respects the data manifold geometry, addressing the pitfalls of isolated prototype sampling.
• The paper demonstrates state-of-the-art performance on benchmarks, improving incremental learning accuracy by up to 7.9 percentage points.

Revisiting Prototype Rehearsal for Exemplar-Free Continual Learning: Manifold-Aware Boundary Sampling with Adaptive Class-Balanced Loss

Introduction

Exemplar-free continual learning (EFCIL) presents the challenge of accumulating knowledge over sequential tasks in the absence of access to past data, typically because of privacy or memory restrictions. Prototype rehearsal, which generates synthetic features around stored class prototypes to mitigate catastrophic forgetting, has historically underperformed compared to prototype drift-compensation methods. This paper systematically analyzes the limitations of standard prototype rehearsal, identifying sources of performance degradation—isolated prototype sampling and emerging class imbalance—and proposes two new mechanisms: Constrained Expansive Over-Sampling (CEOS) and Adaptive Class-Balanced (ACB) Loss. Together, these innovations reinstate prototype rehearsal as a competitive paradigm, frequently surpassing state-of-the-art drift-compensation models.

Prototype Rehearsal: Classical Limitations

Conventional prototype rehearsal methods synthesize features by sampling from stationary Gaussians centered on class prototypes, ignoring the topography of the data manifold and the positions of competing classes. As a result, synthetic samples cluster too narrowly, rarely inhabiting regions proximal to decision boundaries, and therefore fail to reinforce challenging margins. Moreover, prototype rehearsal induces a temporally compounding class imbalance: while new classes are represented with diverse real data, older classes rely on a handful of drifting prototypes.

Figure 1: Naïve prototype rehearsal—Gaussian sampling excessively concentrates samples near class prototypes, leading to poor decision boundary coverage and prototype dominance loss.

Theoretical analysis confirms two fundamental pathologies: (1) as the number of tasks increases and the prototype-per-class budget remains fixed, old-class posterior probabilities under the softmax classifier approach zero; (2) prototype drift—caused by the continual evolution of the feature extractor—further detaches synthetic prototypes from the true data distribution, causing their informativeness to asymptotically vanish. Both effects bias the model decisively towards recent classes and rapidly erode early-task performance.

Proposed Method: CEOS and Adaptive Class-Balanced Loss

Constrained Expansive Over-Sampling (CEOS)

CEOS interpolates each prototype with its closest "enemy" (i.e., a different-class sample from current data), but crucially only generates synthetic samples on the manifold segment between prototype and enemy, restricted to remain on the correct side of the class boundary. The mixing coefficient is sampled adaptively for each prototype–enemy pair, with a theoretically derived lower bound ensuring margin preservation with respect to a local Mahalanobis metric. By construction, CEOS focuses on boundary regions, incrementally expanding old-class support toward, but not beyond, the decision boundary, resulting in much higher boundary coverage and improved margin maintenance.

Figure 2: Pipeline for task $t$—(1) rehearsal from per-class prototypes, (2) CEOS boundary-aware interpolation, (3) Adaptive Class-Balanced loss for temporal weighting.

Figure 3: Visualization of old vs. new features under different synthesis strategies: (a) spherical Gaussian, (b) heavy Gaussian, (c) random bidirectional interpolation (PRAKA), (d) the proposed CEOS boundary-aware interpolation.

Ablation experiments show that CEOS outperforms both additional Gaussian and random prototype–feature interpolation. Notably, introducing more Gaussian samples does not improve and can even degrade performance, emphasizing that boundary-aligned, manifold-conformal augmentation rather than increased off-manifold sampling is the key lever for improved retention.

Adaptive Class-Balanced (ACB) Loss

To address the prototype-induced long-tail class imbalance, ACB Loss maintains a time-dependent, virtual sample count per class. A class’s influence begins high immediately after introduction—where prototypes are most informative—then systematically decays as more real data appear in later tasks. This is implemented as a monotonically decreasing weight in the cross-entropy loss, parameterized to interpolate smoothly between fresh prototypes and heavily supervised classes. Importantly, this approach is more effective than standard focal loss in the continual setting, since it aligns imbalance correction with the non-stationary evolution of per-class sampling.

Experimental Results

Quantitative comparison against both prototype rehearsal and drift-compensation baselines, across diverse image classification datasets (CIFAR-100, TinyImageNet, ImageNet-100, CUB-200), establishes consistent superiority of the proposed method. On TinyImageNet (20 tasks), the model achieves $31.8\%$ $A_{\text{last}}$ and $44.3\%$ $A_{\text{inc}}$, exceeding LDC and other SOTA methods by up to $7.9$ percentage points. Even at greater task counts (e.g., 40 incremental tasks), the advantage persists. Critically, these gains are obtained without incurring substantial additional computational overhead compared to lighter rehearsal-based baselines.

Ablation studies confirm the effectiveness and synergy of CEOS and ACB loss. Using a single nearest enemy neighbor in CEOS yields the best result; utilizing multiple neighbors can introduce feature overlap and degrade separability, highlighting the importance of minimal, margin-focused support expansion.

Figure 4: On TinyImageNet, mean prototype–real class mean distance remains much lower with the proposed approach, indicating reduced prototype drift.

Practical and Theoretical Implications

The decoupling of the weakness of prototype rehearsal from the rehearsal principle itself has important implications for EFCIL. By developing rehearsal mechanisms that are both manifold- and imbalance-aware, prototype rehearsal can be rendered robust to both drift and long-tail bias, mitigating the necessity for explicit drift-compensation modules. Practically, this enables memory- and privacy-constrained deployments to avoid the cost or complexity of experience replay or advanced drift compensation, achieving SOTA performance with simpler architectures and faster training times.

Theoretically, the results motivate further study of prototype manipulation: both on how to reliably identify enemy features in highly non-stationary or high-dimensional settings, and on deeper integration with uncertainty quantification or hierarchical prototype structures.

Conclusion

This work provides a rigorous analysis and systematic revisioning of prototype rehearsal in exemplar-free continual learning. By designing a rehearsal process that is sensitive to the geometry of decision boundaries (via CEOS) and dynamically imbalance-aware (via ACB loss), the approach closes—and often reverses—the performance gap with contemporary drift-compensation methods. Remaining limitations include the reliance on the informativeness of single prototypes under severe drift and potential ambiguity under poor local data coverage. Future directions include further drift-aware prototype weighting, adaptive mixing informed by confidence calibration, and extension to transformer-based and multimodal continual learning systems.

In sum, this framework reestablishes prototype rehearsal as an effective, efficient, and theoretically principled foundation for exemplar-free continual learning (2606.05695).