Preventing Latent Rehearsal Decay in Online Continual SSL with SOLAR

Abstract: This paper explores Online Continual Self-Supervised Learning (OCSSL), a scenario in which models learn from continuous streams of unlabeled, non-stationary data, where methods typically employ replay and fast convergence is a central desideratum. We find that OCSSL requires particular attention to the stability-plasticity trade-off: stable methods (e.g. replay with Reservoir sampling) are able to converge faster compared to plastic ones (e.g. FIFO buffer), but incur in performance drops under certain conditions. We explain this collapse phenomenon with the Latent Rehearsal Decay hypothesis, which attributes it to latent space degradation under excessive stability of replay. We introduce two metrics (Overlap and Deviation) that diagnose latent degradation and correlate with accuracy declines. Building on these insights, we propose SOLAR, which leverages efficient online proxies of Deviation to guide buffer management and incorporates an explicit Overlap loss, allowing SOLAR to adaptively managing plasticity. Experiments demonstrate that SOLAR achieves state-of-the-art performance on OCSSL vision benchmarks, with both high convergence speed and final performance.

• The paper proposes SOLAR, a novel approach that prevents latent rehearsal decay by monitoring latent space metrics, Deviation and Overlap.
• It employs a deviation-aware buffer and explicit overlap loss to dynamically balance stability and plasticity during online continual learning.
• Empirical results across benchmarks show that SOLAR delivers state-of-the-art accuracy and robust performance even with constrained memory.

Preventing Latent Rehearsal Decay in Online Continual SSL with SOLAR

Introduction and Motivation

The paper "Preventing Latent Rehearsal Decay in Online Continual SSL with SOLAR" (2604.10586) addresses Online Continual Self-Supervised Learning (OCSSL), where models confront unlabeled, non-stationary data streams and memory replay is essential. The authors challenge the prevailing dichotomy in online continual learning (CL): stability-biased strategies (e.g., Reservoir buffers) ensure resistance to forgetting, whereas plasticity-biased mechanisms (e.g., FIFO buffers) offer fast adaptability at the expense of stability. Contrary to canonical expectations, they demonstrate that Reservoir buffers, although converging rapidly under short training schedules, exhibit catastrophic latent representation degradation—termed Latent Rehearsal Decay—with increased training length, ultimately resulting in marked accuracy drops. This phenomenon is accompanied by distinctive changes in two newly introduced latent space metrics, Overlap and Deviation.

Figure 1: Motivation and overview. (a) Reservoir buffer achieves faster convergence but causes Latent Rehearsal Decay. (b) Deviation and Overlap metrics characterize latent structure quality. (c) SOLAR mitigates these issues and improves representation quality.

The authors propose SOLAR (Self-supervised Online Latent-Aware Replay), which leverages online proxies for these metrics to dynamically balance stability and plasticity. SOLAR enhances buffer composition via deviation-aware sample selection and explicitly regularizes Overlap during training, thereby suppressing latent space collapse and yielding superior state-of-the-art (SOTA) performance on multiple benchmarks.

Analysis of Stability–Plasticity Dynamics in OCSSL

Standard online CL replay strategies, such as Reservoir and FIFO, represent the two ends of the stability–plasticity spectrum. The paper demonstrates a non-classical regime in OCSSL:

• Reservoir buffers: Unbiased sampling promotes stability and quick convergence with few passes over data but ultimately stagnate, overfitting to early or static content and resulting in latent representational specialization. This is quantified as Latent Rehearsal Decay—a phase marked by sudden accuracy drops after a period of stable or improving learning.
• FIFO buffers: By inserting only the most recent samples and eliminating older ones, FIFO buffers facilitate plasticity, improving over long schedules at the expense of slow initial convergence.

Figure 2: Impact of training length. Reservoir achieves fast convergence for short regimes but suffers substantial accuracy degradation as training length increases; FIFO does not.

Latent Rehearsal Decay: Characterization and Metrics

Latent Rehearsal Decay describes the progressive collapse of latent feature spaces when a model is repeatedly trained on a limited, unchanging set of samples, suppressing adaptability to new data and harming generalization. The paper introduces two key metrics:

• Deviation: Intra-sample diversity, computed as average pairwise cosine distance among augmentations (hyperball spread). Collapse manifests as a sharp decrease in Deviation.
• Overlap: Inter-sample ambiguity, measured via overlap of different samples’ hyperballs (average angular separation). Collapse triggers a sustained rise in Overlap.

Empirical results affirm that catastrophic performance drops in Reservoir buffers are presaged by these metrics' collapse, while FIFO maintains healthy latent dynamics.

Figure 3: Latent Rehearsal Decay: Reservoir exhibits abrupt Deviation collapse and Overlap surge, directly preceding accuracy crashes, unlike FIFO.

The SOLAR Methodology

SOLAR is designed to adaptively modulate plasticity and stability by integrating two core innovations:

1. Deviation-Aware Buffer: Instead of population-agnostic accumulation, the buffer tracks per-sample statistics—EMA of SSL loss (a linear proxy for Deviation), average representation, and extraction count. Buffer insertion and replay preferentially select high-Deviation (high-loss, under-trained) samples, discarding those with best-converged (i.e., lowest loss/Deviation) representations and tracking under-sampled content.
2. Explicit Overlap Loss: To directly combat latent space crowding, SOLAR introduces a regularizer that penalizes positive Overlap between current minibatch samples and top high-Deviation buffer entries. This guides the encoder to maximize distinction among overlapping latent representations, enforcing global feature diversity and safeguarding against representational collapse.

Figure 4: SOLAR overview: buffer tracks loss, Deviation, representation statistics, discards low-Deviation samples, and uses Overlap loss to adaptively regularize representation space.

The total OCSSL loss for SOLAR is:

$$\mathcal{L} = \mathcal{L}_{SSL} + \omega \mathcal{L}_{overlap}$$

where $\omega$ modulates the contribution of Overlap regularization.

Empirical Results and Numerical Findings

SOLAR consistently achieves the best trade-off between final representation quality and convergence rate across all evaluated datasets (CIFAR-100, ImageNet100, CLEAR100, and iNaturalist). Key findings include:

• State-of-the-art final and average accuracy: SOLAR outperforms all baselines—including explicit distillation-based methods—independently of training duration or buffer size.

Figure 5: SOLAR maintains superior performance across all training lengths, outperforming both fast-converging (Reservoir) and plasticity-dominant (FIFO, FIFO-distillation) baselines.

• Buffer efficiency and robustness: SOLAR exhibits insensitivity to buffer size; performance remains superior and stable even under highly constrained memory, while Reservoir and other sample-prioritization techniques degrade rapidly as buffer shrinks Figure 6.
• Prevention of catastrophic latent collapse: SOLAR’s combination of Deviation tracking and Overlap regularization directly stabilizes the latent metrics, maintaining both representational richness and inter-sample separability throughout extended training regimes.
• Explicit ablations confirm that removing Overlap regularization mainly impairs final accuracy (stability), while eliminating deviation-aware buffer policies reduces convergence (plasticity).

Theoretical and Practical Implications

This work demonstrates that classic OCSSL forgetting analysis is insufficient and that latent space pathology—not catastrophic forgetting—is the principal failure mode under prolonged online replay. The findings suggest that methods focused solely on stability (e.g., Reservoir or over-strong distillation) can paradoxically induce representational collapse regardless of overfitting avoidance or memory size. As such, effective OCSSL must employ explicit, adaptive management of buffer composition and latent space regularization.

Figure 7: Training trajectories across buffer sizes confirm that SOLAR suppresses Latent Rehearsal Decay and yields faster, more stable convergence than FIFO/Reservoir.

This insight extends to the design of online and continual learners in both self-supervised and supervised settings, highlighting the necessity for fine-grained, sample-aware buffer and loss designs as opposed to monolithic memory or regularization schemes.

Future Developments

The study raises several avenues for additional research:

• Task-aware and class-aware extensions: Integrating task or class labels could further optimize buffer dynamics and latent regularization by leveraging known semantic structure.
• Generalization across architectures and modalities: Adapting Deviation/Overlap-guided mechanisms for transformers and multi-modal inputs could yield robust OCSSL for wider real-world deployment.
• Latent Rehearsal Decay in supervised online CL: The hypothesis of latent space collapse may generalize beyond SSL, indicating a common pathology for all forms of over-stabilized replay.

Conclusion

This paper rigorously formulates and experimentally validates Latent Rehearsal Decay as a critical failure mode in OCSSL, characterized by distinctive latent space metrics. It introduces SOLAR, which adaptively balances stability and plasticity by dynamically managing buffer diversity and regularizing overlap in latent space, thus preventing collapse and securing state-of-the-art OCSSL performance (2604.10586). The approach underscores the importance of continual latent quality diagnostics and adaptive buffer/loss strategies in online self-supervised learning frameworks, providing a blueprint for future advances in continual and lifelong vision representation learning.