Distributional Replay in Continual Learning

• Distributional replay is a continual learning strategy that models data distributions to generate synthetic samples, thereby avoiding storage of raw past examples.
• It utilizes approaches like marginal, conditional, and feature-space replay to balance computational efficiency, label accuracy, and privacy constraints.
• Despite its success, the method’s effectiveness is sensitive to replay buffer size, sample selection, and task geometry, driving ongoing research for optimization.

Distributional replay is a class of continual learning strategies in which the system mitigates catastrophic forgetting by replaying samples generated from an explicit or implicit model of the data distribution, rather than by direct storage and rehearsal of real past examples. Techniques in this family model the evolving feature or data distribution across tasks, enabling pseudo-rehearsal, supporting privacy constraints (by avoiding raw data storage), providing explicit mechanisms for out-of-distribution (OoD) detection, and, in some cases, addressing the challenge of forward transfer. Although distributional replay has shown empirical and theoretical effectiveness, recent results indicate that its efficacy is highly sensitive to task geometry, replay buffer size, and sample selection heuristics.

1. Conceptual Foundations of Distributional Replay

Distributional replay originated as a memory-efficient alternative to classical rehearsal strategies in continual learning. Instead of retaining a subset of raw data from earlier tasks, the approach maintains a compact generative model that approximates the joint or marginal distribution of observed inputs (and possibly labels). As new tasks are introduced, synthetic samples from this generative model are interleaved with real samples from the current task during continued training, thereby stabilizing learned representations and alleviating forgetting. Notably, distributional replay contrasts with regularization-based methods (e.g., EWC, SI), which constrain parameter drift directly but do not re-expose the network to prior modes of the data distribution (Lesort et al., 2018, Lemke et al., 2024).

Two canonical variants are:

• Marginal Replay: The generator models only the marginal $p(x)$ over all past tasks; sample labels require separate inference by a frozen classifier.
• Conditional Replay: The generator learns $p(x|y)$, sampling both features and labels jointly via conditioning, thus eliminating label inference error (Lesort et al., 2018).

The generality of distributional replay encompasses applications in vision, medical imaging, and reinforcement learning, with key distinctions in the level at which generation and replay occur—raw input space, feature space, or in the state-action space of RL agents.

2. Methodologies and Architectures

Classification and Segmentation

In continual image classification, GANs or VAEs are commonly used as generative models for distributional replay. Marginal replay requires O($tN$) synthetic examples at task $t$ to maintain class balance since generated samples are class-agnostic and must be labeled ex post facto. By contrast, conditional replay efficiently produces O($N$) examples with correct labels by design, significantly reducing computational overhead and error rates when class-conditional mode complexity is manageable (Lesort et al., 2018).

For continual medical image segmentation, such as continual MRI segmentation, distributional replay is implemented via a two-stage architecture: a base UNet encoder performs feature extraction, and a conditional variational autoencoder (ccVAE) models the feature distribution in latent space. The ccVAE is conditioned on task and slice indices, allowing sampling of pseudo-features for replay and serving as a deep density estimator for OoD detection (Lemke et al., 2024).

Reinforcement Learning

In continual offline RL, distributional replay can be instantiated via generative models over state-action pairs. For example, the CuGRO framework decouples the policy into a score-based diffusion state generator and a diffusion-based behavior (action) generator, both conditioned on task identity. Pseudo-samples from these generators are used to augment new-task data, regularizing the multi-head critic and preventing forgetting of value functions learned in previous tasks (Liu et al., 2024). In distributed actor-critic methods such as D4PG, distributional replay refers specifically to the use of prioritized replay buffers and categorical (distributional) parameterization of the value function, enabling the learning and replay of value distributions (Barth-Maron et al., 2018).

Feature-Space and Activation Replay

Recent advances have recognized the importance of constraining distributional drift at the feature level. Compressed Activation Replay (CAR) stores compressed codes of intermediate activations—usually via average pooling—alongside input-output pairs, achieving superior regularization over vanilla experience replay when buffer sizes are small and raw activation storage is infeasible. This restricts drift in representational space, sharply reducing forgetting in both large-scale multitask learning and standard continual benchmarks (Balaji et al., 2020).

3. Empirical Effectiveness and Comparative Performance

Empirical benchmarks on MNIST/FashionMNIST, Split-CIFAR, Taskonomy, and medical segmentation datasets consistently demonstrate that distributional replay methods (GAN/WGAN-GP, CGAN, CVAE, ccVAE, CAR) outperform parameter-regularization schemes, especially in the presence of distributional shift or small memory budgets (Lesort et al., 2018, Lemke et al., 2024, Balaji et al., 2020). Table-based results indicate:

Method	Forgetting (%)	Final Acc. (%)	Memory Growth	Label Quality
EWC	Catastrophic	≈20 (MNIST)	O(1)	N/A
Marginal Replay (GAN)	≈5	93–94 (MNIST)	O(t×N)	Requires inference
Conditional Replay	≈5–8	91–93 (MNIST)	O(N)	By construction
CAR	13–18 (Taskonomy)	62 (Split-CIFAR)	Negligible	Intact
ccVAE (Segmentation)	–1.3 (BWT)	87.8 (Dice)	O(1)	Native
CuGRO (RL)	Near zero	Near-oracle	O(1)	N/A

A common observation is that marginal replay achieves marginally higher accuracy if computational resources permit large sample budgets, but conditional or feature/activation replay strategies are strongly preferred for efficiency, privacy, and pipeline simplicity. Methods like ccVAE provide explicit privacy guarantees by transiently buffering only synthetic latent feature codes, which are not invertible to raw images (Lemke et al., 2024).

4. Distributional Drift, Sample Selection, and Theoretical Analysis

Distributional replay aims to anchor the model’s representation in the union of all previously encountered task distributions, but empirical and theoretical results have established that its effectiveness is non-trivial and context-dependent. For small replay buffers, especially under sample-level randomization, catastrophic drift in intermediate features can still occur, as visualized by t-SNE in CAR experiments (Balaji et al., 2020). Moreover, recent theoretical analysis has demonstrated in continual over-parameterized regression that replay—both worst-case and average-case (i.e., distributional)—can increase forgetting in certain regimes, especially when the subspaces corresponding to different tasks are geometrically close in parameter space (Mahdaviyeh et al., 4 Jun 2025).

The critical factors affecting forgetting include:

• The principal angles between task nullspaces: If they are small ($<\pi/4$), small replay buffers can push the effective solution toward peak-forgetting configurations.
• The size and diversity of the replay buffer: Sufficiently large buffers or subspace coverage safely recovers monotonic improvement.
• The selection heuristics for replayed samples: Harmful replay can arise when the buffer contains unrepresentative or adversarially chosen samples.

This analytic finding holds for both linear models and deep neural networks trained with SGD, and is substantiated empirically for both regression and classification scenarios.

5. Out-of-Distribution Detection and Forward Transfer

A unique advantage of modeling the full data distribution is the natural emergence of mechanisms for OoD detection. In continual MRI segmentation, the reconstruction error from the ccVAE acts as a probabilistic density proxy, allowing explicit identification and rejection of OoD slices by thresholding (Lemke et al., 2024). In continual RL scenarios, task-conditioned diffusion models enable highly expressive coverage of progressive task distributions, supporting forward transfer by expanding the generative capacity of behavior and state models (Liu et al., 2024).

By maintaining an evolving generative model, distributional replay directly enables the integration of density estimation and replay functions, which can be leveraged for both model-selection (task-detection) and reliability estimation at test time.

6. Limitations, Practical Considerations, and Future Directions

Distributional replay does not guarantee monotonic reduction in forgetting across all settings. Key limitations and recommendations include:

• Harmful replay is possible if replay is performed with small, unrepresentative, or adversarially correlated sample buffers, especially when task distributions are non-orthogonal (Mahdaviyeh et al., 4 Jun 2025).
• Feature-level and activation-code replay (e.g., CAR) can require careful tuning of the compression and regularization parameters for effectiveness (Balaji et al., 2020).
• Joint training of generative and predictive models can be sensitive to class-conditional complexity and mode-collapse in high-dimension or nonstationary feature spaces (Lesort et al., 2018).
• Privacy, memory, and computational efficiency considerations motivate the preference for conditional replay or models that replay in latent or feature space (Lemke et al., 2024).

Recommended practice includes monitoring forgetting as a function of buffer size, ensuring subspace coverage in the replay buffer, and integrating explicit OoD detection where possible. Promising directions involve hybrid replay modalities, continual adaptation of the generative models themselves, and scaling distributional replay to complex real-world settings where both privacy and long-term knowledge retention are critical.