Latent Replay: Mechanisms & Applications

• Latent Replay is a rehearsal strategy that retains and reuses compact, internal representations to overcome catastrophic forgetting in continual learning.
• It exploits stable latent spaces from intermediate network layers, reducing computation and storage compared to raw-sample replay.
• Recent advances extend latent replay into generative and inference-time applications, impacting robotics, imaging, and foundation-model systems.

Latent replay is a family of rehearsal strategies in which previously learned information is retained and reused in an internal representation space rather than in the raw input space. In the dominant continual-learning usage, a model stores, regenerates, or resamples activations, latent codes, or latent distributions, then interleaves them with current-task data to mitigate catastrophic forgetting while reducing storage and computation relative to raw-sample replay (Pellegrini et al., 2019). Across the literature, the replayed object ranges from intermediate activations volumes and compressed feature tensors to class-conditional latent distributions, multimodal token sequences, and spike trains; more recently, the term has also been extended beyond continual learning to denote inference-time reuse of latent actions in transformer hidden states (Shi et al., 4 Feb 2026). The unifying idea is that internal states are often more compact, more semantically structured, and more stable than pixels, but the utility of latent replay depends critically on how that latent space is defined, stabilized, and sampled (Ostapenko et al., 2022).

1. Conceptual scope and historical development

The canonical formulation of latent replay arose in continual learning for edge and robotic systems, where rehearsal at input level was too expensive. The defining move was to store activations at an intermediate layer $L^\*$, replay them directly into the upper part of the network, and slow or freeze learning below $L^\*$ so that stored activations remained valid over time (Pellegrini et al., 2019). This established two principles that have remained central: replay should occur in a space cheaper than pixels, and representation drift must be controlled.

Subsequent work diversified the notion. One line used learned compressors that map images into compact discriminative latents, then replay only those stored latent exemplars without any decoder (Borde, 2021). Another line treated frozen foundation-model encoders as stable feature extractors and performed continual learning entirely in their latent spaces, sometimes with non-parametric classifiers such as nearest class mean or SLDA, thereby shifting the problem from end-to-end optimization to latent-space classification (Ostapenko et al., 2022). A third line replaced stored examples with latent-space generators or latent density models, including VAEs over CLIP embeddings, Gaussian mixtures over pathology features, and prototype-centered latent world models for class-incremental learning (Frascaroli et al., 2024).

Terminological broadening has produced an important ambiguity. In most continual-learning work, latent replay remains a training-time rehearsal mechanism over feature representations. In STIR, by contrast, “latent replay” denotes inference-time reuse of distilled corrective impulses in transformer residual space; the replayed objects are not past examples or features but latent actions retrieved and reinjected during reasoning (Shi et al., 4 Feb 2026). This broader usage preserves the reuse-of-internal-state intuition, but it is methodologically distinct from rehearsal-based continual learning.

2. Core mechanism: stable latent spaces, replay paths, and classifiers

In its standard form, latent replay decomposes a model into a lower mapping $h$ and an upper mapping $g$, so that

$f(x;\theta)=g(h(x;\theta_h);\theta_g).$

A replay item is the latent state

$z=h(x;\theta_h)$

computed at a chosen layer or by a chosen encoder. Current examples traverse both $h$ and $g$, whereas replayed latents are injected directly into $g$. The training objective is typically the sum of a current-task loss and a replay loss, with gradients from replay stopped below the replay layer or with the encoder fully frozen (Pellegrini et al., 2019).

The requirement that the latent space remain stable appears in multiple forms. In the original real-time formulation, learning below the replay layer is slowed or frozen and replay gradients do not flow into the lower layers, precisely to avoid invalidating stored activations (Pellegrini et al., 2019). In latent-space memory replay for classification, the encoder is trained jointly with the classifier on the first task, then frozen; a small percentage of first-task samples is stored as $6\times 6$ latent images and replayed directly to the classifier, yielding a compression ratio of $L^\*$0 relative to $L^\*$1 inputs (Borde, 2021). In foundation-model settings, the encoder is frozen from the outset, every image is encoded once, and continual learning proceeds only over cached latents; this substantially changes the compute regime and, in some cases, makes non-parametric classification competitive with gradient-based latent replay (Ostapenko et al., 2022).

The classifier on top of replayed latents can be either parametric or non-parametric. Parametric variants train lightweight heads or adapters on mixtures of current-task latents and replayed old-task latents. Non-parametric variants maintain class means or streaming covariance statistics in latent space, avoiding iterative replay training altogether (Ostapenko et al., 2022). This distinction is not merely computational: on in-distribution streams, the representation geometry of strong frozen encoders can make simple decision rules sufficient, whereas on fine-grained or out-of-distribution streams, additional latent-space adaptation remains beneficial (Ostapenko et al., 2022).

A recurrent misconception is that latent replay necessarily requires decoding back to pixels. Many methods do not decode at all. Some replay the latent directly into a classifier (Borde, 2021); others generate or sample in latent space and train only latent consumers (Nie et al., 28 Jun 2026). Decoder-based replay is one subclass, not the definition.

3. Distributional and generative formulations

A major branch of latent replay replaces stored latent exemplars with latent generators or latent density models. This shift changes replay from memorizing particular codes to modeling where a class or task “lives” in representation space.

In Prototype Latent World Model Replay, a frozen ImageNet-pretrained encoder $L^\*$2 defines a stable hidden state space $L^\*$3. For each class $L^\*$4, latent states are clustered into $L^\*$5 groups, and each prototype $L^\*$6 stores a mean $L^\*$7 and diagonal variance $L^\*$8. Replay samples are drawn from the class-conditional mixture

$L^\*$9

A lightweight adapter and classifier are then trained on sampled old latent states and real new-class features, optionally with a supervised contrastive term in adapter space. On Split CIFAR-100, the full model raises LastAcc from $h$0 to $h$1, from $h$2 to $h$3, and from $h$4 to $h$5 in Inc5, Inc10, and Inc20, respectively, without storing raw exemplars (Nie et al., 28 Jun 2026).

Other works instantiate latent replay through explicit generative models. CGIL fits one class-specific VAE per class in CLIP’s frozen visual embedding space, stores only the decoders, and later samples synthetic visual embeddings to realign class-specific and shared prompts during incremental prompt learning (Frascaroli et al., 2024). AGLR-CL fits per-class Gaussian Mixture Models on attention-filtered patch embeddings and a one-dimensional GMM on patch counts, then generates synthetic WSI latent bags for domain-incremental pathology without retaining any raw slides (Kumari et al., 13 May 2025). In text-to-image diffusion, latent replay stores Stable Diffusion VAE latents $h$6 together with prompts and replays them under the same denoising loss used for current data; a 512×512 float32 image is approximately $h$7 MB, whereas a $h$8 float32 latent is approximately $h$9 KB, a ratio of about $g$0 (Otani, 4 Sep 2025).

There are also decoder-only memory formulations. KRNet dispenses with both an encoder at replay time and per-sample latent-code storage by learning a direct mapping from discrete sample identity to the corresponding feature map. On ImageNet-Subset, the latent-code storage required by an autoencoder grows from $g$1 MB at 50 classes to $g$2 GB at 250 classes, whereas KRNet’s latent storage ranges from $g$3 MB to $g$4 MB over the same settings, yielding an approximately $g$5 reduction (Zhang et al., 2022). BinPlay pursues a different route, defining deterministic binary codes from sample indices via prime-based constructions, then decoding those codes to reconstruct past samples on the fly; because the codes are computed from indices, neither raw samples nor latent codes need to be stored (Deja et al., 2020).

These generative and distributional variants differ in what they assume about the latent manifold. Prototype mixtures and GMMs assume local Gaussian structure; VAEs learn class-conditional decoders; decoder-only recitation models treat replay primarily as memorization rather than distribution learning. What unites them is the relocation of replay from raw data to a feature space regarded as lower-dimensional, semantically organized, or more privacy-preserving than the original input domain.

4. Replay policies, scheduling, and stabilization mechanisms

Latent replay is not only a question of what is stored but also of which latent states are selected, how often they are replayed, and what additional constraints shape the evolving latent geometry.

One line varies replay by network depth. Progressive Latent Replay distributes replay frequency across several hidden layers through a schedule $g$6 with $g$7, replaying deeper layers more frequently because earlier layers forget less abruptly. If a replay batch is injected at layer $g$8, only layers $g$9 are updated. The corresponding update-cost surrogate

$f(x;\theta)=g(h(x;\theta_h);\theta_g).$0

formalizes the reduction in parameter updates relative to single-layer internal replay. On Split CIFAR-100, several such schedules improved accuracy while reducing relative cost $f(x;\theta)=g(h(x;\theta_h);\theta_g).$1 below that of Internal Replay (Pawlak et al., 2022).

Another line prioritizes replayed items by instability. Stable-Drift computes layer-wise latent drift between a source model $f(x;\theta)=g(h(x;\theta_h);\theta_g).$2 and a naively adapted model $f(x;\theta)=g(h(x;\theta_h);\theta_g).$3,

$f(x;\theta)=g(h(x;\theta_h);\theta_g).$4

aggregates drift across the final two backbone layers, then ranks patients by average drift and stores the top $f(x;\theta)=g(h(x;\theta_h);\theta_g).$5 slices per patient until a class-balanced 30,000-slice buffer is filled. On a cross-hospital COVID-19 CT task, this patient-aware multi-layer drift replay achieved per-patient H1/H2 accuracies of $f(x;\theta)=g(h(x;\theta_h);\theta_g).$6 with a Swin Transformer and yielded the best BWT among compared replay methods (Theofilou et al., 27 Nov 2025).

Spiking continual-learning systems introduce yet another policy dimension. SESLR stores binary spike-based latent features and adds a sleep phase after the online stream in which the classifier trains exclusively on a class-balanced replay buffer with additive Gaussian noise $f(x;\theta)=g(h(x;\theta_h);\theta_g).$7, $f(x;\theta)=g(h(x;\theta_h);\theta_g).$8. On Split CIFAR10, the best noise scale was $f(x;\theta)=g(h(x;\theta_h);\theta_g).$9, and the method increased average accuracy from $z=h(x;\theta_h)$0 to $z=h(x;\theta_h)$1 at buffer size 200 while using about one-third of the memory of naive experience replay (Lin et al., 23 Jun 2025).

In lifelong imitation learning, stabilization is expressed geometrically rather than through selection alone. Multimodal Latent Replay stores fused latent sequences $z=h(x;\theta_h)$2 and actions $z=h(x;\theta_h)$3, while Incremental Feature Adjustment regularizes the global latent $z=h(x;\theta_h)$4 of the current task to be angularly closer to its own task reference than to similar past-task references: $z=h(x;\theta_h)$5 with $z=h(x;\theta_h)$6. On LIBERO benchmarks, latent replay alone substantially improved AUC, and adding IFA yielded further gains, including $z=h(x;\theta_h)$7 AUC on LIBERO-OBJECT and $z=h(x;\theta_h)$8 on LIBERO-GOAL (Yu et al., 11 Mar 2026).

Across these variants, the replay policy increasingly incorporates structure: depth, drift, class balance, temporal phase, or task similarity. Latent replay has therefore evolved from simple latent buffering into a family of schedulers and selectors for preserving specific regions of representation space.

5. Modalities, systems implications, and empirical operating regimes

Latent replay has been applied across image classification, medical imaging, spiking neuromorphic learning, robotics, pathology, federated learning, and diffusion-based generation. Its attraction in each domain is similar, but the system-level consequences differ substantially.

In foundation-model regimes, the principal gain is compute. The latent replay study with frozen encoders reports that end-to-end raw-data replay consumes almost two orders of magnitude more compute than latent replay for comparable training schedules; after one-time encoding, latent continual learning is nearly flat in compute, and the best latent-replay model on each evaluated stream matched or exceeded the best end-to-end replay model’s final accuracy while remaining far cheaper (Ostapenko et al., 2022). This is a particularly strong result because it reframes continual learning as representation selection plus lightweight latent-space training rather than unavoidable end-to-end optimization.

In edge deployment, the system argument is more explicit. The original latent replay work demonstrated training on a OnePlus 6 smartphone with MobileNetV1, reporting approximately $z=h(x;\theta_h)$9 ms per frame for inference and an overall training session that completed in under one second after acquisition because feature extraction overlapped with video capture (Pellegrini et al., 2019). The key mechanism was storing activations at an intermediate layer such as conv5_4/dw or pool6 and replaying them directly into the upper network, thereby eliminating lower-network computation for replay items.

Spiking systems exploit an even more aggressive storage regime. “Compressed Latent Replays for Lightweight Continual Learning on Spiking Neural Networks” stores spike trains from a chosen SNN layer and compresses them in time by a factor $h$0, achieving up to $h$1 memory reduction relative to naive rehearsal with at most a $h$2 accuracy drop; on SHD it reaches $h$3 Top-1 accuracy in Sample-Incremental learning and $h$4 in Class-Incremental learning (Dequino et al., 2024). SESLR goes further by storing spike tensors at one bit per activation, reducing memory by a factor of $h$5 relative to naive replay on Split CIFAR10-DVS while improving accuracy by approximately $h$6 (Lin et al., 23 Jun 2025).

Federated and multimodal settings highlight another operational advantage: separation of shared and private computation. FedLGR performs replay in a latent feature space of dimension $h$7 produced by a MobileNetV2-based root, keeps the task head and generator local, and aggregates only the shared feature extractor. Reported resource savings reach up to $h$8 CPU and $h$9 GPU usage reduction for FedRoot-based FCL methods, with FedLGR providing the best overall performance across evaluations on social appropriateness prediction (Churamani et al., 2024). In lifelong imitation learning, replaying multimodal latent sequences rather than raw trajectories yields replay buffers around $g$0 MB on LIBERO-OBJECT and $g$1 MB on LIBERO-GOAL under the reported storage policy, while per-inference forward time remains essentially unchanged relative to the base policy (Yu et al., 11 Mar 2026).

In generative modeling, the same latent-space move acts against both forgetting and mode collapse. In text-to-image diffusion, replaying latent codes and prompts rather than images preserved $g$2 Image Alignment on the earliest concept after five tasks, approximately $g$3 higher than baseline methods, while maintaining diversity measured by Vendi Score (Otani, 4 Sep 2025). Here latent replay is not primarily a classifier rehearsal trick; it is a mechanism for rehearsing the denoiser on a compact latent manifold defined by a frozen VAE and text encoder.

6. Limitations, misconceptions, and open research directions

Latent replay is often presented as a remedy for catastrophic forgetting, but the literature repeatedly emphasizes that its success depends on assumptions about latent stability, representation adequacy, and sampling coverage.

The most recurrent limitation is dependence on a frozen or slowly changing encoder. If the encoder poorly separates classes, undergoes domain shift, or drifts substantially after replay states have been stored, replay quality degrades because old latent states no longer align with the current geometry (Nie et al., 28 Jun 2026). This issue appears in simple frozen-compressor methods, where a task-1 encoder may be suboptimal for later tasks (Borde, 2021), and in edge formulations, where “aging” of stored activations must be controlled by freezing lower layers or slowing their learning (Pellegrini et al., 2019). Foundation-model studies similarly show that continual performance is not determined solely by the replay algorithm but by representation geometry, especially class prototype similarity and task subspace overlap (Ostapenko et al., 2022).

A second misconception is that latent replay automatically solves privacy and memory constraints. It does reduce them in many settings—sometimes dramatically—but privacy is not absolute, since latents may still encode recoverable information, and some approaches retain prompts, labels, or identity indices in addition to the latent itself (Otani, 4 Sep 2025). Memory benefits also vary widely. Shallow activation volumes can still be large, while class-conditional decoders or per-class VAEs may grow with the number of classes (Frascaroli et al., 2024). The most storage-efficient variants often achieve this by imposing strong structural assumptions, such as diagonal Gaussian covariances, binary spikes, or deterministic code generation (Nie et al., 28 Jun 2026).

A third point of confusion concerns the term itself. In continual learning, latent replay usually refers to training-time rehearsal of stored or generated representations. In STIR, however, latent replay is the inference-time reuse of latent actions: a controller retrieves a tool $g$4 by similarity to the current hidden state and injects the steering impulse $g$5 when value-modulated gating permits it. This formulation improves average accuracy by $g$6 to $g$7 across arithmetic and logical benchmarks while reducing average token consumption by up to $g$8, but it is a control mechanism rather than a rehearsal buffer (Shi et al., 4 Feb 2026).

Open directions in the literature largely concern richer latent models and better stability-control interfaces. Proposed extensions include adaptive encoder updates constrained by distillation or latent-distribution preservation, richer density models such as latent-space diffusion or normalizing flows, full or low-rank per-prototype covariances, adaptive basis updates for latent actions, representation ensembling, dynamic buffer management, and open-world or cross-domain continual learning built atop self-supervised representations (Nie et al., 28 Jun 2026). A plausible synthesis is that latent replay is increasingly less a single algorithm than a design pattern: learn or choose a latent space whose geometry is sufficiently stable, store enough structure to reconstruct or approximate old decision regions, and replay that structure with a policy matched to the domain’s failure mode—be it forgetting, mode collapse, class imbalance, or hidden-state drift.