Compressed Activation Replay (CAR)

• Compressed Activation Replay (CAR) is a paradigm that stores compressed intermediate activations to improve memory and compute efficiency in neural networks.
• It utilizes diverse methods such as quantization, autoencoders, and pooling to address challenges in continual learning, large-scale training, and reinforcement learning.
• CAR reduces memory burden and prevents representation drift while offering minimal accuracy loss, making it valuable for optimizing deep neural network performance.

Compressed Activation Replay (CAR) is a broad methodological paradigm for improving memory- and compute-efficiency in neural representation learning by storing and replaying compressed representations of intermediate activations, rather than raw inputs or full-precision activations. CAR strategies are deployed across multiple machine learning domains including continual learning, large-scale neural network training, online reinforcement learning, and biologically inspired sequence generation. Core objectives are to minimize memory/storage requirements, stabilize feature-space evolution, maintain statistical efficiency, and, when relevant, accelerate replay. CAR encompasses a diverse set of implementation and compression strategies—from quantization and pooling to learned autoencoders and random projections—tailored to the architectural and task-specific constraints of deep learning systems.

1. Conceptual Foundations and Motivating Problems

Memory replay is an essential tool for addressing catastrophic forgetting in continual learning, managing compute graph storage in large-scale training, and enabling efficient inference or replay in sequential models. Experience Replay (ER), which stores and replays past input-output pairs, is widely used but becomes suboptimal with stringent buffer limits. This is because ER fails to constrain the intermediate latent-space evolution, resulting in “representation drift” even if input-output behaviors are preserved. CAR directly addresses this by storing a compressed version of the feature activation at selected network layers, maintaining explicit control over representation space occupancy and providing better regularization of the latent state (Balaji et al., 2020).

CAR is also a central mechanism in various memory- and compute-constrained scenarios:

• Continual learning pipelines, where feature (and not image) replay enables aggressive buffer compression and better task performance (Weißflog et al., 2024, Wang et al., 2021).
• Large model training, where compressed forward activations (“activation checkpoints”) permit larger batches or models given strict accelerator memory (Chen et al., 2021, Shamshoum et al., 2024, Barley et al., 2024).
• Inference acceleration and resource reduction in autoregressive generative models, including memory-efficient key-value (KV) cache storage in transformers (Roy et al., 7 Dec 2025).
• Sampling and replay acceleration in noisy recurrent networks, notably for biologically inspired replay in path integration tasks (Casco-Rodriguez et al., 20 Feb 2026).

2. Mechanisms and Compression Strategies

CAR is instantiated via a general pipeline:

1. Activation Extract: Compute an intermediate activation $z = f(x)$ for incoming data $x$ at a chosen network cut-point.
2. Compression: Apply a mapping $C(z)$ (parametric or nonparametric), yielding a compressed code $h$. Compression can be:
   • Uniform or non-uniform quantization (e.g., 2 bits—ActNN (Chen et al., 2021); 8–16 levels—FETCH (Weißflog et al., 2024)).
   • Sparsification or thinning (zeroing smallest $k\%$ entries).
   • Learned autoencoder bottlenecks (Wang et al., 2021, Roy et al., 7 Dec 2025).
   • Pooling (average/max), channel/spatial aggregation, or random projection/sketching (Shamshoum et al., 2024, Barley et al., 2024).
3. Storage & Replay: Store $(h, y)$ (plus optional metadata) in memory. During replay or backward pass, decompress $z = D(h)$ for subsequent use (e.g., as network input for learning, or as a proxy for full activations during weight-gradient computation).

Table 1 summarizes representative CAR mechanisms across domains:

Domain	Compression Method	Code Type	Notable Implementations
Continual learning	Quantization, AE, PQ	Int8, PQ, float	FETCH, ACAE-REMIND
Large model training	Stochastic quantization,	2–4b floats	ActNN, CompAct
	low-rank random proj.	Low-rank float
Inference/LLM KV cache	Autoencoder, reuse	Float bottleneck	KV-CAR
Biologically inspired	State momentum/leakage	Full precision	Hippocampal RNN replay

3. Mathematical Formulation and Theoretical Properties

CAR formalizes the tradeoff between storage reduction and information preservation as an encoding-decoding problem. Let $z$ denote the activation, $h = C(z)$ the code, and $z = D(h)$ the decoded activation. Performance is governed by the properties of $x$0, including:

• Quantization variance: For stochastic CAR (e.g., ActNN), unbiasedness is achieved by stochastic rounding. The impact on convergence is given by exact gradient variance decompositions, with overall optimization behavior remaining intact if quantization noise is subordinate to minibatch sampling variance (Chen et al., 2021).
• Reconstruction loss: For autoencoder variants (e.g., ACAE-REMIND, KV-CAR), $x$1 is minimized in tandem with task performance losses (Roy et al., 7 Dec 2025, Wang et al., 2021).
• Projection accuracy: Random projection methods (CompAct (Shamshoum et al., 2024)) leverage Johnson-Lindenstrauss-type results, providing theoretical guarantees that top singular directions are well preserved in expectation.
• Gradient approximation: Pooling-based CAR (e.g., 2×2 average-pooling (Barley et al., 2024)) introduces controlled bias for weight gradients but not for activation gradients; empirical results show negligible degredation when moderate compression is used with sufficient schedule extension.

4. Workflows and Empirical Instantiations

Implementation details vary by task and architecture:

• Continual Learning: Typically, CAR buffers store compressed activations for episodic or online replay. Techniques include uniform quantization (FETCH (Weißflog et al., 2024)), lightweight autoencoders, or product quantization (ACAE-REMIND (Wang et al., 2021)). In these pipelines, only the head classifier is retrained per task, with the encoder often frozen to facilitate inter-task feature transfer.
• Large-Scale Training (CNNs/LLMs): For both vision (ActNN (Chen et al., 2021)) and LLMs (CompAct (Shamshoum et al., 2024), KV-CAR (Roy et al., 7 Dec 2025)), compressed activations replace dense, full-precision context storage during backpropagation or decoding. Compression occurs immediately after forward propagation; the decompressed activation (quantized or projected) is used for gradient computation, with minor accuracy loss and substantial reduction in memory allocation.
• Biological Replay: CAR analogies in recurrent networks incorporate additional dynamical elements: momentum (velocity), leakage, and adaptation terms, collectively implementing underdamped Langevin dynamics. This enables “compressed” replay—accelerated traversal of replayed paths while maintaining exploration (see below) (Casco-Rodriguez et al., 20 Feb 2026).

Pseudocode for a typical forward+CAR step: $x$2

5. Trade-Offs: Memory, Compute, Fidelity, and Performance

The tradeoff surface for CAR is characterized across several axes:

• Memory reduction: Typical compression ratios range from 8× (ActNN, KV-CAR with d = D/8) to >32× (Latent-space replay, ACAE-REMIND with 32B codes vs 150 KB images).
• Accuracy/fidelity impact: Loss in final task performance is negligible (<1–2%) for moderate compression settings, but can become substantial with over-aggressive pooling or excessive quantization (e.g., >4×4 pooling in ResNet leads to >10% accuracy drop (Barley et al., 2024)).
• Training/compute overhead: Compression/decompression incurs minimal additional compute for quantization and pooling; autoencoder and random projection costs are higher but remain subdominant to overall layer computation (Chen et al., 2021, Roy et al., 7 Dec 2025).
• Speed-exploration tradeoff (in replay tasks): In hippocampal replay models, state momentum (underdamped Langevin CAR (Casco-Rodriguez et al., 20 Feb 2026)) accelerates sweep-through but adaptation (negative feedback) recovers exploration. The balance of these terms achieves temporally compressed yet still diverse replay, with empirical reach-time reductions up to 40% without sacrificing path diversity.

6. Empirical Results Across Domains

Key published findings are summarized below:

Method/Domain	Memory Reduction	Accuracy Impact	Special Features
ActNN (CNNs)	12× (2-bit avg.)	<0.5% top-1 loss	Heterogeneous bits
FETCH (CIFAR-10, quant)	>85%	+12% vs raw replay	Simple quant beats AE
ACAE-REMIND (ICL)	>4 orders mag.	+1–2% vs PQ-only	Joint AE/classifier
KV-CAR (LLM KV cache)	47.85%	<2% PPL rise	AE+head-reuse
CompAct (LLM training)	25–30% (pretrain)	≤1.5% PPL, ≤0.3% score	Random projection
Pooling CAR (ResNet)	29% (r=2)	–1.3% top-1 (120 ep)	Exact act-grad flow
Hippocampal RNN CAR	N/A	Maintained fidelity,	Momentum accelerates
		faster/denser replay	replay sampling

Empirical evidence from (Weißflog et al., 2024) corroborates that scalar quantization outperforms learned autoencoders in strict buffer regimes. In buffer-limited online continual learning, CAR regularly yields >5–10% accuracy improvements over raw example replay (Balaji et al., 2020, Wang et al., 2021). For training large DNNs and LLMs, CAR implementation enables 6.6–14× larger batches and substantially relaxes memory constraints (Chen et al., 2021, Shamshoum et al., 2024). In biological and path-integration RNNs, temporally compressed replay with CAR achieves both speed-up and diversity of replayed trajectories (Casco-Rodriguez et al., 20 Feb 2026).

7. Design Choices, Limitations, and Extensions

CAR implementations exhibit domain-specific best practices and open challenges:

• Quantization vs. Autoencoder: Simple uniform quantization is robust and easy to implement for classification-relevant activations; autoencoders offer more expressivity at extra compute/memory and are sensitive to rare class representation (Weißflog et al., 2024, Wang et al., 2021).
• Cut-point selection: In feature replay, aggressive early-layer compression enables more network adaptation but risks discarding discriminative information; mid-to-deep layer cuts are safer but confer less flexibility (Wang et al., 2021).
• Layerwise heterogeneity: Mixed-precision strategies in ActNN and low-rank projection in CompAct reflect significant variance in activation statistics across layers, warranting adaptive compression (Chen et al., 2021, Shamshoum et al., 2024).
• Task frequency: Full retraining of replay heads per task (FETCH, GDumb) is effective but may be computationally prohibitive if tasks arrive frequently (Weißflog et al., 2024).
• Extension potential: Integration with quantization-aware encoder training, sample condensation, or architectural search may further optimize memory-performance Pareto frontiers (Weißflog et al., 2024, Shamshoum et al., 2024).

A plausible implication is that as model and input scales increase further, the scope and necessity of sophisticated CAR schemes, blending both structural (autoencoder, projection) and parametric (learned, adaptive quantization) methods, will broaden. Alignment of compression mechanisms with loss landscape geometry, class balance, and batch statistics will become an even more critical active field of research.

---

References:

• (Balaji et al., 2020): "The Effectiveness of Memory Replay in Large Scale Continual Learning"
• (Chen et al., 2021): "ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training"
• (Wang et al., 2021): "ACAE-REMIND for Online Continual Learning with Compressed Feature Replay"
• (Borde, 2021): "Latent Space based Memory Replay for Continual Learning in Artificial Neural Networks"
• (Weißflog et al., 2024): "FETCH: A Memory-Efficient Replay Approach for Continual Learning in Image Classification"
• (Barley et al., 2024): "Less Memory Means smaller GPUs: Backpropagation with Compressed Activations"
• (Shamshoum et al., 2024): "CompAct: Compressed Activations for Memory-Efficient LLM Training"
• (Roy et al., 7 Dec 2025): "KV-CAR: KV Cache Compression using Autoencoders and KV Reuse in LLMs"
• (Casco-Rodriguez et al., 20 Feb 2026): "Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay"