CRAM: Continual Amodal Memory for Video Learning

• The paper presents CRAM, a framework that replaces raw video storage with compressed neural codes to overcome memory limitations in long-video continual learning.
• CRAM employs an evolving autoencoder-based compressor with a refreshing mechanism to maintain compatibility of stored codes amid representation drift.
• Empirical results on Kinetics-700 and EpicKitchens-100 demonstrate competitive accuracy and reduced forgetting while drastically reducing memory usage.

Searching arXiv for the CRAM paper and key related work mentioned in the provided data. Continually Refreshed Amodal Memory (CRAM) is a rehearsal-based framework for large-scale video continual learning in which a system stores compressed neural codes rather than raw video, trains a neural video compressor and a classifier over a non-IID stream, and refreshes stored codes so that past samples remain compatible with the current compressor (Mall et al., 7 Aug 2025). The method is formulated for video continual learning under tight memory constraints, especially when long videos, dense temporal coverage, and continual streams make raw-frame rehearsal infeasible. In the reported large-scale setting, CRAM is evaluated on Kinetics-700 and EpicKitchens-100, storing thousands of relatively long videos in under 2 GB while retaining competitive accuracy and low forgetting.

1. Problem setting and motivation

Continual learning studies sequential learning from a non-stationary stream rather than from an IID dataset available all at once. In the task-incremental formulation used for CRAM, the model receives tasks $t_1, t_2, \dots, t_n$, each with its own data distribution, and learns them in sequence: $$D = \{x_{j,i}, y_{j,i}\}_{j=1..m,\, i=1..n},\quad x_{j,i} \sim X_{t_i},\; y_{j,i} \sim Y_{t_i}.$$ The central difficulty is catastrophic forgetting: after training on later tasks, performance on earlier tasks collapses (Mall et al., 7 Aug 2025).

Video continual learning intensifies this problem because videos add a temporal dimension and therefore greatly increase raw data volume. In realistic long egocentric videos such as EpicKitchens-100, clips can be minutes to hours long, and continual streams can span many hours of footage. If the objective is to model realistic distribution shifts over time, dense temporal coverage is preferred over sparse frame sampling. Standard rehearsal buffers that store raw RGB frames therefore scale poorly: a single clip of shape $224\times224\times3\times32$ is already measured in megabytes, and performance-preserving rehearsal would require many clips from many past tasks, pushing storage toward hundreds of GB or more.

CRAM is motivated by the observation that rehearsal remains the most effective class of continual learning methods, but conventional raw-input rehearsal is poorly matched to long-video regimes. Earlier video continual learning approaches often restrict themselves to short videos, heavily subsample in time, or rely on pretrained large models. CRAM addresses precisely the scalability bottleneck created by video storage.

2. Representation, architecture, and amodal memory

CRAM replaces raw-video storage with compressed vision. Its core components are a video compressor $c=(\phi,\psi)$, a bounded memory buffer $B$, and a classifier $q$ (Mall et al., 7 Aug 2025). The encoder $\phi$ maps a video clip $x$ to a latent code $e$, and the decoder $\psi$ reconstructs an approximation $$D = \{x_{j,i}, y_{j,i}\}_{j=1..m,\, i=1..n},\quad x_{j,i} \sim X_{t_i},\; y_{j,i} \sim Y_{t_i}.$$0: $$D = \{x_{j,i}, y_{j,i}\}_{j=1..m,\, i=1..n},\quad x_{j,i} \sim X_{t_i},\; y_{j,i} \sim Y_{t_i}.$$1 The buffer stores tuples $$D = \{x_{j,i}, y_{j,i}\}_{j=1..m,\, i=1..n},\quad x_{j,i} \sim X_{t_i},\; y_{j,i} \sim Y_{t_i}.$$2, and the classifier is trained directly on codes rather than on decoded RGB. “Amodal” denotes that the memory is not stored in the original sensory modality, namely pixels, but in a different code space.

The compressor is based on the ResNet-style compression model from “Compressed Vision,” implemented as a VQ-VAE-style autoencoder. The encoder has the form

$$D = \{x_{j,i}, y_{j,i}\}_{j=1..m,\, i=1..n},\quad x_{j,i} \sim X_{t_i},\; y_{j,i} \sim Y_{t_i}.$$3

with $$D = \{x_{j,i}, y_{j,i}\}_{j=1..m,\, i=1..n},\quad x_{j,i} \sim X_{t_i},\; y_{j,i} \sim Y_{t_i}.$$4 frames per clip, and the decoder maps back to RGB space. For classification, CRAM uses an S3D backbone adapted to compressed-code input on Kinetics-700 and a short-term S3D variant on EpicKitchens-100. A key architectural fact is that CRAM does not re-decode to RGB for classification; inference and training of the classifier operate purely in latent space.

In the ideal IID setting, the compressor minimizes reconstruction error,

$$D = \{x_{j,i}, y_{j,i}\}_{j=1..m,\, i=1..n},\quad x_{j,i} \sim X_{t_i},\; y_{j,i} \sim Y_{t_i}.$$5

while the classifier minimizes cross-entropy over encoded inputs,

$$D = \{x_{j,i}, y_{j,i}\}_{j=1..m,\, i=1..n},\quad x_{j,i} \sim X_{t_i},\; y_{j,i} \sim Y_{t_i}.$$6

In practice, the continual setting departs from IID because the compressor itself evolves over time.

CRAM maintains a rolling buffer

$$D = \{x_{j,i}, y_{j,i}\}_{j=1..m,\, i=1..n},\quad x_{j,i} \sim X_{t_i},\; y_{j,i} \sim Y_{t_i}.$$7

where codes are produced by the encoder current at the time of storage. During task $$D = \{x_{j,i}, y_{j,i}\}_{j=1..m,\, i=1..n},\quad x_{j,i} \sim X_{t_i},\; y_{j,i} \sim Y_{t_i}.$$8, new videos are encoded by the current encoder $$D = \{x_{j,i}, y_{j,i}\}_{j=1..m,\, i=1..n},\quad x_{j,i} \sim X_{t_i},\; y_{j,i} \sim Y_{t_i}.$$9, added to the buffer, and mixed with uniformly sampled past codes to approximate IID training over all seen data.

3. Refreshing mechanism and representation drift

The defining mechanism of CRAM is continual refreshing of the amodal memory. Because the compressor is trained online from scratch in the incremental setting, it is itself susceptible to catastrophic forgetting (Mall et al., 7 Aug 2025). If the compressor is updated only on current-task data, older codes in the buffer become stale: they were produced by older encoders $224\times224\times3\times32$0, whereas the current decoder $224\times224\times3\times32$1 may have drifted and no longer reconstruct them consistently. This representation drift degrades both reconstruction quality and downstream classification.

Two naive alternatives are explicitly rejected. One is to retain all historical decoders $224\times224\times3\times32$2 and decode each code with the decoder from the time of its creation; this avoids drift but causes model memory to grow linearly with the number of tasks. The other is to freeze the compressor after pretraining; this is incompatible with incremental-from-scratch training.

CRAM instead keeps only a single snapshot from the immediately previous task. For task $224\times224\times3\times32$3, the compressor is trained only on $224\times224\times3\times32$4: $224\times224\times3\times32$5 The buffer $224\times224\times3\times32$6 is then filled with codes $224\times224\times3\times32$7 and labels $224\times224\times3\times32$8.

At task $224\times224\times3\times32$9, CRAM first decodes old codes using the previous decoder,

$c=(\phi,\psi)$0

then trains the new compressor $c=(\phi,\psi)$1 jointly on current-task clips $c=(\phi,\psi)$2 and the decoded old clips $c=(\phi,\psi)$3: $c=(\phi,\psi)$4 After training, each old code is refreshed by re-encoding the decoded clip,

$c=(\phi,\psi)$5

and the buffer is updated so that all stored codes now inhabit the current latent space.

The same construction generalizes inductively to task $c=(\phi,\psi)$6: keep only $c=(\phi,\psi)$7, decode current buffer codes with $c=(\phi,\psi)$8, retrain $c=(\phi,\psi)$9 on both new and decoded old data, and then re-encode the whole buffer with $B$0. The central consequence is that model memory remains constant while the buffer stays fresh. Conceptually, the method treats the compressor as an evolving feature extractor whose state is regularized by replay via decoded buffer codes. The source describes this as a kind of top-down/bottom-up consistency inspired by hippocampal indexing theory and dreams; this suggests a stabilization view in which earlier experiences are periodically projected into the latest latent space.

For classifier training at task 2, CRAM mixes new examples and refreshed replay: $B$1 After full refresh, $B$2 corresponds to the new codes for old data.

4. Training regimes, rehearsal policy, and metrics

CRAM is evaluated in two regimes: continual learning with pretraining and incremental learning from scratch (Mall et al., 7 Aug 2025). In both cases, data are organized into tasks that are assumed IID within each task but subject to distribution shift across tasks.

In the pretraining regime, phase 1 performs IID training of the compressor and classifier on half of the dataset’s classes. The compressor is then frozen, and all compressed codes for the pretraining data are stored in the buffer. Phase 2 introduces the remaining classes incrementally: for Kinetics-700, 10 tasks of 30 classes each; for EpicKitchens-100, 17 tasks of one participant each. New videos are encoded with the frozen encoder, appended to the buffer within the memory budget, and only the classifier is trained on mixed minibatches of new-task codes and buffered codes. Because $B$3 and $B$4 are frozen after pretraining, no refreshing is needed in this setting.

In the incremental-from-scratch regime, there is no initial large-scale pretraining. Learning starts at task $B$5. The task organization is 35 tasks of 20 non-overlapping classes each for Kinetics-700 and 33 tasks of one participant’s videos each for EpicKitchens-100. For each task, the compressor is trained for 1 epoch on current-task clips plus decompressed clips from the buffer, and the classifier is trained for 30 epochs on minibatches mixing new codes and replay codes. CRAM is task-aware in experiments for evaluation and scheduling, but its mechanisms do not require explicit task labels at inference time; the classifier is a single head over all classes seen so far.

The rehearsal policy uses a fixed-capacity memory buffer $B$6, measured in samples determined by the MB budget and code size. At task $B$7, storage per past task is $B$8 in the incremental-from-scratch regime and $B$9 in the pretraining-plus-incremental regime when pretraining is counted as task 1. Within each task, videos are sampled IID; rehearsal examples are sampled randomly from the buffer.

The key evaluation metrics are Train, Eval, and Average Forgetting (AvgF). Train is the average training accuracy across tasks, measured at the end of each task and averaged. Eval is the average evaluation accuracy on held-out test sets across tasks. Average Forgetting measures performance degradation on previous tasks. If $q$0 is accuracy on task $q$1 after training up to task $q$2, then forgetting for task $q$3 at time $q$4 is

$q$5

and average forgetting over tasks $q$6 is

$q$7

Smaller $q$8 is better.

5. Compression characteristics, memory profile, and quantitative results

CRAM divides each video into clips of shape $q$9. Each clip is compressed into a code of size approximately 0.0013 MB, with a typical compression rate of $\phi$0; an ablation with $\phi$1 compression is also reported (Mall et al., 7 Aug 2025). Long videos are therefore represented by many codes rather than by aggressive temporal thinning: Kinetics-700 has roughly 250 frames per video, corresponding to about 8 codes per video, while EpicKitchens-100 has roughly 27k frames per video, corresponding to about 850 codes per video. Because the codes are small, CRAM can store codes for every frame without temporal subsampling while staying within a few GB.

Under the fixed total memory budget, including both model weights and the buffer, CRAM at $\phi$2 compression uses 654 MB for the buffer and 750 MB for the models on Kinetics-700, for a total of 1404 MB. On EpicKitchens-100 it uses 714 MB for the buffer and 750 MB for the models, for a total of 1464 MB. The equivalent raw-RGB storage for the same amount of data is approximately $\phi$3 MB on Kinetics-700 and approximately $\phi$4 MB on EpicKitchens-100. With the fixed budget, CRAM stores approximately $\phi$5 frames on Kinetics-700 and approximately $\phi$6 frames on EpicKitchens-100. The comparison figures reported for other methods are $\phi$7 and $\phi$8 frames for REMIND, and approximately $\phi$9 and approximately $x$0 frames for RGB-buffer baselines such as GDumb, SMILE, and vCLIMB.

The datasets are large-scale. Kinetics-700 contains approximately 14M training frames and approximately 3M evaluation frames, with average video length 10 seconds and maximum length approximately 15 seconds. EpicKitchens-100 contains approximately 16M training frames and approximately 4M evaluation frames, with average video length approximately 20 minutes and longest video approximately 1.5 hours. The EpicKitchens-100 experiments use noun classification over 331 noun classes; Kinetics-700 uses action classification. Data are organized class-incrementally following vCLIMB guidelines.

In the pretraining regime, the unbounded RGB upper bound reaches Train $x$1 and Eval $x$2 on Kinetics-700, and Train $x$3 and Eval $x$4 on EpicKitchens-100. CRAM reaches Train $x$5, Eval $x$6, and AvgF $x$7 on Kinetics-700, and Train $x$8, Eval $x$9, and AvgF $e$0 on EpicKitchens-100. REMIND records Train 45.8, Eval 38.20, and AvgF 43.10 on Kinetics-700, and Train 32.3, Eval 26.50, and AvgF 51.0 on EpicKitchens-100. Under a roughly 1.4 GB budget, CRAM is therefore within approximately 1.7 to 2.7 percentage points of the unbounded upper bound in Eval while substantially reducing forgetting relative to REMIND.

In the incremental-from-scratch regime, the unbounded RGB upper bound reaches Train $e$1 and Eval $e$2 on Kinetics-700, and Train $e$3 and Eval $e$4 on EpicKitchens-100. CRAM reaches Train $e$5, Eval $e$6, and AvgF $e$7 on Kinetics-700, and Train $e$8, Eval $e$9, and AvgF $\psi$0 on EpicKitchens-100. On the same datasets, SMILE reports Train 40.56, Eval 29.20, and AvgF 62.50 on Kinetics-700, and Train 28.71, Eval 19.20, and AvgF 67.8 on EpicKitchens-100; vCLIMB reports Train 39.12, Eval 28.65, and AvgF 65.10 on Kinetics-700, and Train 27.11, Eval 18.5, and AvgF 66.5 on EpicKitchens-100; GDumb reports Train 37.61, Eval 18.70, and AvgF 52.40 on Kinetics-700, and Train 25.30, Eval 15.60, and AvgF 60.10 on EpicKitchens-100.

The reported performance-versus-memory trade-off on Kinetics-700 shows CRAM at an approximately 0.65 GB buffer and approximately 1.4 GB total memory achieving strong accuracy, whereas baselines such as SMILE and REMIND require much larger memory, in the 2–20 GB range, to approach similar accuracies. Under the same small budget, the baselines perform markedly worse. Ablations indicate that $\psi$1 compression causes only a slight performance drop relative to $\psi$2, that 40 rather than 30 classifier epochs per task yields small gains, and that varying class splits does not change the overall conclusions. As total hours of video increase into the tens of hours, performance remains stable; the authors estimate that under a 20 GB budget, as used by REMIND for good performance, CRAM could process up to approximately 5,470 hours of video before buffer saturation becomes an issue.

6. Relation to other approaches, limitations, and scope

CRAM differs from standard rehearsal on raw inputs by storing compressed codes instead of RGB frames (Mall et al., 7 Aug 2025). In long-video continual learning, raw rehearsal rapidly exhausts memory, which has led methods such as SMILE and vCLIMB to store only a single or a few frames per video. CRAM instead uses up to $\psi$3 compression and can store codes for all frames, thereby avoiding frame selection or temporal condensation. This does not imply that CRAM is a generative replay method: the compressor is a reconstruction model used to compress and decompress real data, not to synthesize novel samples.

CRAM also differs from feature-buffer rehearsal such as REMIND. The comparison given in the source is that REMIND stores quantized intermediate features for images, uses lower compression ratios than CRAM, does not refresh low-level representations, and was not designed for long videos. Relative to offline compressed-vision approaches, CRAM generalizes the use of compressed codes from a pretraining-then-freeze setting to an online continual setting in which the compressor itself is updated and therefore requires refreshing. Relative to prompt-based or rehearsal-free continual learning methods such as L2P, ST-Prompt, and DPAT, CRAM shifts less of the memory burden into a large foundation model and, in the incremental-from-scratch regime, does not require external pretraining.

Several limitations are explicitly identified. Refreshing incurs non-zero computational overhead because many codes must be decoded and re-encoded. Performance depends on the quality of the autoencoder; if the compressor underfits, the latent codes may be suboptimal for downstream classification. The experiments use explicit task boundaries and controlled class splits, so fully task-free non-segmented streams are not evaluated. The demonstrated applications are action and noun classification rather than segmentation, tracking, or reinforcement learning, and strict real-time latency constraints are not analyzed. These limitations define the current scope rather than a contradiction in the method’s premise.

The stated future directions are scaling to even longer-duration videos and larger continual horizons, extending the framework to other video tasks such as action segmentation and object detection, and developing more sophisticated compression schemes or architectures to improve the accuracy-per-bit trade-off. A broader implication is that compressed vision may make it possible to port image-based continual learning techniques to video while preserving dense temporal coverage.