Efficient Continual Training Methods

• Efficient Continual Training Methodology encompasses strategies to update models incrementally while reducing compute, memory, and energy usage.
• Techniques such as replay compression, modular adapters, and meta-learning achieve concrete benefits including up to 4.88× speed-ups and significant memory savings.
• Formal metrics like computational, memory, and storage overhead ratios benchmark these methods across diverse domains from neuromorphic systems to large-scale transformers.

Efficient continual training methodology encompasses algorithmic and system-level strategies for updating machine learning models on streams of evolving data while minimizing compute, memory, and energy overheads, all without sacrificing predictive performance or exacerbating catastrophic forgetting. Methods span workloads from spiking neuromorphic systems to deep neural networks and large-scale transformers. Approaches include replay-based compression, task-adaptive parameter modules, low-overhead meta-learning, efficient pretraining procedures, and algorithmic exploitation of activation and memory footprints. This article surveys the principles, formal metrics, technical advances, and state-of-the-art empirical outcomes reported in leading research on the topic.

1. Formal Efficiency Metrics and Evaluation Criteria

Efficient continual training methods are evaluated along axes of computational, memory, and storage efficiency, always relative to batch retraining or non-incremental baselines. Harun et al. formalized the following key metrics (Harun et al., 2023):

• Computational Overhead Ratio: $\gamma_C = C_{\textrm{CL}} / C_{\textrm{scratch}}$, where $C_{\textrm{CL}}$ is total backpropagation updates in continual mode; $\gamma_C<1$ signals compute gains over offline retraining.
• Memory Overhead Ratio: $\gamma_M = M_{\textrm{CL}} / M_{\textrm{scratch}}$, ratio of memory consumption (model, buffer, and working memory).
• Storage Overhead Ratio: $\gamma_S = S_{\textrm{CL}} / S_{\textrm{scratch}}$, reflecting persistent disk/storage requirements; not always reported.
• NetScore (Composite Metric):

$$\Omega(G) = s \cdot \log_{10} \left[ \frac{a(G)^\alpha}{p(G)^\beta \cdot u(G)^\gamma \cdot m(G)^\zeta} \right]$$

where $a$ = accuracy, $p$ = parameters (M), $u$ = updates (M), $m$ = memory (GB), with standard exponents for normalization.

Reporting these metrics enables researchers to benchmark efficiency across methods and application regimes, establishing whether proposed continual learning (CL) strategies deliver practical resource savings or impose hidden costs (Harun et al., 2023).

2. Algorithmic Advances in Efficient Continual Training

2.1. Replay Compression and Activation-based Methods

Replay4NCL compresses latent activations (post-layer spike traces) from past tasks, downsampling temporal resolution and replaying at reduced timestep granularity. This yields direct memory and latency reductions while preserving SNN task performance. Adaptive scaling of neuron thresholds and learning rates compensates for coarser spike trains (Minhas et al., 21 Mar 2025). Concrete improvements include $4.88\times$ latency speed-up, $20\%$ replay memory size reduction, $36.43\%$ energy saving, and $+4.21$pp old-task accuracy gain over the prior state-of-the-art on class-incremental SHD.

Alchemist targets LLMs. It eliminates redundant recomputation by reusing activations and cached key-value (KV) tensors recorded during user-facing serving passes (“prefill”) for subsequent training updates. A system of offload maps and memory hedging between CPU and GPU, pipelined with training, yields up to $1.72\times$ throughput and $47\%$ training memory savings (Huang et al., 3 Mar 2025).

TiC-CLIP proposes a simple warm-start plus cumulative rehearsal strategy for web-scale vision-LLMs: at each step, the model checkpoint from the previous period is fine-tuned on a mixture of newly arrived and replayed past samples, rather than retrained from scratch on the pooled dataset. This approach attains Oracle-equivalent downstream performance with $2.5$–$4\times$ less compute on the multi-billion scale TiC benchmarks (Garg et al., 2023).

2.2. Efficient Adapter-based and Modular Parameterization

Transformer-based continual learning leverages Adapters: lightweight, task-specific modules inserted into frozen pre-trained backbones (Ermis et al., 2022). The Adaptive Distillation of Adapters (ADA) framework maintains a fixed-size pool of $K$ adapters (constant in the number of tasks), merging them via data-driven distillation when new tasks arrive. ADA matches full adapter fusion in accuracy but uses $\sim45\%$ fewer parameters and maintains $O(1)$ parameter and inference cost with respect to the number of tasks, achieving $0.90$ accuracy on Arxiv with $478$ MB parameter memory and $6.3$ ms inference latency (versus $872$ MB and $30.5$ ms for AdapterFusion).

2.3. Meta-Learning and Online Regularization

Taylor expansion-based meta-learning attains first-order approximations of parameter importance, enabling efficient regularization and closed-form meta-gradient computation without second-order derivatives (Zou et al., 2022). Proximal Gradient Descent further streamlines the solver. Compared with La-MAML, this approach (EMCL) reduces meta-update time by $\sim30$–$40\%$, matches accuracy, and avoids memory overhead.

Complementary to this, PackNet-inspired iterative pruning with time or wall-clock budgets exploits analogies to human sleep cycles, showing that tuning the prune–retrain (“day–night”) cycles per task under a fixed budget achieves an optimal balance of memory and computational efficiency, particularly in low-capacity and on-device environments (Ball et al., 2020).

3. Memory-Efficient and Compressed Replay Strategies

Latent buffer compression, generative recollection, and compressed activation replay can dramatically reduce memory/storage overhead.

• REMIND and SIESTA: SIESTA replaces conventional replay with product-quantized mid-layer features, storing only compressed codes, and separates lightweight online running-mean class updates (“wake” phase) from backprop-constrained rehearsal (“sleep” phase). On ImageNet-1K, SIESTA matches offline top-5 accuracy (83.6–87.0%), executes $4\times$ faster than REMIND, and requires only $2$ GB buffer memory (versus $20+$ GB for batch learners) (Harun et al., 2023).
• Scalable Recollections (SRM): Employs a discrete latent VAE that compresses each experience to a binary latent code, maintaining a buffer of these codes. On streaming benchmarks, SRM + replay matches or exceeds classical memory-replay schemes under tight memory budgets and sublinear buffer growth (Riemer et al., 2017).
• Feature Buffering (Candidates Voting): Online, class-incremental learning via storing small sets of task-specific feature embeddings (not raw images) achieves state-of-the-art accuracy under tight buffer constraints, with memory cost $O(DQ)$ (feature dim $\times$ exemplar count) rather than $O(3S^2 Q)$ (raw RGB pixels) (He et al., 2021).

4. Data and Parameter-Efficient Continual Pretraining

Efficient continual pretraining extends to language domains and foundation models:

• ICL-Augmented Pretraining (ICL-APT): Augments each in-domain target sample with top-$k$ nearest neighbors from both in-domain and broad domain corpora, concatenated for masked language modeling. This achieves $3.6\times$ reduction in GPU time over standard DAPT, while improving IR metrics by 18.6% relative (+3.42 absolute points in mean IR) on German shift-book logs (Zhukova et al., 28 Apr 2025).
• Continual Pretraining with Adapters and Preference Optimization (PureTC-1B): A three-stage LoRA-based pipeline (CPT → SFT → DPO) modularly specializes a 1B-parameter LLaMA model for monolingual Traditional Chinese adherence. The full training (pretraining, SFT, preference tuning) fits into 17 GPU-hours on a single 12 GB consumer card and reduces non-target token generation by 51.3% (micro-OLR), achieving stability improvements up to 77.2% in named-entity translation over much larger open models (Chih et al., 2 Oct 2025).
• Arabic Dialect Adaptation: Truncating BERT to 4 layers (TinyBERT), continual MLM on as few as 100K dialectal sentences improves robustness to dialectal shift by +4.15 average points on ALUE benchmarks. All updates are full-model; no adapters are required, and training time is reduced by 40% when warm-starting from mBERT (Sarkar et al., 2022).

5. Empirical Results and Application Domains

Empirical evaluations consistently demonstrate:

• Efficiency gains can reach $2$–$5\times$ reduction in compute versus retraining from scratch while matching or exceeding the baseline performance in vision, language, and spiking neuromorphic domains (Verwimp et al., 28 Feb 2025, Minhas et al., 21 Mar 2025, Harun et al., 2023, Zhukova et al., 28 Apr 2025, Garg et al., 2023).
• Memory-optimized methods such as REMIND and SIESTA achieve compute overhead $\gamma_C=0.51$ (REMIND) and peak buffer sizes $<2$ GB on ImageNet, with accuracy close to batch retraining (Harun et al., 2023).
• Adapter-based frameworks (ADA) maintain constant inference cost and parameter footprint with respect to the number of tasks, a crucial property for scalable, multitask continual learning (Ermis et al., 2022).
• System-level innovations (Alchemist) yield throughput and memory improvements on real LLM workloads, supporting longer input sequences (up to $7$k tokens) and reducing serving latency to $<3\%$ above baseline (Huang et al., 3 Mar 2025).

Method	Compute Saving	Additional Gains	Notable Benchmarks
Replay4NCL	$4.88\times$	$+4.21$pp old-task accuracy	SHD (SNNs, embedded)
ADA (adapter pool)	$O(1)$ param	$\sim45\%$ param saving	Arxiv, Reuters, Vision
SIESTA	$4\times$	$\approx$Offline accuracy	ImageNet-1K, MobileNetV3-L
ICL-APT	$3.6\times$	$+3.42$ mean IR points	German process industry
TiC-CLIP (Cumulative)	$2.5$–$4\times$	Oracle-matching accuracy	TiC-DataComp (12.7B pairs)
PureTC-1B (LoRA-based)	$>$order-of-magnitude	$-51.3\%$ OLR, +20pp Pass@TC	TC LLaMA-1B, NER translation

6. Design Guidelines and Practical Recommendations

Emerging best practices for efficient continual training include:

• Compress and sample: Downsample activation or feature maps for replay, use kNN or PQ buffers, and prefer compressed representations where possible (Minhas et al., 21 Mar 2025, Harun et al., 2023).
• Adapterization with pooling: Instead of growing separate modules for each task, use a capped pool with knowledge distillation-based merging (Ermis et al., 2022).
• Replay at the right level: For SNNs or transformer models, replay compressed activations instead of raw sequences to optimize both bandwidth and energy (Minhas et al., 21 Mar 2025, Huang et al., 3 Mar 2025).
• Leverage meta-learning with efficient importance estimation: First-order importance weights and closed-form regularization are sufficient to prevent forgetting without prohibitive Hessian computation (Zou et al., 2022).
• Optimize hyperparameters for plasticity and speed: “Shrink & perturb” initialization, L2 regularization toward initial weights, aggressive LR annealing, and data selection targeting “medium difficulty” replay examples together yield 2–5$\times$ speed-ups across image domains (Verwimp et al., 28 Feb 2025).

7. Limitations, Open Issues, and Future Directions

Despite significant advances, current efficient continual training methods may be limited by:

• The need for storing or replaying compressed activations, which, while dramatically reducing memory, still require non-zero buffer overhead.
• Model rigidity in methods relying on frozen backbones or infrequently updated feature extractors; flexibility for dynamic representation learning in fully nonstationary settings remains a challenge (He et al., 2021, Sarkar et al., 2022).
• Difficulty scaling certain replay or buffer schemes to unbounded task horizons or truly streaming, non-iid sequence regimes.
• Additional research is needed to extend efficient continual learning to foundation models in vision and multimodal domains, under resource-constrained or privacy-preserving constraints.

Continued convergence of algorithmic, architectural, and systems-level innovation is leading toward continual learners that are not only more robust to forgetting, but also demonstrably efficient by formal resource metrics. This progress paves the way for real-world deployment of adaptive, low-cost AI across embedded, cloud, and web-scale applications (Harun et al., 2023, Minhas et al., 21 Mar 2025, Huang et al., 3 Mar 2025).