Continual Pretraining (CPT) Overview

• Continual Pretraining (CPT) is a paradigm where pretrained foundation models are continually updated via self-supervised objectives on new data, integrating domain-specific knowledge while preventing catastrophic forgetting.
• By interleaving unsupervised adaptation with downstream specialization, CPT leverages techniques like replay buffers and distillation to efficiently balance retaining general capabilities with learning new tasks.
• Empirical advances show that CPT improves domain adaptation, language expansion, and scaling efficiency, reducing retraining compute costs by up to 60% compared to full model retraining.

Continual Pretraining (CPT) is a paradigm wherein the parameters of a pretrained foundation model—across vision, language, and multimodal domains—are continually updated via self-supervised (or composite) objectives on new data distributions, domains, or modalities. In contrast to frozen-feature transfer or isolated fine-tuning, CPT interleaves ongoing unsupervised or weakly supervised adaptation before or alongside downstream specialization. This approach aims to integrate new skills or knowledge while mitigating catastrophic forgetting, enabling efficient domain adaptation, language expansion, and multi-task model evolution at a fraction of the cost and environmental impact associated with full retraining.

1. Formal Definition and Core Objectives

CPT reuses parameters $\theta_0$ from a pretrained model—such as a LLM, vision transformer, or speech model—and continues optimizing a self-supervised loss over a new corpus $D_{\text{CPT}}$. The canonical CPT loss in language is

$$L_{\text{CPT}}(\theta) = -\mathbb{E}_{x \sim D_{\text{CPT}}}\, \sum_{t=1}^{|x|}\log P_\theta(x_t|x_{<t})$$

for next-token prediction, or its analogs for masked-token or masked-image objectives. In foundation models, CPT is typically applied to incrementally encode new distributions (e.g., domain-specific text, satellite images, classroom audio, underrepresented languages) while seeking to preserve general capabilities from the original pretraining (Mendieta et al., 2023, Wang et al., 5 Oct 2024, Wu et al., 2 Feb 2024, Elhady et al., 30 May 2025, Zheng et al., 2 Jul 2024, Shi et al., 24 Feb 2025, Attia et al., 15 May 2024).

CPT is the first and central stage of the modern continual learning framework for large models, typically preceding (i) continual instruction tuning (supervised adaptation) and (ii) continual alignment (preference/behavioral reinforcement), as formalized in (Wu et al., 2 Feb 2024).

2. Canonical Methodologies and Loss Structures

CPT encompasses a spectrum of formulations, often combining multiple objectives:

• Pure self-supervised continuation: Resume the original masked modeling or autoregressive objective on domain/target data (e.g., masked language modeling for LLMs, contrastive/quantization objectives for SSL speech models).
• Multi-objective CPT: Incorporate auxiliary loss terms for distillation or feature matching, as in geospatial foundation modeling, where a feature-distillation loss (matching partial-context student to teacher features) is combined with masked image modeling (Mendieta et al., 2023).
• Hybrid architectures: For cross-modal adaptation (e.g., text-to-speech LLMs), CPT may interleave modalities (joint next-token objectives over both original tokens and new representations, such as codec-based audio tokens), with explicit data-mix or multi-task sampling (Shi et al., 24 Feb 2025).
• Regularization and anti-forgetting: To prevent the erosion of general capacities, CPT methods may use one or more of: (a) replay buffers (generative or sampled) from older distributions; (b) distillation-based regularizers enforcing similarity of logits or intermediate activations to pre-CPT models; (c) parameter-isolation or masking (adapters/LoRA/layer-freeze) (Wu et al., 2 Feb 2024, Jiang et al., 15 Jul 2024, Attia et al., 15 May 2024, Nag et al., 13 Dec 2024).

The general joint CPT objective can be written as

$$L_{\text{CPT-total}}(\theta) = L_{\text{new}}(\theta; D_{\text{CPT}}) + \lambda\cdot R(\theta, \theta_\text{old})$$

where $L_{\text{new}}$ is the task objective on new data, $R$ is a replay or regularization term (Kullback-Leibler, quadratic, or feature-based), and $\lambda$ is a tradeoff coefficient (Wu et al., 2 Feb 2024, Jiang et al., 15 Jul 2024).

3. Empirical Advances: Efficiency, Transfer, and Domain Adaptation

CPT provides substantial gains in efficiency and adaptability relative to full retraining:

• Domain adaptation: CPT consistently boosts performance on challenging domains (e.g., geospatial imagery, classroom speech) with minimal compute, by leveraging both robust pretraining features and domain-specific adaptation (Mendieta et al., 2023, Attia et al., 15 May 2024, Attia et al., 13 Sep 2024).
• Language expansion: CPT is widely adopted for low-resource and cross-lingual adaptation of LLMs, delivering rapid convergence and strong transfer by initializing from high-resource languages and mixing replay to prevent catastrophic forgetting (Zheng et al., 2 Jul 2024, Wu et al., 2 Feb 2024, Elhady et al., 30 May 2025, Nag et al., 13 Dec 2024, Vo et al., 21 Aug 2024).
• Cost-reduction and scaling laws: Analytical scaling laws for CPT curves enable precise prediction and optimization of learning hyperparameters (e.g., mixture ratios, learning rates, batch schedules), resulting in compute-optimal strategies for model size, data size, and replay mixing (Que et al., 3 Jun 2024, Wang et al., 12 May 2025, Wang et al., 5 Oct 2024). Empirically, CPT can decrease compute cost by 40–60% with negligible downstream loss compared to full pretraining (Wang et al., 5 Oct 2024, Zheng et al., 2 Jul 2024).
• Self-supervised superiority: Experimental comparisons demonstrate that self-supervised CPT retains general representations and resists forgetting far better than supervised or task-specific continual adaptations (Cossu et al., 2022, Sun et al., 2023, Wu et al., 2 Feb 2024).

4. Advanced Procedures and Recent Innovations

Recent literature explores sophisticated CPT extensions:

• Multi-objective and staged CPT: The geospatial GFM paradigm employs feature-distillation alongside domain adaptation; Mix-CPT in LLMs uses logit-swap self-distillation and mixed-data objectives to couple knowledge acquisition with utilization while decoupling format alignment (Mendieta et al., 2023, Jiang et al., 15 Jul 2024).
• Curricula and data selection: PPL-based curricula, synthetic example generation (QA, code, chain-of-thought), and corpus ranking/fragmentation analyses allow for targeted, efficient CPT especially for low-resource languages and scientific/technical domains (Chen et al., 26 Jul 2024, Nag et al., 13 Dec 2024, Ishibashi et al., 15 May 2025).
• Path-switching and version-updating: Hierarchical CPT schedules, where a mainline is trained with high LR to produce a robust “flat” initialization and per-update branches decay toward convergence, optimize both performance and training time in sequential LLM versioning (Wang et al., 5 Oct 2024).
• Scaling law control: D-CPT Law and its cross-domain generalization provide closed-form predictions of general and domain loss as functions of model size, data size, and mixture ratio, supporting rapid hyperparameter selection across new domains with minimal pilot data (Que et al., 3 Jun 2024).

5. Mitigating Catastrophic Forgetting

CPT fundamentally seeks to mitigate catastrophic forgetting, the degradation of previously acquired general capabilities when models adapt to new data:

• Replay strategies: Interleaving a fraction ($r$) of source-domain data (typically 5–30%) during CPT robustly prevents forgetting in source tasks without slowing convergence on target domains (Zheng et al., 2 Jul 2024, Elhady et al., 30 May 2025, Wu et al., 2 Feb 2024, Attia et al., 15 May 2024).
• Distillation and regularization: Stage-wise or per-batch distillation (KL-divergence or MSE) to the pre-CPT model on source data, logit/probability matching (logit-swap self-distillation), or maintaining exponential moving averages constrains parameter drift and preserves emergent abilities (notably in zero-shot or in-context learning) (Jiang et al., 15 Jul 2024, Elhady et al., 30 May 2025).
• Parameter-isolation schemes: LoRA adapters, stage-wise freezing, or split-block tuning are used to focus adaptation capacity on target features while keeping the bulk of the original model stable (e.g., RedWhale for Korean, Llama-3-SynE for science/Chinese) (Vo et al., 21 Aug 2024, Chen et al., 26 Jul 2024, Nag et al., 13 Dec 2024).

6. Applications and Empirical Benchmarks

CPT has demonstrated gains in diverse application domains:

• Language modeling: MMLU, C-Eval, CMMLU, MT-Bench, and translation tasks for cross-lingual and domain-adapted LLMs (Chen et al., 26 Jul 2024, Wu et al., 2 Feb 2024, Liu et al., 2021, Nag et al., 13 Dec 2024, Zheng et al., 2 Jul 2024, Vo et al., 21 Aug 2024, Ishibashi et al., 15 May 2025).
• Vision and remote sensing: Change detection, scene classification, semantic segmentation, and super-resolution in geospatial imagery (Mendieta et al., 2023).
• Speech and multimodal models: ASR, TTS, S2TT, and S2ST with codec-based speech LLMs, including end-to-end systems for multi-task speech-language understanding and generation (Shi et al., 24 Feb 2025, Attia et al., 15 May 2024, Attia et al., 13 Sep 2024).
• Scientific and technical reasoning: Synthetic data CPT (e.g., chain-of-thought “hidden thoughts”) consistently improves reasoning accuracy, cross-domain transfer, and adaptive chain length, particularly on difficult problems (Ishibashi et al., 15 May 2025).
• Model merging in CPT: Frameworks for combining CPT “experts” across finance/math/language domains to recover lost general capabilities and achieve synergy on cross-domain tasks have been systematically evaluated (Ueda et al., 4 Nov 2025).

7. Analytic Frameworks and Future Directions

Scaling laws and analytic tools for CPT are now foundational:

• CPT scaling laws: Closed-form models for loss trajectories as a function of model size, data size, learning rate, and training steps enable precise planning of CPT duration, mixtures, and replay (with $R^2>0.97$ on held-out tasks and setups) (Que et al., 3 Jun 2024, Wang et al., 12 May 2025, Zheng et al., 2 Jul 2024).
• Cross-domain prediction: “Cross-Domain D-CPT Law” accurately extrapolates the optimal mixture/loss behavior for a new target domain using minimal pilot runs (Que et al., 3 Jun 2024).
• Efficient adaptation: Analytical criteria and grid/tuning strategies enable minimal-cost, optimal-overlap CPT for both general and domain-specific goals.

Open research directions include: development of computation-efficient (modular, sparse) architectures, dynamic detection of data distribution shift and autonomous CPT scheduling, explicit unlearning and provenance-aware adaptation, and exploration of CPT dynamics for multimodal, multi-lingual, and multi-domain fusions.

---

References:

(Mendieta et al., 2023, Wang et al., 5 Oct 2024, Liu et al., 2021, Chen et al., 26 Jul 2024, Shi et al., 24 Feb 2025, Wu et al., 2 Feb 2024, Que et al., 3 Jun 2024, Attia et al., 15 May 2024, Nag et al., 13 Dec 2024, Zheng et al., 2 Jul 2024, Vo et al., 21 Aug 2024, Attia et al., 13 Sep 2024, Jiang et al., 15 Jul 2024, Cossu et al., 2022, Sun et al., 2023, Li et al., 5 Apr 2025, Elhady et al., 30 May 2025, Ishibashi et al., 15 May 2025, Wang et al., 12 May 2025, Ueda et al., 4 Nov 2025).