CPT: Efficient Model Adaptation

• CPT is a method that resumes unsupervised pre-training on large models to adapt to new domains, languages, or modalities while preserving prior knowledge.
• It leverages original pre-training objectives with adaptive data mixing, curriculum scheduling, and replay strategies to prevent catastrophic forgetting.
• Empirical studies show that CPT can reduce compute costs by 25–50% and improve performance in low-resource settings while maintaining emergent model abilities.

Continued Pre-training (CPT) refers to the process of resuming the unsupervised or self-supervised pre-training of a large neural model—typically a transformer-based LLM or Seq2Seq encoder-decoder—after its initial pre-training has been completed, with the goal of adapting it to new domains, languages, or modalities. Unlike conventional fine-tuning, which uses supervised objectives for specific downstream tasks, CPT leverages the same (or extended) pre-training objectives—such as denoising, masked token prediction, or next-token prediction—applied to new or reweighted data regimes. This adaptation protocol is crucial in scenarios where retraining from scratch is computationally expensive or impractical, and when target domains or languages are underrepresented or absent from the original pre-training mixture.

1. Formal Definition and Position in Model Adaptation

CPT operates as an intermediate adaptation stage situated between initial large-scale pre-training and task-specific fine-tuning. It accepts an already well-trained model checkpoint as input and exposes it to new data distributions by further optimizing the canonical pre-training loss (e.g., cross-entropy for autoregressive models, denoising reconstruction for Seq2Seq). This is distinct from conventional continual learning, which usually involves incremental supervised task exposure and complex mechanisms to mitigate catastrophic forgetting. CPT, by contrast, emphasizes representation-level adaptation at the scale of billions of tokens—usually via unsupervised protocols (e.g., masked language modeling, denoising autoencoding)—with the aim of efficiently infusing new knowledge or capabilities while maintaining or minimally degrading prior competencies (Chen et al., 2024, Cossu et al., 2022, Liu et al., 2021).

Formally, let $M_0$ denote the base model pretrained on corpus $D_{\mathrm{base}}$. At CPT step $t$, parameters $\theta^{(t)}$ are optimized according to: $$\underset{\theta}{\operatorname{min}} \;\; \mathbb{E}_{x\sim D_{\mathrm{CPT}}} [ L_{\mathrm{pretrain}}(M_\theta(x)) ] + R(\theta)$$ where $D_{\mathrm{CPT}}$ is the new or expanded data regime, $L_{\mathrm{pretrain}}$ is the original self-supervised loss (e.g., next-token, MLM, denoising), and $R(\theta)$ may represent regularization or replay.

2. Methodological Variants and Data-Mixing Strategies

CPT protocols vary according to the adaptation goal (domain, language, modality), mixture strategy, curriculum, and mitigation of forgetting.

• Domain or Language Adaptation: Target data can be monolingual (for language adaptation), domain-specific (e.g., medical or code), or multimodal (speech, text, code-mixed). Data-mixing is driven by explicit mixture weights $\{w_i\}$, ratios, or perplexity-aware adaptive schedules. For example, in "Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation," noisy code-switched input is constructed for target domains by corrupting and partially translating monolingual text before denoising, enabling adaptation to both source and target tokens (Liu et al., 2021).
• Mixture Schedules: Fixed ratios (e.g., 80:20 English:Chinese), adaptive mixture weights based on held-out perplexity, and curriculums (e.g., “easy-to-hard” based on model perplexity) are applied to prioritize data that bridges knowledge gaps while minimizing redundancy (Chen et al., 2024, Zheng et al., 2024).
• Synthetic and Augmented Data: Synthetic QA or problem-solving data can be incorporated to target emergent capabilities, such as scientific or mathematical reasoning (Chen et al., 2024, Chen et al., 23 Jan 2025). Construction of this data involves prompting models to generate new problem-solution pairs or code snippets, often with manual discipline filtering and corruption tolerance thresholds.
• Replay and Forgetting Control: Replay of base data (e.g., English while adapting to a new language) is critical for catastrophic forgetting prevention. Experimental results show that even modest (5–30%) replay ratios effectively stabilize original-domain performance (Zheng et al., 2024, Thérien et al., 6 Mar 2025, Elhady et al., 30 May 2025). When omitted, severe early-phase degradation of “emergent abilities” is observed.

3. Objective Functions and Optimization

The CPT loss is typically a sum (or weighted sum) of the standard pre-training loss over all mixture components: $$\mathcal{L}_{\text{CPT}}(\theta) = \sum_{i=1}^K w_i \, \mathbb{E}_{x\sim D_i} [ L_{\text{pretrain}}(x) ]$$ Specialized objectives may be introduced for multi-modal adaptation, e.g., for speech, the cross-entropy for both text and codec token prediction, sampled via task-weighted probabilities (Shi et al., 24 Feb 2025). Additional regularization (e.g., dropout, weight decay) is sometimes used; anti-drift regularization (e.g., KL to initial weights) and knowledge distillation are generally not the default, but have been explored to modulate stability and transfer (Wang et al., 12 May 2025, Mousavi et al., 7 Jan 2026).

4. Scaling Laws, Data Selection, and Efficiency

Recent research has established closed-form scaling laws to predict loss and performance after CPT given model size $D_{\mathrm{base}}$0, adaptation dataset size $D_{\mathrm{base}}$1, mixture ratios $D_{\mathrm{base}}$2, and, in advanced formulations, perplexity statistics and pre-training budgets (Que et al., 2024, Liu et al., 25 Dec 2025, Goffinet et al., 27 Oct 2025). These scaling laws serve multiple purposes:

• Loss Prediction & Mixture Optimization: The D-CPT Law and its extensions express validation loss as

$D_{\mathrm{base}}$3

enabling practitioners to optimize (via pilot experiments and constrained search) the domain-vs-general mixture ($D_{\mathrm{base}}$4). Cross-domain extensions and PTPP-aware adaptations incorporate learnable domain coefficients and pretraining budget ($D_{\mathrm{base}}$5) explicitly, ensuring accuracy and transferability across domains (Que et al., 2024, Goffinet et al., 27 Oct 2025).

• Data Subset Selection: Perplexity-aware CPT scaling laws prescribe selecting “knowledge gap” data with intermediate perplexity—yielding maximal loss reduction per token—and, via greedy or optimal subset selection (e.g., DOS), maximize utility of limited adaptation tokens (Liu et al., 25 Dec 2025, Nag et al., 2024).
• Compute Efficiency: CPT converges significantly faster (25–50% FLOP savings at equal loss) vs. training from scratch, particularly for large models and high-similarity domains/languages (Zheng et al., 2024).

5. Catastrophic Forgetting, Knowledge Dynamics, and Emergent Abilities

• Mechanisms of Forgetting: Empirical studies demonstrate that, without replay or tailored curriculum, CPT can induce catastrophic forgetting—especially of emergent in-context learning abilities—early in training, even if in-distribution perplexity suggests no degradation (Elhady et al., 30 May 2025). The parameter-shift profile is a sensitive marker for this phase—excessive early drift results in irreversible skill loss. Replay and adaptive interventions (English injection, EMA of weights, or curriculum mixing in initial steps) are necessary to preserve generalization (Elhady et al., 30 May 2025).
• Knowledge Instability: Direct probes into knowledge circuits during factual CPT illustrate non-monotonic, unstable acquisition and consolidation of new information: learning and forgetting alternate within early epochs, and post-hoc recall peaks often occur ahead of minimum loss, indicating standard optimization criteria are fundamentally misaligned with true knowledge absorption (Mousavi et al., 7 Jan 2026).

6. Practical Applications and Empirical Outcomes

• Language Adaptation: CPT is especially effective for expanding coverage to low-resource or unseen languages in multilingual LLMs—boosting BLEU scores by 1–3 points over strong baselines with minimal adaptation compute (Liu et al., 2021), and raising accuracy on African/Indic/Estonian languages by 8–15 points on diverse benchmarks without harming English/general reasoning (Dorkin et al., 2 Mar 2026, Nag et al., 2024, Yu et al., 10 Jan 2026, Zheng et al., 2024).
• Domain Adaptation: Task-aligned data mixtures (math/code/synthetic QA) can be leveraged to reach or exceed native reasoning capabilities, outperforming supervised SFT at equivalent data scales. Repeatable empirical gains in domain-specific performance, robustness to noise (ASR), and long-context translation have been demonstrated (Chen et al., 23 Jan 2025, Attia et al., 2024, Attia et al., 2024, Yu et al., 10 Jan 2026).
• Architectural Generalization: CPT is effective for adaptation in both dense and mixture-of-expert (MoE) transformers. MoE transformers maintain sample efficiency and router balance under CPT even without replay, achieving performance equal to full retraining at a fraction of compute cost (Thérien et al., 6 Mar 2025).
• Emergent Abilities: Transfer of emergent abilities (in-context learning) to new languages during CPT requires careful management of the “critical period.” English-mixture, or alternatives such as curriculum or EMA, are essential to prevent their collapse (Elhady et al., 30 May 2025).

7. Best Practices and Limitations

• Mixture Calibration: Adjust mixture ratios with scaling laws or perplexity-aware selection, and prioritize pilot ablations over grid search for cost-efficiency (Que et al., 2024, Liu et al., 25 Dec 2025).
• Replay/Ratios: Maintain 10–30% source/replay in cross-lingual/domain CPT to control forgetting (Zheng et al., 2024, Chen et al., 2024, Thérien et al., 6 Mar 2025).
• Curriculum and EMA/Anchoring: Use curriculum scheduling or EMA if replay is impractical (Elhady et al., 30 May 2025).
• Synthetic Data: Limit corruption to ≤30% in synthetic QA generation for reasoning tasks; higher fractions cause severe performance drops (Chen et al., 2024).
• Evaluate Beyond Perplexity: Incorporate explicit knowledge, ICL, and OOD probes throughout CPT; loss minimization is a poor proxy for generalization or factual learning (Mousavi et al., 7 Jan 2026, Elhady et al., 30 May 2025).
• Model Scale/Architecture: Architectural choices (normalization, attention kernel) dominate transfer gains in challenging languages/domains, but larger parameter models are always preferable within family (Yu et al., 10 Jan 2026, Zheng et al., 2024).
• Limitations: Scaling law generalization may require re-fitting when switching domains, families, or pre-training budgets; capacity limits of continual consolidation are poorly understood (Goffinet et al., 27 Oct 2025, Mousavi et al., 7 Jan 2026).

In summary, CPT provides a robust, flexible, and compute-efficient paradigm for post-hoc adaptation and capability expansion of LLMs and related models, with empirical best practices and analytic theory converging to enable traceable, testable, and predictively optimized adaptation pipelines. Continued research is clarifying the interplay between optimization dynamics, knowledge stability, and emergent behavior, guiding the development of future scalable CPT-informed adaptation methodologies.