Continual Pre-Training: Efficient Model Updates

• Continual pre-training is the process of updating pre-trained models on new data or tasks while retaining prior knowledge to prevent catastrophic forgetting.
• It utilizes strategies like data replay, curriculum design, and adaptive learning-rate schedules to handle distribution shifts and integrate new modalities.
• Applications include cross-lingual adaptation, domain-specific enhancement, and multimodal integration, providing cost-effective alternatives to training from scratch.

Continual pre-training is the practice of taking an already pre-trained model and continuing its pre-training on new data, new self-supervised tasks, or newly arrived modalities, rather than restarting training from random initialization. In the contemporary literature, the term covers several closely related regimes: large-scale unsupervised continuation of LLMs on new corpora, continual self-supervised or multimodal pre-training of vision backbones, and continual learning with pre-training in which a pre-trained model is adapted across a stream of tasks while retaining prior capabilities (Gupta et al., 2023, Cossu et al., 2022, Wang et al., 2023). Across these settings, the central technical problem is the same: to acquire new knowledge under distribution shift without catastrophic forgetting, while preserving the computational advantages of reusing an existing foundation model.

1. Definition, scope, and historical formulations

A minimal formulation of continual pre-training treats pre-training data as a sequence of stages $D_1^{pr}, D_2^{pr}, \ldots, D_T^{pr}$, with model parameters updated stage by stage under a pre-training loss and later evaluated through downstream adaptation (Cossu et al., 2022). In language modeling, this has also been described as “fine-tuning at scale”: an LLM pre-trained on an upstream corpus is continued on a new downstream corpus, typically under the same causal language-model objective, because retraining from scratch whenever new data arrives is substantially more expensive (Gupta et al., 2023).

The scope of the term is broader than domain adaptation alone. "ERNIE 2.0" (Sun et al., 2019) framed continual pre-training as the incremental addition of new pre-training tasks through continual multi-task learning, rather than merely exposing a model to a new corpus. Later work on continual learning with pre-training formalized a different but adjacent setting: a strong pre-trained model is adapted to a stream of downstream tasks, and the pre-trained representation itself becomes part of the stability–plasticity trade-off (Wang et al., 2023, Zhang et al., 2024). As a result, the literature now includes upstream corpus continuation, sequential task adaptation with pre-trained backbones, and multimodal or multilingual expansion of already trained foundation models.

This breadth matters because the same term is used for distinct operational goals. Some studies seek monolingual or domain-specialized gains while preserving open-domain ability, as in finance, medicine, phone conversations, or English-to-Japanese adaptation (Xie et al., 2023, Guo et al., 2024, Fu et al., 7 Oct 2025, Fujii et al., 2024). Others seek representational continuity under a task stream, or the incorporation of new visual resolutions and modalities into existing vision systems (Sun et al., 2023, Chen et al., 24 Mar 2025, Yao et al., 24 Jun 2025). The unifying feature is that pre-existing parameters are treated as assets to be updated rather than discarded.

2. Objectives and optimization regimes

In LLMs, the dominant continual pre-training objective is unchanged from original pre-training: standard autoregressive next-token prediction with cross-entropy. Swallow minimizes

$$\mathcal{L}_{CE} = - \sum_{t=1}^T \log P(x_t \mid x_{<t}; \theta),$$

over mixed Japanese and English text, with no auxiliary objectives (Fujii et al., 2024). FinPythia uses the same next-token cross-entropy for financial domain-adaptive continual pre-training (Xie et al., 2023). The phone-conversation summarization system in "DACP: Domain-Adaptive Continual Pre-Training of LLMs for Phone Conversation Summarization" (Fu et al., 7 Oct 2025) likewise retains the standard next-token prediction objective on conversation transcripts, and the recommendation framework CPRec applies next-token cross-entropy to promptified behavioral sequences before any downstream supervised fine-tuning (Ma et al., 11 Apr 2025). In this sense, much of LLM continual pre-training is architecturally conservative: the model is updated under the same unsupervised criterion, while the main design freedom shifts to data composition, scheduling, and parameter update policy.

Other settings introduce auxiliary preservation or alignment terms. Continual domain-adaptive pre-training of masked LLMs in "Continual Pre-Training of LLMs" (Ke et al., 2023) combines masked-language modeling with a soft-masking mechanism that modulates gradients according to unit importance and an InfoNCE-style contrastive loss that pushes current full-network representations away from previously learned representations while integrating new knowledge. ERNIE 2.0 aggregates a growing set of task-specific losses in a continual multi-task objective,

$$\mathcal{L}(\theta) = \sum_{t \in T} \lambda_t \mathcal{L}_t(\theta),$$

covering knowledge masking, capitalization prediction, token-document relation, sentence reordering, sentence distance, discourse relation, and IR relevance (Sun et al., 2019).

In vision and multimodal regimes, continual pre-training objectives are often explicitly composite. CoMP adds an alignment loss between visual and textual features to the usual generative decoding loss during its first two stages, updating only the vision foundation model and adapter through the alignment term while gradients are stopped on the LLM and its word-embedding matrix (Chen et al., 24 Mar 2025). RetCoP combines CLIP-style contrastive pre-training with an off-diagonal information distillation loss that preserves the previous stage’s cross-modal similarity structure (Yao et al., 24 Jun 2025). GLARE sums a global DINO-style loss with local patch-level consistency and regional consistency constraints during continual self-supervised pre-training for semantic segmentation (Ebouky et al., 22 Sep 2025). These formulations indicate that outside causal language modeling, continual pre-training frequently couples acquisition of new information with explicit mechanisms for representation preservation or cross-modal alignment.

3. Data mixture, selection, and curriculum design

Data design is a primary control surface in continual pre-training. A recurrent pattern is replay or mixture with data that approximates the original pre-training distribution. Swallow uses a $9{:}1$ Japanese-to-English token ratio, with 10% of training tokens sampled from English—5% from RefinedWeb and 5% from the English ArXiv subset of The Pile—to mitigate catastrophic forgetting during English-to-Japanese adaptation (Fujii et al., 2024). The conversation-summarization DACP system mixes 25 billion in-domain transcript tokens with 25 billion tokens from FineWeb-Edu in a 1:1 ratio (Fu et al., 7 Oct 2025). The medical study "Efficient Continual Pre-training by Mitigating the Stability Gap" (Guo et al., 2024) advocates distribution-match mixing that replays original pre-training data in the same mixture ratios used originally, and MoE continual pre-training uses replay fractions such as 30% for code and 40% for German when continuing from FineWeb-trained checkpoints (Thérien et al., 6 Mar 2025). These recipes differ numerically, but all treat replay as a way to reduce distribution gap while retaining prior capabilities.

A second pattern is aggressive data selection. FinPythia defines a large unlabeled financial corpus $U$, optional unlabeled task data $T$, and a selected subset $D \subset U$, then proposes Efficient Task-Similar DACP (ETS-DACP), Efficient Task-Agnostic DACP (ETA-DACP), and a combined metric that averages quantile ranks of similarity, perplexity, and token-type entropy (Xie et al., 2023). The resulting schedules use only 10% of the corpus, about 2.39 billion tokens, and are explicitly framed as efficient alternatives to vanilla full-corpus continual pre-training. The phone-conversation DACP system ranks transcripts by token-type entropy and selects the top 25 million transcripts, while the medical stability-gap work uses a KenLM-trained high-quality reference corpus to choose low-perplexity subsets and then trains for multiple epochs on a smaller subset instead of a single pass over the full corpus (Fu et al., 7 Oct 2025, Guo et al., 2024). This suggests that token budget alone is not the sole relevant variable; corpus curation can dominate naive scale.

Curriculum design appears when the new distribution is especially distant from the original one. Llama-3-SynE performs 92.5 billion tokens of bilingual adaptation followed by 7.5 billion tokens of synthetic enhancement, using topic-based sampling weights adjusted by topic-wise perplexity changes and a perplexity-based curriculum that feeds Chinese documents from easy to hard (Chen et al., 2024). CPRec uses a Warmup–Stable–Annealing learning-rate schedule while also switching from domain-specific behavioral sequences to all-domain mixed behavioral sequences during the annealing phase (Ma et al., 11 Apr 2025). By contrast, the conversation-summarization DACP system reports that ordering transcripts by increasing ASR confidence or length did not outperform mixed-shuffled training (Fu et al., 7 Oct 2025). The literature therefore does not present curriculum as universally necessary; its value depends on the severity and structure of the shift.

Cross-lingual continual pre-training introduces additional data-format decisions. Swallow expands Llama 2’s vocabulary by adding 16,176 Japanese tokens, increasing the vocabulary from 27,000 original tokens to 43,176 tokens, because the raw tokenizer otherwise splits Japanese into UTF-8 bytes and lengthens sequences by about 56% (Fujii et al., 2024). For translation-oriented continuation, "Enhancing Translation Accuracy of LLMs through Continual Pre-Training on Parallel Data" (Kondo et al., 2024) shows that the ordering and formatting of parallel data are decisive: alternating source and target sentences is essential, improvements occur only for directions whose source-target order matches the order seen in continual pre-training, and the best results come from interleaved source and target sentences with language tags added to the source.

4. Forgetting, the stability gap, and schedule control

Catastrophic forgetting remains the canonical failure mode of continual pre-training. In LLM adaptation, the newer literature distinguishes ordinary forgetting from a more specific transient phenomenon: the stability gap. "Efficient Continual Pre-training by Mitigating the Stability Gap" (Guo et al., 2024) reports a V-shaped trajectory in which target-domain zero-shot accuracy first drops and then recovers, even while in-domain perplexity decreases monotonically. The paper interprets this through a plasticity–stability decomposition: early in continual pre-training the plasticity gradient dominates, domain knowledge is acquired quickly, but pre-existing instruction-following ability is destabilized; later, stability increases and performance recovers.

Learning-rate control is therefore treated as a first-class design variable. "Continual Pre-Training of LLMs: How to (re)warm your model?" (Gupta et al., 2023) studies linear warm-up followed by cosine decay when moving from The Pile to SlimPajama. It finds that warm-up length between 0% and 2% of downstream tokens has almost no long-term effect on validation perplexity, that higher $\ell r_{\max}$ improves downstream adaptation but increases forgetting on upstream data, and that every rewarmed fine-tuned model ultimately outperforms the from-scratch SlimPajama baseline. The same study also shows that applying the reheating schedule while continuing on the same dataset produces the same loss spike and that the model does not fully recover, implying that re-warming is not harmless in the absence of a genuine data shift.

The schedule question was revisited in "Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-training" (Singh et al., 4 Mar 2025). There, repeated cosine restarts are criticized for causing forgetting during the re-warming phase, and an Infinite Cosine schedule is proposed that combines an initial warm-up, cosine cooldown, constant-$\eta$ plateau, and final exponential annealing. The reported result is that the infinite schedule consistently improves continual pre-training performance relative to repeated cosine decay in both image and language settings, while also removing the need to predetermine a fixed terminal iteration budget.

In continual learning with pre-training, the analogous pathology is progressive overfitting. SLCA++ argues that sequential fine-tuning had been underestimated because prior CLPT baselines reused the same learning rate for backbone and classifier; when the backbone learning rate is reduced to $10^{-4}$ and the classifier learning rate is $$\mathcal{L}_{CE} = - \sum_{t=1}^T \log P(x_t \mid x_{<t}; \theta),$$0, sequential fine-tuning becomes a strong baseline (Zhang et al., 2024). The framework supplements slow backbone updates with post-hoc classifier alignment and symmetric cross-entropy, and also offers a Hybrid Slow Learner using LoRA. A common misconception addressed by this line of work is that frozen backbones plus prompts are the natural default in CLPT; the reported evidence instead favors full-parameter or hybrid adaptation when representation learning is slowed appropriately.

5. Representative application regimes

Cross-lingual adaptation is one of the clearest demonstrations of continual pre-training’s efficiency. Swallow extends Llama 2 to Japanese by vocabulary expansion and continual pre-training on a Japanese web corpus, with Japanese performance rising monotonically up to 100 billion tokens and improving by 7–9 average points across model sizes; the largest gains appear in question answering, translation from English to Japanese increases by 15–41%, and performance exceeds Japanese LLMs trained from scratch despite using fewer Japanese tokens (Fujii et al., 2024). Llama-3-SynE applies a different cross-lingual recipe: bilingual adaptation with an EN:ZH ratio of $$\mathcal{L}_{CE} = - \sum_{t=1}^T \log P(x_t \mid x_{<t}; \theta),$$1, then synthetic scientific and code QA enhancement with EN:ZH:SYN of $$\mathcal{L}_{CE} = - \sum_{t=1}^T \log P(x_t \mid x_{<t}; \theta),$$2, yielding gains of +8.81 on C-Eval, +6.31 on CMMLU, +12.00 on MATH, and +4.13 on SciEval while keeping degradation on MMLU and MBPP near 1.4 points (Chen et al., 2024). Parallel-data continuation further refines the translation-specific case by showing that source–target ordering and language tags determine whether benefits appear in English-to-Japanese, Japanese-to-English, or both directions (Kondo et al., 2024).

Domain-adaptive LLMs form a second major regime. FinPythia-6.9B continues Pythia-6.9B on a financial corpus and improves average financial-task F1 from 50.64 to 54.83, while ETS-DACP reaches 59.76 average F1 on Pythia-1B using only 2.39 billion tokens and no significant degradation on open-domain tasks (Xie et al., 2023). In medicine, multi-epoch subset training, high-quality subcorpus selection, and replay improve the average medical task performance of OpenLlama-3B from 36.2% to 40.7% using only 40% of the original training budget (Guo et al., 2024). In phone-conversation summarization, one epoch of 50 billion-token DACP followed by instruction tuning improves both internal and external summarization benchmarks, with especially large relative gains on QMSUM and QMSUM-I for both LLaMA-3.1-8B and Mistral-7B (Fu et al., 7 Oct 2025). These studies treat continual pre-training as a scalable substitute for training a new specialist model from scratch.

Recommendation extends the idea to nonlinguistic but text-serialized behavior. CPRec introduces an all-domain continual pre-training stage before any downstream supervised fine-tuning, using unified prompts over user behavioral sequences, domain-specific and all-domain mixed histories, and a Warmup–Stable–Annealing schedule (Ma et al., 11 Apr 2025). On five downstream datasets, the reported HR@1 values for CPRec are 0.4764, 0.4286, 0.3730, 0.2706, and 0.3129, with relative gains over LLaRA of +8.0%, +7.9%, +4.3%, +6.0%, and +11.1%, all with $$\mathcal{L}_{CE} = - \sum_{t=1}^T \log P(x_t \mid x_{<t}; \theta),$$3. The paper also reports that CPRec without downstream supervised fine-tuning outperforms SASRec on 4/5 datasets.

Vision and multimodal foundation models exhibit parallel but not identical dynamics. CoMP continually pre-trains vision foundation models in a multimodal way, introducing Continual Rotary Position Embedding for arbitrary image resolutions and a prototype-based alignment loss to align visual features with the LLM embedding space (Chen et al., 24 Mar 2025). After this continuation, CoMP-SigLIP-So/14 reports large gains on text-rich multimodal tasks such as ChartQA and DocVQA while largely preserving ImageNet-1K accuracy and improving ADE20K segmentation. RetCoP incrementally integrates CFP, FFA, and OCT modalities into a unified retinal vision–LLM using representative rehearsal pairs and off-diagonal information distillation, reporting markedly smaller forgetting than SeqFT on CFP and strong performance on OCT benchmarks (Yao et al., 24 Jun 2025). GLARE, by contrast, targets continual self-supervised pre-training for downstream semantic segmentation, freezing the original ViT backbone, training only UniAdapter layers, and improving mIoU over the underlying UDI baseline on ADE20k, Pascal Context, Cityscapes, and LoveDA (Ebouky et al., 22 Sep 2025).

6. Empirical regularities, trade-offs, and unresolved issues

A recurring empirical pattern is that continual pre-training can outperform from-scratch alternatives at substantially lower cost. Swallow is explicitly presented as an efficient path to a strong Japanese LLM, requiring about 100 billion tokens rather than at least 1 trillion for from-scratch training, and surpassing calm2-7B, LLM-jp-13B, and PLaMo-13B despite using fewer Japanese tokens (Fujii et al., 2024). FinPythia reports that selecting 10% of the financial corpus corresponds to a 90% cost reduction, with 24 billion tokens taking 18 days and 2.39 billion tokens taking about 1.8 days (Xie et al., 2023). The medical stability-gap work exceeds its 50-billion-token one-epoch baseline using 20 billion tokens, and MoE continual pre-training matches the performance of a fully retrained MoE at about one third of the FLOPs while preserving the sample-efficiency advantage of MoEs over dense transformers (Guo et al., 2024, Thérien et al., 6 Mar 2025). In this respect, efficiency claims in the literature are not merely qualitative; they are tied to explicit token and compute reductions.

The gains are nonetheless conditional and often asymmetric. Swallow reports small decreases in English average score, with reasoning tasks such as TriviaQA and GSM8K dropping by 6–23%, and vocabulary expansion is nearly performance-neutral except on XL-Sum summarization, where it reduces ROUGE-2 by about 5–15% (Fujii et al., 2024). The re-warming study documents a direct trade-off between downstream adaptation and upstream forgetting as maximum learning rate rises (Gupta et al., 2023). Parallel-data mixing improves translation but does not produce consistent gains on non-machine-translation tasks in Swallow, and in the dedicated translation study, continual pre-training improves only those translation directions whose source-target order matches the order seen during continuation (Fujii et al., 2024, Kondo et al., 2024). These results complicate any simple claim that continual pre-training is uniformly beneficial across all retained capabilities.

Another important issue is compatibility across model upgrades and downstream artifacts. "Recyclable Tuning for Continual Pre-training" (Qin et al., 2023) argues that when a continually pre-trained model is upgraded, previously stored adapted weights are typically discarded even though the upgraded model remains compatible with them to some extent. The paper analyzes this through linear mode connectivity and functional similarity, then proposes initialization-based and distillation-based recyclable tuning methods. This line of work broadens the topic from learning dynamics to model lifecycle management: continual pre-training changes not only the base model’s knowledge state but also the value of existing adapters, prompts, or deltas built on top of it.

A plausible implication is that future continual pre-training research will continue to converge on coupled recipes rather than isolated interventions. Across language, vision, multimodal, and recommendation settings, the reported successes combine several elements at once: retained or replayed source-distribution data, carefully selected or curriculum-ordered target data, schedules that control abrupt optimization shocks, and either explicit or implicit mechanisms for preserving prior representations (Cossu et al., 2022, Singh et al., 4 Mar 2025, Yao et al., 24 Jun 2025). The encyclopedia of continual pre-training is therefore best read not as a catalog of one-off tricks, but as an emerging design space for updating foundation models under non-stationary data.