Continual Pre-training Overview

• Continual pre-training is a sequential adaptation of pre-trained models to new data while retaining previously acquired knowledge.
• Techniques involve self-supervised tasks such as next-token prediction and masked language modeling to balance stability and plasticity.
• Data strategies like replay, selection, and mixture design are crucial for mitigating catastrophic forgetting and enhancing cross-domain transfer.

Searching arXiv for recent and foundational papers on continual pre-training to ground the article. arxiv_search(query="continual pre-training LLMs domain adaptation replay forgetting", max_results=10, sort_by="relevance") Continual pre-training is the paradigm in which a pre-trained model is updated sequentially as new corpora, domains, modalities, or pre-training tasks become available, rather than being trained once on a fixed corpus or re-trained from scratch. In contemporary language modeling, the term often denotes continuing pretraining with self-supervised next-token prediction on unlabeled target-domain data; in broader usage it also includes masked-language-model updates, continual multi-task pre-training, multimodal contrastive continuation, and modality-incremental vision-language pre-training. Across these settings, the central technical problem is to increase plasticity toward new data while preserving previously acquired general knowledge, downstream adaptability, and cross-domain transfer (Fu et al., 7 Oct 2025, Gupta et al., 2023, Sun et al., 2019, Chen et al., 24 Mar 2025).

1. Conceptual scope and formal settings

A canonical formulation treats continual pre-training as a sequence of model upgrades. In recyclable tuning, for example, continual pre-training produces a chain

$$\mathcal{M}_0 \rightarrow \mathcal{M}_1 \rightarrow \cdots \rightarrow \mathcal{M}_4,$$

where $\mathcal{M}_0$ is the original released PLM and each $\mathcal{M}_{i+1}$ is obtained by further training $\mathcal{M}_i$ on newly arriving data domains (Qin et al., 2023). In continual domain-adaptive pre-training, the same idea is instantiated as sequential adaptation to unlabeled domain corpora $D_1, D_2, \dots, D_T$, with the additional requirement that previously seen corpora are no longer accessible after each step (Ke et al., 2023).

The objective used during continual pre-training depends on model class and modality. In decoder-only LLM adaptation, the dominant form is continued autoregressive language modeling on raw unlabeled data, as in domain-adaptive continual pre-training for phone conversation summarization, Japanese cross-lingual adaptation, scientific and bilingual adaptation of Llama-3, and agent-oriented pre-training (Fu et al., 7 Oct 2025, Fujii et al., 2024, Chen et al., 2024, Zhuang et al., 10 Feb 2025). Encoder-style work instead uses masked language modeling and related pre-training tasks, including the continual multi-task framework of ERNIE 2.0 and the soft-masked continual DAP setting of DAS (Sun et al., 2019, Ke et al., 2023). In multimodal work, continual pre-training may optimize image-text contrastive alignment, decoding losses, or alignment-specific auxiliary objectives, as in CoMP, RetCoP, and retinal modality-incremental pre-training (Chen et al., 24 Mar 2025, Yao et al., 24 Jun 2025).

Several papers explicitly distinguish continual pre-training from both supervised fine-tuning and full retraining. DACP frames it as a self-supervised stage that teaches an LLM the language, structure, terminology, and discourse patterns of business conversations without using human-written summaries (Fu et al., 7 Oct 2025). Rewarming studies frame it as a compute-efficient alternative to restarting optimization from random initialization when a new large corpus arrives (Gupta et al., 2023). This suggests that “continual pre-training” is best understood as a family of sequential pre-training regimes rather than a single algorithm.

2. Historical development and relation to neighboring paradigms

An early influential formulation appears in ERNIE 2.0, where continual pre-training is not merely continued exposure to new corpora but the incremental construction of new pre-training tasks. ERNIE 2.0 adds lexical, structural, and semantic objectives over time and updates a shared encoder through continual multi-task learning, rather than fixing the objective set at the outset (Sun et al., 2019). This established a broad interpretation of continual pre-training as extensible pre-training.

Later work shifted attention toward continual updating of already strong pretrained models. “Continual Pre-Training Mitigates Forgetting in Language and Vision” formalized the setting in which a model is repeatedly pre-trained on incoming data and only afterward fine-tuned for downstream tasks, showing that self-supervised continual pre-training is more effective in retaining previous knowledge than supervised protocols in both language and vision (Cossu et al., 2022). Around the same period, continual DAP-training for RoBERTa highlighted a distinct requirement of language-model continual adaptation: the full LM must remain useful after sequential domain updates, rather than being partitioned into isolated domain-specific subnetworks (Ke et al., 2023).

With the rise of LLMs, continual pre-training increasingly became the practical mechanism for domain specialization, cross-lingual transfer, and capability extension. FinPythia-6.9B adapts Pythia to finance instead of training a finance model from scratch (Xie et al., 2023). Swallow adapts Llama 2 to Japanese through cross-lingual continual pre-training (Fujii et al., 2024). Llama-3-SynE uses a two-stage continual pre-training pipeline to improve Chinese language ability and multidisciplinary scientific reasoning (Chen et al., 2024). Hephaestus treats agentic competence as something that should be learned at pre-training time rather than imposed purely through prompting or post-training (Zhuang et al., 10 Feb 2025). DACP applies the same logic to noisy conversational summarization in industrial settings (Fu et al., 7 Oct 2025).

The neighboring paradigms remain important. Supervised fine-tuning is still used after continual pre-training in DACP, translation adaptation on parallel data, Hephaestus, and medical pipelines, because continual pre-training alone does not teach prompt following or task formatting (Fu et al., 7 Oct 2025, Kondo et al., 2024, Zhuang et al., 10 Feb 2025, Guo et al., 2024). Prompting, adapters, and LoRA are also not displaced; rather, continual pre-training often serves as the upstream stage that makes later parameter-efficient adaptation more effective.

3. Objectives, forgetting, and optimization dynamics

The defining optimization tension in continual pre-training is the stability–plasticity tradeoff. A medical LLM study makes this explicit through

$$\mathcal{L}=\alpha\mathcal{L}_{plasticity}+(1-\alpha)\mathcal{L}_{stability},$$

using it to interpret the “stability gap”: downstream task performance first drops, then recovers, even while domain perplexity decreases steadily during continual pre-training (Guo et al., 2024). In that study, the drop appears during the first 5B tokens for the main medical setup, and relative weight updates show that bottom layers change more than top layers early in training, with a bottom/top update ratio $> 1.35$ (Guo et al., 2024). This suggests that early continual updates can transiently damage instruction-following or task-execution behavior before a more stable regime is reached.

Learning-rate scheduling is therefore not a minor detail. Rewarming experiments on Pythia 410M show that when moving from the Pile to SlimPajama, the maximum learning rate is the main control knob: higher maximum LR improves downstream adaptation but increases forgetting on the upstream data, while warmup length from $0\%$ to $2\%$ of downstream tokens has little effect on final perplexity (Gupta et al., 2023). That work also reports a transient loss spike at the start of rewarming, yet finds that rewarmed continual pre-training eventually outperforms training from scratch on the downstream corpus (Gupta et al., 2023).

A later study argues more strongly that repeated cosine rewarming is itself harmful in continual self-supervised pre-training. It compares repeated cosine decay with the Infinite Cosine Schedule and attributes forgetting partly to the re-warming phase, recommending checkpointing on the constant-LR plateau and resuming future stages without rewarming (Singh et al., 4 Mar 2025). Across MAE pre-training and autoregressive language-model pre-training, the infinite schedule improves retention and often improves average performance, especially when combined with replay (Singh et al., 4 Mar 2025). A plausible implication is that continual pre-training requires schedules designed for open-ended continuation rather than finite single-run optimization.

Architectural continuity also matters. Recyclable tuning shows that continually pre-trained PLMs remain compatible with outdated adapted weights to a non-trivial degree, and analyzes this through linear mode connectivity and functional similarity (Qin et al., 2023). That result is technically significant because it implies that continual pre-training does not necessarily relocate the model to a disconnected region of parameter space.

4. Data engineering: replay, selection, mixture, and ordering

If optimization controls plasticity, data engineering controls what the model becomes plastic to. Replay is the most direct mechanism for retaining prior competence. DACP mixes 25B tokens of in-domain business conversation data with 25B tokens of replay data from FineWeb-Edu, explicitly using experience replay to reduce catastrophic forgetting (Fu et al., 7 Oct 2025). Swallow includes roughly 90% Japanese and 10% English in its continual-pretraining mix to mitigate forgetting during English-to-Japanese adaptation (Fujii et al., 2024). MoE continual pre-training on FineWeb to Stack or German Common Crawl also treats replay percentage as a hyperparameter that trades adaptation against forgetting (Thérien et al., 6 Mar 2025).

Data selection is not reducible to corpus size. FinPythia proposes task-aware and task-agnostic selection strategies and reports that carefully selected 10% subsets can outperform vanilla continual pre-training on 100% of the financial corpus, while preserving open-domain performance on ARC, MMLU, TruthfulQA, and HellaSwag (Xie et al., 2023). DACP uses token-type entropy to choose 25M transcripts from an initial 50M business transcripts, emphasizing quality and diversity rather than indiscriminate scale (Fu et al., 7 Oct 2025). Medical continual pre-training similarly reports that using a properly sized subset for multiple epochs can recover faster than one-epoch training on a large corpus, with 5B high-quality tokens emerging as the best tradeoff in the main setup (Guo et al., 2024).

Mixture design is often domain-specific. Hephaestus performs a scaling-law study over agent data, text, and code, finding that the optimal agent-data fraction is about 36% and that the best overall mix is approximately 1:1:1 among agent data, code data, and text data (Zhuang et al., 10 Feb 2025). Llama-3-SynE uses a 2:8 Chinese:English ratio in its bilingual adaptation stage and a 1:7:2 Chinese:English:synthetic ratio in its synthetic enhancement stage, alongside topic-based mixture adjustment and a perplexity-based easy-to-hard curriculum (Chen et al., 2024). CPRec organizes recommendation behavior into domain-specific and all-domain mixed sequences, then trains with a Warmup-Stable-Annealing schedule so that easier single-domain sequences precede harder mixed-domain sequences (Ma et al., 11 Apr 2025).

Ordering and formatting can themselves be decisive. In continual pre-training for translation, parallel data helps only when source and target sentences are alternated in the pre-training stream, and gains occur only for translation directions aligned with the source–target order used during continual pre-training (Kondo et al., 2024). The best results arise from interleaved source-target data with explicit language tags on the source sentence (Kondo et al., 2024). This indicates that continual pre-training can encode directionality and control structure from sequence format alone.

5. Domains, tasks, and modalities of application

The dominant application remains domain specialization of LLMs. In finance, FinPythia-6.9B improves average 5-shot financial-task performance over Pythia-6.9B by about 8.3% average F1, with particularly strong gains on FPB, Headline, and NER (Xie et al., 2023). In medicine, strategies designed to mitigate the stability gap improve the average medical task performance of OpenLLaMA-3B from 36.2% to 40.7% using only 40% of the original training budget, while later yielding Llama-3-Physician-8B with 76.7 average on the reported medical fine-tuning benchmark (Guo et al., 2024). In phone conversation summarization, DACP improves both internal and external summarization benchmarks; for Mistral-v0.3-7B, it increases Action Items ROUGE-1 by 6.32% and Support Call Summarization ROUGE-1 by 4.11%, while also improving external meeting summarization benchmarks and winning 45% of pairwise LLM-judge comparisons versus 29% for the non-DACP baseline (Fu et al., 7 Oct 2025).

Cross-lingual adaptation is another major use case. Swallow shows that Japanese performance increases monotonically with training data up to 100B tokens, with average Japanese scores improving from 32.0 to 39.4 for 7B, 39.6 to 46.3 for 13B, and 48.3 to 55.3 for 70B over the base Llama 2 models (Fujii et al., 2024). It also reports that vocabulary expansion yields 56.2% token reduction in the Swallow Corpus and up to 78% improvement in Japanese text generation efficiency, while combined use of parallel corpora improves translation ability (Fujii et al., 2024). Translation-specific continual pre-training on JParaCrawl v3.0 further shows that decoder-only LLMs become more robust on spoken-language and dialogue-like test sets when continual pre-training on parallel data is followed by supervised fine-tuning (Kondo et al., 2024).

Continual pre-training is also being used to inject capabilities not well represented in ordinary web text. Hephaestus continually pre-trains LLaMA-3-8B on Hephaestus-Forge, a 103B agent-specific corpus covering 76,537 APIs, and reports that Hephaestus-8B-IFT achieves 70.78 on BFCL-v2 overall, improving over LLaMA-3-8B-IFT at 62.12 (Zhuang et al., 10 Feb 2025). Llama-3-SynE uses synthetic scientific and code QA data to improve C-Eval from 49.43 to 58.24, CMMLU from 51.03 to 57.34, and MATH from 16.20 to 28.20 relative to Llama-3-8B (Chen et al., 2024).

The paradigm is not confined to text LLMs. CoMP continually pre-trains vision foundation models in a multimodal image-text setting, introducing C-RoPE for native-resolution processing and an Alignment Loss for cross-modal alignment; after continual pre-training, models such as SigLIP and DINOv2 improve on multimodal understanding tasks while preserving or improving generic classification and segmentation performance (Chen et al., 24 Mar 2025). RetCoP extends continual vision-language pre-training to modality-incremental retinal imaging, using rehearsal and off-diagonal information distillation to integrate CFP, FFA, and OCT sequentially into a unified model (Yao et al., 24 Jun 2025). CPRec uses all-domain continual pre-training of recommendation-oriented behavioral sequences to align LLMs with universal user behavior patterns before downstream recommendation fine-tuning (Ma et al., 11 Apr 2025).

6. Downstream reuse, evaluation, and unresolved issues

A recurring question is what happens to downstream adaptations when the backbone is continually pre-trained. Recyclable tuning addresses this directly by showing that old task-specific deltas remain useful on upgraded PLMs, and by proposing initialization-based and distillation-based transfer methods that improve convergence speed and final performance when adapting the upgraded backbone (Qin et al., 2023). This is one route to making continual pre-training operationally compatible with real deployment pipelines in which tuned checkpoints already exist.

Evaluation remains nontrivial. Some work measures validation perplexity on old and new corpora (Gupta et al., 2023); some emphasizes downstream task metrics such as ROUGE, BERTScore, and AlignScore for summarization (Fu et al., 7 Oct 2025); some uses continual-learning metrics such as Average Accuracy, Forward Transfer, and Backward Transfer (Singh et al., 4 Mar 2025); and some argues that frozen-feature evaluation can be misleading. In continual representation learning, kNN-only protocols can overstate the benefits of some continual methods, whereas LP-FT-style adaptation changes the ranking substantially (Sun et al., 2023). This suggests that continual pre-training should be judged by how well the resulting representation supports the adaptation method actually used in practice.

Several limitations recur across the literature. Many domain-specific studies evaluate only within a narrow target family: DACP focuses on summarization tasks in a conversational domain and cannot release internal data because of privacy concerns (Fu et al., 7 Oct 2025); medical continual pre-training notes that broader evaluation across more domains and models would be expensive (Guo et al., 2024); RetCoP studies three retinal modalities in a fixed incremental order (Yao et al., 24 Jun 2025). Cross-domain transfer is also not automatic. Translation work shows that benefits depend on sentence ordering and explicit direction cues (Kondo et al., 2024). HPrompt-CPT argues that many existing continual pre-training methods can decrease performance on unseen domains under an “anytime fine-tuning” criterion, and proposes hypernetwork-generated prompts with agreement and disagreement losses to improve adaptability, generalization, and final performance across sequential domains (Jiang et al., 2023).

Taken together, the literature portrays continual pre-training as an upstream adaptation layer between one-shot foundation-model pre-training and downstream fine-tuning. Its most mature forms combine conservative optimization, replay or mixture design, high-quality or high-diversity data selection, and task-aware evaluation. The open problems are correspondingly clear: preserving unseen-domain transfer, choosing schedules that do not induce boundary instability, extending the paradigm to tokenizer and modality evolution, and defining evaluation protocols that capture not only immediate domain gains but also long-horizon reuse of the continually updated model (Singh et al., 4 Mar 2025, Jiang et al., 2023, Qin et al., 2023).