Continued Domain-Specific Pre-training

• Continued domain-specific pre-training is the practice of further training a general pretrained model on unlabeled target-domain data to adapt it for specialized tasks.
• It employs techniques such as domain-adaptive pretraining (DAPT), task-adaptive pretraining (TAPT), and structure-aware masking to align models with niche distributions.
• Empirical studies across clinical, finance, and scientific fields demonstrate improved performance and efficiency while mitigating issues like catastrophic forgetting.

Continued domain-specific pre-training is the practice of taking an already pretrained model and training it further on additional unlabeled target-domain data before downstream fine-tuning. In the literature, it appears as domain-adaptive pretraining, task-adaptive pretraining, domain-tuning, or continual pre-training, and it is motivated by the mismatch between broad pretraining corpora and specialized downstream distributions. The central premise is that a strong general model remains improvable when its self-supervised objective is re-exposed to the vocabulary, discourse structure, metadata, or semi-structured formats of a target domain; several studies also position it as a cheaper alternative to training specialist models from scratch (Gururangan et al., 2020, Rongali et al., 2020, Xie et al., 2023).

1. Historical framing and conceptual boundaries

The modern formulation was established clearly by "Don't Stop Pretraining" (Gururangan et al., 2020), which distinguished domain-adaptive pretraining (dapt) from task-adaptive pretraining (tapt). In that framework, dapt continues RoBERTa on a large unlabeled corpus from a broad domain such as biomedical papers, computer science papers, news, or reviews, whereas tapt continues pretraining on the unlabeled corpus associated with the downstream task itself. Across four domains and eight classification tasks, the paper reported that dapt improved over RoBERTa on all domains in the main results table, and that dapt + tapt gave the best overall results (Gururangan et al., 2020). That study also made an enduring distinction between broad domain alignment and narrower task-distribution alignment.

A parallel line of work framed the same phenomenon as domain-tuning. "Continual Domain-Tuning for Pretrained LLMs" (Rongali et al., 2020) treated the simplest baseline as simple domain tuning (SDT), where the model is optimized only on the target-domain self-supervised objective, and argued that this can induce catastrophic forgetting of source-domain patterns. This shifted the discussion from pure adaptation toward a stability–plasticity problem: a domain-tuned model should improve on the target domain without erasing useful source-domain competence.

Subsequent work extended the same logic to increasingly diverse settings. Finance and e-commerce papers explicitly describe continual pre-training as a practical alternative to training a domain model from scratch, particularly when domain data is smaller, more expensive, or proprietary (Xie et al., 2023, Ma et al., 2023). More recent studies apply the same idea to low-resource languages, materials science, protein LLMs, and long-context mental-health modeling, indicating that continued pre-training is no longer a narrow NLP recipe but a general adaptation paradigm (Hassan et al., 13 Jan 2026, Kim et al., 2024, Mares et al., 16 Jul 2025, Ji et al., 2023).

2. Objectives, architectures, and parameterization

A striking feature of the literature is how often continued pre-training preserves the original backbone and reuses the original pretraining objective. For encoder models, the default remains masked language modeling (MLM) or denoising-style reconstruction. Clinical dialogue adaptation starts from RoBERTa-base, keeps the Transformer architecture, and uses cross-entropy loss in a bi-directional mask-recovering training scheme over corrupted conversations (Liu et al., 2022). Materials adaptation in MELT also remains standard MLM on a materials corpus; the novelty is not the loss form but the definition of which entities are masked (Kim et al., 2024).

For decoder-style LLMs, the objective is usually standard next-token prediction. Qalb begins from LLaMA-3.1 8B and performs continued pre-training with the autoregressive objective

$$\mathcal{L}_{\text{LM}} = - \sum_{t=1}^{T} \log p(x_t \mid x_{<t}),$$

over 1.97 billion total tokens of Urdu and English replay data (Hassan et al., 13 Jan 2026). EcomGPT-CT likewise inserts a continual pre-training stage between general pretraining and supervised fine-tuning while keeping the usual autoregressive next-token objective on mixed general and e-commerce corpora (Ma et al., 2023). MentalXLNet preserves Permutation Language Modeling (PLM), whereas MentalLongformer preserves MLM with a longer context window (Ji et al., 2023).

Parameter updating strategies vary. Some studies fully update the model; others use parameter-efficient routes. Qalb uses LoRA for parameter-efficient continued pre-training with Rank $r = 128$, LoRA alpha = 32, and about 1.18B trainable parameters, or 14.72% of the base model (Hassan et al., 13 Jan 2026). The insight-learning study on medicine and finance also adopts LoRA, but its main result is that LoRA-based continual pre-training on original documents has only a marginal effect, whereas simplified documents retaining only essential triples help much more, especially for declarative and statistical insights (Pezeshkpour et al., 29 Jan 2025).

The broader implication is that continued pre-training is usually architecturally conservative. The dominant operation is not redesigning the model, but redirecting its self-supervised learning pressure toward a new distribution.

3. Corpus construction, selection, and mixture design

Corpus engineering is central because the adaptation signal is unsupervised. In the original DAPT study, domain corpora were large and explicitly delineated: 2.68M biomedical full-text papers from S2ORC (7.55B tokens), 2.22M computer science papers (8.10B tokens), 11.90M news articles from RealNews (6.66B tokens), and 24.75M Amazon reviews (2.11B tokens) (Gururangan et al., 2020). Later work retained the same intuition but made corpus design more targeted.

Low-resource language adaptation makes mixture design explicit. Qalb uses 1.84 billion Urdu tokens and 140 million English Wikipedia tokens, with the English portion included to prevent catastrophic forgetting and preserve general reasoning and English fluency (Hassan et al., 13 Jan 2026). In finance, FinPythia-6.9B is continued on a 23.9 billion token corpus built from financial news and SEC filings, but the paper’s strongest methodological contribution is not scale alone: ETS-DACP and ETA-DACP show that carefully selected subsets can outperform vanilla full-corpus DACP, with ETS-DACP hard sampling on only 10% of the corpus reaching an average F1 of 59.76 against 55.14 for full-corpus DACP on the 1B model (Xie et al., 2023).

Other work focuses on mining or restructuring the corpus itself. DoPAMine uses a four-stage pipeline—corpus preprocessing and indexing, synthetic seed generation, nearest-neighbor retrieval, and downstream labeling / continual pre-training—to mine domain-specific real-world text from Common Crawl rather than relying on synthetic text directly. In its continual pre-training runs, the mixture is 25% domain-specific mined data and 75% unlabeled Common Crawl, explicitly to reduce catastrophic forgetting (Arannil et al., 2024). EcomGPT-CT uses a 2:1 token ratio between general and domain-specific data and further groups titles, properties, descriptions, and reviews by product ID before constructing mixed samples through a heterogeneous graph, so that semi-structured product information is encountered jointly rather than as isolated snippets (Ma et al., 2023).

Corpus representation can matter as much as corpus membership. The insight-learning study finds that training on original domain documents helps only a little, while simplifying each document to extracted and rephrased triples substantially improves declarative and statistical insight learning; training directly on triples makes declarative insight near-perfect but leaves probabilistic insight difficult (Pezeshkpour et al., 29 Jan 2025). This suggests that continued pre-training is not only a question of how much domain text is used, but also of what informational form that text takes.

4. Domain-aware objectives beyond naive continuation

A major development has been the move from generic continued MLM toward objectives that encode the internal structure of the target domain. Clinical dialogue work offers a clear example. In nurse–patient inquiry-answering conversations, the authors argue that speaker roles, utterance boundaries, and topic-driven discourse flow are semantically central. Their continued pre-training therefore adds six sample-construction strategies—Token Masking, Token Infilling, Speaker Masking, Speaker Permutation, Utterance Masking, and Intra-Topic Utterance Permutation—and prepends a special <u> token to mark utterance boundaries. The ablation progresses from 88.79 EM / 90.30 F1 with only token-level masking to 92.31 EM / 93.69 F1 with the full conversation-aware strategy, indicating that speaker- and utterance-level corruption is especially beneficial (Liu et al., 2022).

"Difference-Masking" (Wilf et al., 2023) generalizes a similar idea: the masking budget should focus on what distinguishes the target corpus from the source pretraining corpus. It identifies difference anchors using TF-ICF and then masks tokens according to cosine similarity to those anchors, rather than at random. Across ACL-ARC, ChemProt, Social-IQ, and TVQA, it is reported as best or tied-best, supporting the view that choosing what to mask is itself a domain-adaptation mechanism (Wilf et al., 2023).

MELT extends this from token selection to knowledge-structured curricula. It builds a materials semantic graph from extracted entities and relation-aware expansion, increasing coverage from 332,724 unique entities to 713,241 with expansion, and then orders masking via a 3-stage curriculum based on node degree. The result is an overall 0.741 Micro-F1 and 0.556 Macro-F1 on MatSci-NLP, with ablations showing degradation when either curriculum-based entity learning or materials-aware masking is removed (Kim et al., 2024).

FastDoc departs further from token-level self-supervision by using document metadata and domain-specific taxonomy as supervision. Continued pre-training is performed on sentence-level embeddings with a triplet loss and a hierarchical classification loss,

$$\mathcal{L} = \mathcal{L}_{t} + \mathcal{L}_{hier},$$

and then the higher-level encoder is fine-tuned with token embeddings. The method reports pre-training compute reductions of around 1,000, 4,500, and 500 times compared to MLM and/or NSP in customer support, scientific, and legal domains, respectively, while either outperforming or matching several baselines (Nandy et al., 2023). A plausible implication is that domain specificity can be injected through supervision structure, not only through lexical distribution.

5. Forgetting, generalization, and compute allocation

Continued pre-training almost always raises the question of how much specialization can be gained without damaging general ability. The early continual domain-tuning study formalized this directly: SDT optimizes only the target-domain objective, whereas continual-learning alternatives add constraints such as Rehearsal, L2 regularization, Elastic Weight Consolidation (EWC), or Gradient Episodic Memory (GEM). EWC, for example, uses

$$L_{EWC} = \ell_t + \frac{\lambda}{2}\sum_i F_i(\theta_i - \theta^*_{s,i})^2,$$

so that parameters important to source-domain performance are changed less aggressively (Rongali et al., 2020). That paper reports that constraining the model from forgetting the source domain can yield downstream models more robust to domain shifts.

Later work often implements the same concern through data mixing rather than explicit regularization. Qalb uses English Wikipedia as a replay buffer (Hassan et al., 13 Jan 2026). DoPAMine mixes mined domain data with Common Crawl at 25% / 75% (Arannil et al., 2024). EcomGPT-CT shows that adding general + domain data has minimal negative impact on general NLP tasks, whereas using only domain-specific data can hurt Chinese general benchmarks (Ma et al., 2023). FastDoc reports a negligible drop on open domain and attributes this partly to much smaller parameter movement than MLM-based pretraining (Nandy et al., 2023). FinPythia finds that selective continual pretraining improves finance performance while largely preserving open-domain scores, and that even full 100% DACP only slightly lowers average normalized open-domain performance (Xie et al., 2023).

A more recent strand turns this balance into an optimization problem. D-CPT Law models validation loss as a function of model size, dataset size, and mixture ratio:

$$L(N,D,r) = E + \frac{A}{N^\alpha} + \frac{B \cdot r^\eta}{D^\beta} + \frac{C}{r'^\gamma}, \quad r' = r+\epsilon.$$

The explicit goal is to predict the optimal ratio between general-corpus and downstream domain-corpus without exhaustive grid search (Que et al., 2024). "Optimal Splitting of LLMs from Mixtures to Specialized Domains" (Seto et al., 19 Mar 2026) extends the same logic to a two-stage process in which a shared seed model is pretrained on a general mixture and then copied into multiple experts for domain-specific continued pretraining, with total training cost $D + K \times D'$. Both studies treat continued pre-training not as a fixed recipe but as a resource-allocation problem.

The stability issue also extends beyond seen domains. HPrompt-CPT evaluates anytime fine-tuning using $A\_Acc$, $O\_Acc$, and $F\_Acc$, arguing that continual pre-training should preserve not only current-domain adaptability and old-domain retention but also generalization to unseen domains. Its hypernetwork prompt method reports improvements over DAS on both DAPset and TWEET, including 1.38% in generalization and 1.09% in final performance on DAPset (Jiang et al., 2023).

6. Empirical behavior across domains and modalities

The empirical record is heterogeneous but broadly favorable. Continued pre-training improves clinical dialogue comprehension, low-resource Urdu adaptation, finance benchmarks, materials NLP, pMHC-I binding prediction, and long-context mental-health classification, while also showing that the magnitude and location of gains depend strongly on data regime and task structure.

Setting	Continued pre-training design	Reported result
Clinical dialogue	RoBERTa-base with speaker/utterance-aware denoising	92.31 EM / 93.69 F1 vs 88.37 EM / 90.15 F1 without domain PT
Urdu LLM	LLaMA-3.1 8B continued on 1.97 billion tokens, then SFT	Weighted average 90.34 vs 87.1 for Alif-1.0-Instruct and 45.7 for base LLaMA-3.1 8B-Instruct
Finance LLM	Pythia continual pre-training on finance corpus	FinPythia-6.9B average F1 54.83 vs 50.64 for Pythia-7B
Materials science	SciBERT adapted with MELT	Overall 0.741 Micro-F1 / 0.556 Macro-F1
Protein LM	ESM Cambrian continued on HLA-associated peptides	Median Spearman $\rho = 0.62$ across 25 common HLA alleles
Mental health long context	Longformer continued on mental-health Reddit corpus	76.32 F1 on CLPsych15; 72.47 F1 on UMD at 4096 tokens

In clinical dialogue, the strongest improvements occur when the pretraining corruption reflects the downstream discourse structure and when downstream supervision is scarce; the paper reports especially large gains below 20k training examples (Liu et al., 2022). In Urdu, the principal claim is that instruction tuning alone is not enough, and that substantial Urdu exposure in pretraining is the decisive step for competence in morphology, script, and domain diversity (Hassan et al., 13 Jan 2026). Finance work emphasizes cost-effectiveness: continual pre-training improves specialist performance, but selective data selection can be even more efficient than naive full-corpus continuation (Xie et al., 2023).

Scientific and biomedical settings show a similar pattern with stronger domain-aware objectives. MELT’s gains are attributed to better masking of materials entities and a curriculum from general to specialized concepts (Kim et al., 2024). For protein LLMs, continued pre-training helps most in the 500–1000 and 1000–2000 peptide regimes, where PT E 30L improves Spearman correlation from 0.378 to 0.480 and from 0.459 to 0.497, respectively; for the scarcest alleles, however, direct fine-tuning can still be better (Mares et al., 16 Jul 2025). Long-context mental-health modeling shows that the interaction between domain adaptation and architecture matters: MentalLongformer improves as context length increases from 512 to 4096 tokens and substantially exceeds earlier 512-token mental-health models on long-document datasets (Ji et al., 2023).

7. Task dependence, limits, and adjacent extensions

The accumulated evidence does not support a one-size-fits-all rule. Several papers explicitly show that continued pre-training is beneficial but not universally dominant. In German medical NLP, ChristBERT converges fastest and reaches roughly 3–4 perplexity by 10k steps, but downstream results are task-dependent: continued pre-training is best on the hardest NER benchmark GGPONC with 77.69 F1, whereas training from scratch or domain-specific BPE can be stronger on classification or terminology-heavy extraction tasks (He et al., 2 Jun 2026). The paper’s stated conclusion is that the optimal adaptation strategy is task-dependent.

Limits also appear within the same domain. The protein-language-model study states plainly that continued domain-specific pre-training is not a cure-all for the rarest alleles; benefits drop sharply for alleles with <500 peptides (Mares et al., 16 Jul 2025). The insight-learning study shows that continual pre-training on raw original documents has only marginal impact on deeper insight acquisition, while simplified documents or triple-only inputs help much more; probabilistic insight remains difficult even after adaptation (Pezeshkpour et al., 29 Jan 2025). In finance, more data is not always better: the paper notes that full 100% DACP can be slightly worse on open-domain tasks than smaller, better-selected subsets (Xie et al., 2023).

The scope of the concept has also broadened beyond ordinary text LMs. Difference-Masking includes multimodal video tasks (Wilf et al., 2023). Continued pre-training of protein LLMs adapts sequence models to immunological binding prediction (Mares et al., 16 Jul 2025). An adjacent but distinct example appears in whole-slide pathology: the WSI study evaluates whether a histopathology-pretrained encoder, KimiaNet, is a better frozen patch feature extractor than ImageNet-pretrained encoders for MIL-based glioma classification. That work supports domain-specific pretraining as a representation strategy and shows more consistent gains in confidence than in accuracy or AUC, but it does not perform new continued pre-training of the downstream MIL model itself, and some gains may be entangled with backbone architecture differences (Chitnis et al., 2023).

Taken together, these results support a constrained synthesis. Continued domain-specific pre-training is most effective when the downstream domain is distributionally distinct, labels are scarce or expensive, and the adaptation objective is aligned with domain structure. Its benefits are amplified by careful corpus design, mixture control, and structure-aware masking or prompting. Its limits become more visible when domain supervision is extremely sparse, when raw documents contain substantial irrelevant text, or when the downstream task rewards representations that training from scratch or tokenizer redesign captures better.