---
title: Domain Adaptive Continual Pretraining (DACP)
url: https://www.emergentmind.com/topics/domain-adaptive-continual-pretraining-dacp
type: topic
---

# Domain Adaptive Continual Pretraining (DACP)

Domain Adaptive Continual Pretraining (DACP) is the practice of taking a general-purpose pretrained model and continuing pretraining on unlabeled text, code, images, video, or other data from a target domain so that the resulting model better matches the target distribution before downstream fine-tuning or deployment. Across the literature, DACP is consistently characterized by a staged pipeline: a broad source-domain checkpoint, a second self-supervised adaptation stage on in-domain corpora, and then either supervised fine-tuning, post-training, or direct transfer to downstream tasks. The central technical questions are whether continued pretraining improves in-domain performance, how much general capability is retained, what pretraining objective should be used during adaptation, and how data selection, replay, parameter efficiency, and domain structure affect forgetting and transfer [2105.13665] [2302.03241] [2402.17400].

## 1. Formalization and scope

In its canonical form, DACP begins with a pretrained model \(M_0\) trained on a broad corpus \(D_0\), then exposes the model to one or more target-domain corpora \(D_1, D_2, \ldots\), using the model’s original pretraining objective rather than a supervised task loss. In the sequential version studied in continual-learning work, once a domain corpus has been used, it is no longer accessible, making forgetting a first-class concern. This distinguishes DACP both from one-shot domain-adaptive pretraining, which adapts to only one domain, and from downstream fine-tuning, which optimizes directly for labeled end tasks rather than for self-supervised domain adaptation [2302.03241] [2402.17400].

The operational form of DACP varies by modality but preserves the same logic. In dialogue understanding, the model starts from BERT pretrained on sources such as Wikipedia and BooksCorpus, then undergoes continued pretraining on dialogue data before fine-tuning for conversational semantic role labeling and spoken language understanding [2105.13665]. In Chinese biomedical NLP, RoBERTa-wwm-ext-base is continually pretrained on biomedical corpora to build RoBERTa-Med [2211.11363]. In industrial language modeling, small autoregressive LLMs are continually pretrained on mixtures of domain text and replay corpora before instruction tuning and service deployment [2507.06795]. In geospatial vision, a large ImageNet-22k model is continually pretrained on GeoPile to produce a geospatial foundation model [2302.04476]. In video understanding, a pretrained V-JEPA model is continually pretrained on unlabeled primate video before behavior classification [2509.12193].

A broader implication of this literature is that DACP is not tied to a single architecture family, objective family, or modality. It has been instantiated with encoder-only masked pretraining, decoder-only causal language modeling, teacher-student masked image modeling, masked latent video prediction, and instruction-formatted self-supervision. This suggests that DACP is best understood as a training regime defined by where the additional self-supervised signal comes from—namely the target domain—rather than by any single loss or model class [2510.20377] [2302.04476] [2509.12193].

## 2. Objectives and adaptation mechanisms

The simplest DACP objective is the model’s original self-supervised pretraining loss. For encoder models, this commonly means masked language modeling. In dialogue understanding, standard MLM selects \(15\%\) of tokens for corruption, with BERT-style \(80/10/10\) masking, and the loss is written as
\[
\mathcal{L}_{MLM} = - \frac{1}{|Y|} \sum_{t=1}^{|Y|} \log p(x_t \mid \mathbf{h}_t; \mathbf{\theta}).
\]
That study also evaluates Span Boundary Objective (SBO), which masks contiguous spans and predicts span tokens from boundary representations, and a dialogue-specific Perturbation Masking Objective (PMO) that encourages stronger modeled dependence between predicates and masked argument spans [2105.13665].

One major theme in DACP research is that objective choice materially affects outcomes. The dialogue paper shows that plain MLM improves a strong baseline, but span-aware and predicate-argument-aware objectives improve more, especially on cross-turn conversational semantic role labeling. The sequential continual-pretraining paper proposes DAS, which adds unit-level soft masking and a contrastive integration term to MLM. Its training objective is
\[
\mathcal{L}_{\text{DAP-train}} = \mathcal{L}_{\text{MLM}} + \lambda \mathcal{L}_{\text{contrast}},
\]
with gradients modulated by accumulated unit importance
\[
\hat{\nabla}_{l} = (1-\bm{I}_{l}^{(\le t-1)}) \otimes \nabla_{l}.
\]
The same work also introduces a proxy KL-based initialization to preserve general knowledge in the original pretrained LM without access to \(D_0\) [2302.03241].

For decoder-only LLMs, DACP typically uses causal language modeling. Industrial small-LLM adaptation is described as continued autoregressive pretraining on mixtures of domain and replay data, with the implied loss
\[
\mathcal{L}_{\mathrm{DACP}} = -\sum_{t=1}^{T}\log p_{\theta}(x_t \mid x_{<t}),
\]
and a later post-training stage to restore instruction following [2507.06795]. Embedded-systems code adaptation likewise continues causal LM training, but in a parameter-efficient form using LoRA with rank-stabilized scaling. The paper writes standard LoRA as
\[
W' = W_0 + \frac{\alpha}{r}BA
\]
and RSLoRA as
\[
W' = W_0 + \frac{\alpha}{\sqrt{r}}BA,
\]
arguing that the latter is necessary for high-rank adaptation in a highly specialized code domain [2603.11139].

A newer line of work reformulates DACP for instruction-tuned models. IKnow argues that naive continued pretraining with ordinary self-supervised objectives degrades instruction-following capability, and instead converts unlabeled domain text into instruction-response examples. Its core tasks are Masked Token Prediction, Masked Phrase Prediction, and bidirectional NL\(\leftrightarrow\)KG transformation. The phrase and tuple sets are extracted as
\[
P_s = \{ p_i \mid p_i \in const(s)\}, \qquad
KG_s = \{(s, r, v) \in dep(s)\},
\]
and the model is adapted in dialogue format rather than raw next-token format [2510.20377]. A more radical extension is DACIP-RC, which departs from next-token prediction entirely and constructs large-scale synthetic instruction-response corpora from business transcripts through reading-comprehension prompting; the paper explicitly frames this as domain-adaptive continual instruction pre-training rather than conventional DACP [2510.08152].

## 3. Data curation, replay, retrieval, and tokenization

DACP is unusually sensitive to corpus construction because the adaptation stage is entirely defined by what counts as “in-domain” data. Some studies deliberately use only the unlabeled training split of the downstream task to isolate objective effects. In dialogue understanding, the adaptation corpora are the target-task training dialogues themselves: 26K DuConv sessions, 20K NewsDialog sessions, and the full CrossWOZ training set, with no extra external in-domain corpus [2105.13665]. In continual biomedical pretraining, the corpus combines biomedical QA text, medical encyclopedia text, and electronic medical records, totaling about 7.397 million sentences and 171 million tokens [2211.11363].

Other papers treat corpus construction as a central contribution. In industrial telco DACP, the adaptation mixture is 140GB telco corpora, 100GB Korean corpora, 50GB English corpora, and 15GB code, totaling 305GB; replay is explicitly introduced to reduce forgetting, and a 50% replay ratio is chosen as the best retention-specialization balance in a reduced experiment [2507.06795]. In finance, the analogous mixture totals 53GB, again with a substantial replay component [2507.06795]. In embedded systems, the training corpus is built from repository-datasheet pairs via a four-stage SpecMap pipeline, then filtered and chunked into a curated dataset of about 23.5B tokens across 13 embedded domains [2603.11139]. In geospatial vision, GeoPile is intentionally compact but diverse, combining about 600k RGB overhead images from NAIP, RSD46-WHU, MLRSNet, RESISC45, and PatternNet, rather than relying only on larger but lower-entropy Sentinel-2 collections [2302.04476].

Several papers propose making DACP more efficient by selecting or constructing higher-yield adaptation corpora. TextGram uses a small in-domain corpus to retrieve a compact pseudo in-domain subset from a large candidate corpus before continued MLM pretraining. Its pipeline combines top in-domain bigrams with a sentence-similarity graph and PageRank-style ranking, reducing RealNews from 1M items to a selected 0.25M subset while slightly improving IMDb downstream performance relative to using all RealNews [2404.18228]. ICL-APT for German process-industry logs takes a different route: instead of pretraining on a broad domain-related corpus, it retrieves semantically similar in-domain and domain-related neighbors for each target document, concatenates them into context-augmented sequences, and then runs standard MLM on those augmented sequences [2504.19856].

Tokenizer adaptation has also been proposed as part of DACP rather than as a fixed preprocessing assumption. IGOT argues that ordinary continued pretraining is suboptimal when the original tokenizer fragments specialized terms. It defines a heuristic information-gain score
\[
\theta =  \log(1+\sum_{i\in \Delta}N_{\delta_i})-\log(1+\alpha),
\]
constructs a candidate set
\[
\lambda = \{\theta_n>\epsilon, \ n\in \Delta \},
\]
and refines it through a learned scorer
\[
S_\lambda = \{ \phi(\theta_n;\ n;\ \Delta)> \epsilon', n \in \Delta \}.
\]
On OpenLane documentation, this yields reported token savings, training-time savings, and lower VRAM usage during continued pretraining [2405.09857].

## 4. Empirical behavior across domains and modalities

The empirical literature largely agrees that DACP can substantially improve in-domain performance, but the magnitude depends on task structure, domain shift, and the adaptation objective.

| Domain | Representative study | Main empirical pattern |
|---|---|---|
| Dialogue understanding | [2105.13665] | Continued pretraining improves CSRL and SLU; objective design matters |
| Chinese biomedical NLP | [2211.11363] | Parameter-efficient continual pretraining improves CBLUE average and reduces forgetting versus full fine-tuning |
| Enterprise telco and finance | [2507.06795] | DACP yields large in-domain gains with modest general degradation, especially for sLLMs |
| Embedded systems code | [2603.11139] | In-domain and held-out perplexity drop sharply after domain CPT |
| Geospatial vision | [2302.04476] | Teacher-guided continual pretraining outperforms direct ImageNet transfer and from-scratch geospatial pretraining |
| Primate behavior video | [2509.12193] | Most of the downstream gain comes from the DAP stage |

The dialogue study is a canonical DACP result because it holds downstream architecture fixed and swaps only the encoder checkpoint. On DuConv CSRL, the no-adaptation baseline reports **88.16 F1\(_{all}\)** and **83.74 F1\(_{cross}\)**, while the strongest noun-phrase-biased MLM+SBO+PMO setup reaches **89.97 F1\(_{all}\)** and **86.68 F1\(_{cross}\)**. On NewsDialog CSRL, the baseline **76.81 F1\(_{all}\)** rises to **81.90** under noun-phrase-biased sampling, and the best SLU setting on CrossWOZ reaches **96.93 F1\(_{all}\)** from a **95.34** baseline [2105.13665].

In Chinese biomedical modeling, AF Adapter shows that a parameter-efficient DACP regime can outperform full-parameter continual pretraining. RoBERTa-Med reaches a **69.904** CBLUE average compared with **69.3** for the RoBERTa-wwm-ext-base backbone and **67.9** for PCL-MedBERT; in the method comparison, AF Adapter at **21M trainable parameters (17.21%)** outperforms full Fine-Tuning at **65.560**, FL-Tuning at **69.275**, and LoRA at **64.338** [2211.11363].

The industrial small-LLM study reports some of the largest domain gains. In the telco benchmark, Llama 3.2 3B IT rises from **47.97** to **72.38** average, EXAONE 3.5 2.4B IT rises from **42.30** to **70.94**, and EXAONE 3.5 32B IT rises from **49.82** to **84.43** after DACP. In finance, EXAONE 3.5 7.8B rises from **45.53** to **59.54**, and this finance-adapted 7.8B model exceeds much larger untailored 32B–72B models on the finance benchmark while keeping general average nearly unchanged, from **60.10** to **59.56** [2507.06795].

Embedded-systems CPT provides a strong code-domain example. Weighted in-domain perplexity drops from **4.06** to **1.20**, a **70.4% reduction**, while held-out perplexity drops from **3.92** to **1.33**, a **66.1% reduction**. On free-form generative code completion, the adapted 7B model outperforms Claude Opus 4.6 and Qwen3-Coder-30B on **8 of 13 categories** in token accuracy [2603.11139]. In geospatial vision, the proposed GFM continual-pretraining method reaches **ARP 3.31**, outperforming direct ImageNet-22k transfer, GeoPile pretraining from scratch, and vanilla continual pretraining, while using **93.3 V100 GPU hours** and **13.3 kg CO\(_2\)e** [2302.04476]. In video action recognition, V-JEPA with DAP improves PanAf500 Top-1 accuracy from **83.68** to **87.24** and ChimpACT mAP from **26.0** to **30.7**, with the authors stating that most of the gain stems from DAP [2509.12193].

The literature also contains negative or qualified results. In Taiwanese legal reasoning, DACP is reported to enhance domain-specific knowledge but not to uniformly improve performance across all legal tasks. On some multiple-choice and prompt-based evaluations, DACP-based models remain below strong general instruction baselines, while improvement is more visible on a legal symposium argumentation task [2504.13603]. This suggests that domain knowledge internalization and downstream reasoning competence are not interchangeable.

## 5. Forgetting, retention, and continual-learning trade-offs

Catastrophic forgetting is one of the defining technical issues in DACP. In the Chinese biomedical setting, full-parameter continual pretraining degrades masked-token prediction accuracy on held-out general-domain text from **87.752** to **70.964**, a difference of **\(-16.788\)**, whereas AF Adapter drops only to **82.428**, i.e. **\(-5.324\)**, which the paper summarizes as alleviating the forgetting problem by **11%** compared to the fine-tuning method [2211.11363]. In industrial small-LLM adaptation, replay is the main retention mechanism: increasing replay from 5% to 50% improves general average in the pilot experiment from **50.14** to **53.80**, close to the base model’s **54.44**, but Telco average drops from **60.91** at 5% replay to **55.65** at 50% and **48.67** at 75%, making the replay ratio itself a retention-specialization control [2507.06795].

Sequential continual-pretraining work provides the clearest formal analysis of forgetting. Over **159 domains**, continual pretraining in semantically similar order improves current-domain specialization relative to isolated per-domain adaptation, but the final checkpoint suffers substantial forgetting of earlier domains. The paper formalizes forgetting as
\[
p_n^{n+x} - p_n^n,
\]
and shows that later stages of continual learning are more prone to forgetting than earlier stages, describing this as knowledge saturation. It also reports that semantically ordered curricula help immediate specialization, whereas randomized order improves average backward transfer and final performance across learned domains [2402.17400]. DAS addresses this regime directly through unit-level importance estimation, backward-only soft gradient masking, and contrastive integration, achieving **77.93 MF1 / 81.91 Acc** average and a **negative forgetting rate** of **-1.09 MF1 / -0.60 Acc**, which the paper interprets as positive backward transfer [2302.03241].

Different modalities exhibit different anti-forgetting mechanisms. In geospatial vision, the retention device is not replay but a frozen ImageNet-22k teacher; the continual-pretraining loss is
\[
\mathcal{L} = \mathcal{L}_{MIM} + \mathcal{L}_{feat},
\]
where
\[
\mathcal{L}_{MIM}=\frac{\left\|\mathbf{O}_\kappa-\mathbf{G}_\kappa\right\|_1}{N}
\]
and \(\mathcal{L}_{feat}\) aligns student and teacher features by cosine similarity [2302.04476]. In embedded-systems code, the paper argues that high-capacity PEFT with RSLoRA is enough to keep degradation on FullStackBench C++ modest, with compile rate moving from **44.9%** to **41.1%** and test pass rate from **15.9%** to **13.1%**, which it describes as minimal degradation rather than catastrophic forgetting [2603.11139]. In instruction-tuned LLM adaptation, IKnow’s central claim is that naive next-token continued pretraining can catastrophically damage instruction-following, as illustrated by a RepliQA score collapse to **0.2** for Llama-3.2-3B-Instruct under full NTP adaptation, while instruction-formatted MTP, MPP, and NLKG objectives avoid that failure [2510.20377].

The literature therefore does not support a single universal anti-forgetting recipe. Replay, soft masking, frozen teachers, parameter isolation, internal expansion, and instruction-compatible self-supervision all appear, and their suitability depends on whether the model is encoder-only, decoder-only, already instruction tuned, or adapted under resource constraints.

## 6. Limitations, disagreements, and future directions

DACP research is empirically strong but methodologically heterogeneous, and several recurring limitations appear across the literature. Many studies are single-step or single-domain, not multi-stage lifelong adaptation. The dialogue paper explicitly does not examine multi-domain continual adaptation, catastrophic forgetting of general-domain knowledge, larger adaptation corpora, or longer schedules [2105.13665]. The biomedical adapter paper is specific to Chinese biomedical text and BERT/RoBERTa-style encoders, does not compare vocabulary redesign, and measures forgetting only through general-domain MLM accuracy rather than downstream tasks [2211.11363]. Industrial studies often rely on in-house benchmarks and partially specified post-training details, which makes it difficult to isolate the effect of DACP from later alignment or service-specific fine-tuning [2507.06795] [2510.08152] [2510.05858].

Several substantive disagreements also remain open. One concerns whether replay is necessary. Industrial small-LLM work presents replay as the main anti-forgetting mechanism and selects around **50% replay** as the best balance [2507.06795], whereas the embedded-systems code paper reports that domain-only training outperformed mixed domain-plus-general-code mixtures and argues that LoRA already mitigates forgetting [2603.11139]. Another concerns whether DACP uniformly improves downstream reasoning. The legal adaptation study argues that DACP improves legal-domain knowledge but does not reliably improve prompt-based legal reasoning and can degrade some exam-style tasks [2504.13603]. A related disagreement concerns what counts as the correct adaptation signal: some papers stay close to the original pretraining objective, whereas DACIP-RC and IKnow argue that instruction-tuned or enterprise-oriented settings may require instruction-formatted or task-synthetic self-supervision rather than raw next-token prediction [2510.08152] [2510.20377].

Future directions suggested by the literature are concrete. One is stronger control of what the model is exposed to during DACP: data selection, retrieval augmentation, phrase-aware masking, and tokenizer adaptation all indicate that corpus design can matter as much as corpus size [2404.18228] [2504.19856] [2405.09857]. Another is more realistic continual evaluation over long domain sequences, where order, similarity, and saturation effects become visible [2402.17400]. A third is adaptation of already instruction-tuned checkpoints without access to the base model, which motivates instruction-compatible continual objectives [2510.20377]. A fourth is stronger analysis of computational efficiency, especially for small or medium models that must operate under industrial resource constraints [2507.06795] [2504.19856].

Taken together, the literature supports a concise but qualified characterization of DACP. Continued in-domain pretraining usually helps, often substantially, and can be effective even with only the target task’s own unlabeled data. However, the benefit is not solely a function of “more in-domain text.” It depends on the match between objective and downstream structure, the construction and mixing of corpora, the retention mechanism, the model’s initialization state, and the evaluation regime. In that sense, DACP has evolved from a simple continuation of pretraining into a broader design space of objective engineering, corpus engineering, and continual-learning control.

Source: https://www.emergentmind.com/topics/domain-adaptive-continual-pretraining-dacp