Domain Adaptive Continual Pretraining (DACP)

• Domain Adaptive Continual Pretraining (DACP) is a training regime that adapts a general pretrained model to a target domain using self-supervised learning on specialized, unlabeled data.
• It employs a multi-stage pipeline—starting with a broad-domain checkpoint followed by in-domain adaptation—to balance improved specialization and retention of general capabilities.
• Empirical studies across modalities demonstrate that carefully engineered DACP techniques enhance downstream performance while mitigating catastrophic forgetting.

Domain Adaptive Continual Pretraining (DACP) is the practice of taking a general-purpose pretrained model and continuing pretraining on unlabeled text, code, images, video, or other data from a target domain so that the resulting model better matches the target distribution before downstream fine-tuning or deployment. Across the literature, DACP is consistently characterized by a staged pipeline: a broad source-domain checkpoint, a second self-supervised adaptation stage on in-domain corpora, and then either supervised fine-tuning, post-training, or direct transfer to downstream tasks. The central technical questions are whether continued pretraining improves in-domain performance, how much general capability is retained, what pretraining objective should be used during adaptation, and how data selection, replay, parameter efficiency, and domain structure affect forgetting and transfer (Wu et al., 2021, Ke et al., 2023, Yıldız et al., 2024).

1. Formalization and scope

In its canonical form, DACP begins with a pretrained model $M_0$ trained on a broad corpus $D_0$, then exposes the model to one or more target-domain corpora $D_1, D_2, \ldots$, using the model’s original pretraining objective rather than a supervised task loss. In the sequential version studied in continual-learning work, once a domain corpus has been used, it is no longer accessible, making forgetting a first-class concern. This distinguishes DACP both from one-shot domain-adaptive pretraining, which adapts to only one domain, and from downstream fine-tuning, which optimizes directly for labeled end tasks rather than for self-supervised domain adaptation (Ke et al., 2023, Yıldız et al., 2024).

The operational form of DACP varies by modality but preserves the same logic. In dialogue understanding, the model starts from BERT pretrained on sources such as Wikipedia and BooksCorpus, then undergoes continued pretraining on dialogue data before fine-tuning for conversational semantic role labeling and spoken language understanding (Wu et al., 2021). In Chinese biomedical NLP, RoBERTa-wwm-ext-base is continually pretrained on biomedical corpora to build RoBERTa-Med (Yan et al., 2022). In industrial language modeling, small autoregressive LLMs are continually pretrained on mixtures of domain text and replay corpora before instruction tuning and service deployment (Kim et al., 9 Jul 2025). In geospatial vision, a large ImageNet-22k model is continually pretrained on GeoPile to produce a geospatial foundation model (Mendieta et al., 2023). In video understanding, a pretrained V-JEPA model is continually pretrained on unlabeled primate video before behavior classification (Mueller et al., 15 Sep 2025).

A broader implication of this literature is that DACP is not tied to a single architecture family, objective family, or modality. It has been instantiated with encoder-only masked pretraining, decoder-only causal language modeling, teacher-student masked image modeling, masked latent video prediction, and instruction-formatted self-supervision. This suggests that DACP is best understood as a training regime defined by where the additional self-supervised signal comes from—namely the target domain—rather than by any single loss or model class (Zhang et al., 23 Oct 2025, Mendieta et al., 2023, Mueller et al., 15 Sep 2025).

2. Objectives and adaptation mechanisms

The simplest DACP objective is the model’s original self-supervised pretraining loss. For encoder models, this commonly means masked language modeling. In dialogue understanding, standard MLM selects $15\%$ of tokens for corruption, with BERT-style $80/10/10$ masking, and the loss is written as

$$\mathcal{L}_{MLM} = - \frac{1}{|Y|} \sum_{t=1}^{|Y|} \log p(x_t \mid \mathbf{h}_t; \mathbf{\theta}).$$

That study also evaluates Span Boundary Objective (SBO), which masks contiguous spans and predicts span tokens from boundary representations, and a dialogue-specific Perturbation Masking Objective (PMO) that encourages stronger modeled dependence between predicates and masked argument spans (Wu et al., 2021).

One major theme in DACP research is that objective choice materially affects outcomes. The dialogue paper shows that plain MLM improves a strong baseline, but span-aware and predicate-argument-aware objectives improve more, especially on cross-turn conversational semantic role labeling. The sequential continual-pretraining paper proposes DAS, which adds unit-level soft masking and a contrastive integration term to MLM. Its training objective is

$$\mathcal{L}_{\text{DAP-train}} = \mathcal{L}_{\text{MLM}} + \lambda \mathcal{L}_{\text{contrast}},$$

with gradients modulated by accumulated unit importance

$$\hat{\nabla}_{l} = (1-\bm{I}_{l}^{(\le t-1)}) \otimes \nabla_{l}.$$

The same work also introduces a proxy KL-based initialization to preserve general knowledge in the original pretrained LM without access to $D_0$ (Ke et al., 2023).

For decoder-only LLMs, DACP typically uses causal language modeling. Industrial small-LLM adaptation is described as continued autoregressive pretraining on mixtures of domain and replay data, with the implied loss

$$\mathcal{L}_{\mathrm{DACP}} = -\sum_{t=1}^{T}\log p_{\theta}(x_t \mid x_{<t}),$$

and a later post-training stage to restore instruction following (Kim et al., 9 Jul 2025). Embedded-systems code adaptation likewise continues causal LM training, but in a parameter-efficient form using LoRA with rank-stabilized scaling. The paper writes standard LoRA as

$D_0$0

and RSLoRA as

$D_0$1

arguing that the latter is necessary for high-rank adaptation in a highly specialized code domain (Singh et al., 11 Mar 2026).

A newer line of work reformulates DACP for instruction-tuned models. IKnow argues that naive continued pretraining with ordinary self-supervised objectives degrades instruction-following capability, and instead converts unlabeled domain text into instruction-response examples. Its core tasks are Masked Token Prediction, Masked Phrase Prediction, and bidirectional NL$D_0$2KG transformation. The phrase and tuple sets are extracted as

$D_0$3

and the model is adapted in dialogue format rather than raw next-token format (Zhang et al., 23 Oct 2025). A more radical extension is DACIP-RC, which departs from next-token prediction entirely and constructs large-scale synthetic instruction-response corpora from business transcripts through reading-comprehension prompting; the paper explicitly frames this as domain-adaptive continual instruction pre-training rather than conventional DACP (Khasanova et al., 9 Oct 2025).

3. Data curation, replay, retrieval, and tokenization

DACP is unusually sensitive to corpus construction because the adaptation stage is entirely defined by what counts as “in-domain” data. Some studies deliberately use only the unlabeled training split of the downstream task to isolate objective effects. In dialogue understanding, the adaptation corpora are the target-task training dialogues themselves: 26K DuConv sessions, 20K NewsDialog sessions, and the full CrossWOZ training set, with no extra external in-domain corpus (Wu et al., 2021). In continual biomedical pretraining, the corpus combines biomedical QA text, medical encyclopedia text, and electronic medical records, totaling about 7.397 million sentences and 171 million tokens (Yan et al., 2022).

Other papers treat corpus construction as a central contribution. In industrial telco DACP, the adaptation mixture is 140GB telco corpora, 100GB Korean corpora, 50GB English corpora, and 15GB code, totaling 305GB; replay is explicitly introduced to reduce forgetting, and a 50% replay ratio is chosen as the best retention-specialization balance in a reduced experiment (Kim et al., 9 Jul 2025). In finance, the analogous mixture totals 53GB, again with a substantial replay component (Kim et al., 9 Jul 2025). In embedded systems, the training corpus is built from repository-datasheet pairs via a four-stage SpecMap pipeline, then filtered and chunked into a curated dataset of about 23.5B tokens across 13 embedded domains (Singh et al., 11 Mar 2026). In geospatial vision, GeoPile is intentionally compact but diverse, combining about 600k RGB overhead images from NAIP, RSD46-WHU, MLRSNet, RESISC45, and PatternNet, rather than relying only on larger but lower-entropy Sentinel-2 collections (Mendieta et al., 2023).

Several papers propose making DACP more efficient by selecting or constructing higher-yield adaptation corpora. TextGram uses a small in-domain corpus to retrieve a compact pseudo in-domain subset from a large candidate corpus before continued MLM pretraining. Its pipeline combines top in-domain bigrams with a sentence-similarity graph and PageRank-style ranking, reducing RealNews from 1M items to a selected 0.25M subset while slightly improving IMDb downstream performance relative to using all RealNews (Hiwarkhedkar et al., 2024). ICL-APT for German process-industry logs takes a different route: instead of pretraining on a broad domain-related corpus, it retrieves semantically similar in-domain and domain-related neighbors for each target document, concatenates them into context-augmented sequences, and then runs standard MLM on those augmented sequences (Zhukova et al., 28 Apr 2025).

Tokenizer adaptation has also been proposed as part of DACP rather than as a fixed preprocessing assumption. IGOT argues that ordinary continued pretraining is suboptimal when the original tokenizer fragments specialized terms. It defines a heuristic information-gain score

$D_0$4

constructs a candidate set

$D_0$5

and refines it through a learned scorer

$D_0$6

On OpenLane documentation, this yields reported token savings, training-time savings, and lower VRAM usage during continued pretraining (Feng et al., 2024).

4. Empirical behavior across domains and modalities

The empirical literature largely agrees that DACP can substantially improve in-domain performance, but the magnitude depends on task structure, domain shift, and the adaptation objective.

Domain	Representative study	Main empirical pattern
Dialogue understanding	(Wu et al., 2021)	Continued pretraining improves CSRL and SLU; objective design matters
Chinese biomedical NLP	(Yan et al., 2022)	Parameter-efficient continual pretraining improves CBLUE average and reduces forgetting versus full fine-tuning
Enterprise telco and finance	(Kim et al., 9 Jul 2025)	DACP yields large in-domain gains with modest general degradation, especially for sLLMs
Embedded systems code	(Singh et al., 11 Mar 2026)	In-domain and held-out perplexity drop sharply after domain CPT
Geospatial vision	(Mendieta et al., 2023)	Teacher-guided continual pretraining outperforms direct ImageNet transfer and from-scratch geospatial pretraining
Primate behavior video	(Mueller et al., 15 Sep 2025)	Most of the downstream gain comes from the DAP stage

The dialogue study is a canonical DACP result because it holds downstream architecture fixed and swaps only the encoder checkpoint. On DuConv CSRL, the no-adaptation baseline reports 88.16 F1$D_0$7 and 83.74 F1$D_0$8, while the strongest noun-phrase-biased MLM+SBO+PMO setup reaches 89.97 F1$D_0$9 and 86.68 F1$D_1, D_2, \ldots$0. On NewsDialog CSRL, the baseline 76.81 F1$D_1, D_2, \ldots$1 rises to 81.90 under noun-phrase-biased sampling, and the best SLU setting on CrossWOZ reaches 96.93 F1$D_1, D_2, \ldots$2 from a 95.34 baseline (Wu et al., 2021).

In Chinese biomedical modeling, AF Adapter shows that a parameter-efficient DACP regime can outperform full-parameter continual pretraining. RoBERTa-Med reaches a 69.904 CBLUE average compared with 69.3 for the RoBERTa-wwm-ext-base backbone and 67.9 for PCL-MedBERT; in the method comparison, AF Adapter at 21M trainable parameters (17.21%) outperforms full Fine-Tuning at 65.560, FL-Tuning at 69.275, and LoRA at 64.338 (Yan et al., 2022).

The industrial small-LLM study reports some of the largest domain gains. In the telco benchmark, Llama 3.2 3B IT rises from 47.97 to 72.38 average, EXAONE 3.5 2.4B IT rises from 42.30 to 70.94, and EXAONE 3.5 32B IT rises from 49.82 to 84.43 after DACP. In finance, EXAONE 3.5 7.8B rises from 45.53 to 59.54, and this finance-adapted 7.8B model exceeds much larger untailored 32B–72B models on the finance benchmark while keeping general average nearly unchanged, from 60.10 to 59.56 (Kim et al., 9 Jul 2025).

Embedded-systems CPT provides a strong code-domain example. Weighted in-domain perplexity drops from 4.06 to 1.20, a 70.4% reduction, while held-out perplexity drops from 3.92 to 1.33, a 66.1% reduction. On free-form generative code completion, the adapted 7B model outperforms Claude Opus 4.6 and Qwen3-Coder-30B on 8 of 13 categories in token accuracy (Singh et al., 11 Mar 2026). In geospatial vision, the proposed GFM continual-pretraining method reaches ARP 3.31, outperforming direct ImageNet-22k transfer, GeoPile pretraining from scratch, and vanilla continual pretraining, while using 93.3 V100 GPU hours and 13.3 kg CO$D_1, D_2, \ldots$3e (Mendieta et al., 2023). In video action recognition, V-JEPA with DAP improves PanAf500 Top-1 accuracy from 83.68 to 87.24 and ChimpACT mAP from 26.0 to 30.7, with the authors stating that most of the gain stems from DAP (Mueller et al., 15 Sep 2025).

The literature also contains negative or qualified results. In Taiwanese legal reasoning, DACP is reported to enhance domain-specific knowledge but not to uniformly improve performance across all legal tasks. On some multiple-choice and prompt-based evaluations, DACP-based models remain below strong general instruction baselines, while improvement is more visible on a legal symposium argumentation task (Chen et al., 18 Apr 2025). This suggests that domain knowledge internalization and downstream reasoning competence are not interchangeable.

5. Forgetting, retention, and continual-learning trade-offs

Catastrophic forgetting is one of the defining technical issues in DACP. In the Chinese biomedical setting, full-parameter continual pretraining degrades masked-token prediction accuracy on held-out general-domain text from 87.752 to 70.964, a difference of $D_1, D_2, \ldots$4, whereas AF Adapter drops only to 82.428, i.e. $D_1, D_2, \ldots$5, which the paper summarizes as alleviating the forgetting problem by 11% compared to the fine-tuning method (Yan et al., 2022). In industrial small-LLM adaptation, replay is the main retention mechanism: increasing replay from 5% to 50% improves general average in the pilot experiment from 50.14 to 53.80, close to the base model’s 54.44, but Telco average drops from 60.91 at 5% replay to 55.65 at 50% and 48.67 at 75%, making the replay ratio itself a retention-specialization control (Kim et al., 9 Jul 2025).

Sequential continual-pretraining work provides the clearest formal analysis of forgetting. Over 159 domains, continual pretraining in semantically similar order improves current-domain specialization relative to isolated per-domain adaptation, but the final checkpoint suffers substantial forgetting of earlier domains. The paper formalizes forgetting as

$D_1, D_2, \ldots$6

and shows that later stages of continual learning are more prone to forgetting than earlier stages, describing this as knowledge saturation. It also reports that semantically ordered curricula help immediate specialization, whereas randomized order improves average backward transfer and final performance across learned domains (Yıldız et al., 2024). DAS addresses this regime directly through unit-level importance estimation, backward-only soft gradient masking, and contrastive integration, achieving 77.93 MF1 / 81.91 Acc average and a negative forgetting rate of -1.09 MF1 / -0.60 Acc, which the paper interprets as positive backward transfer (Ke et al., 2023).

Different modalities exhibit different anti-forgetting mechanisms. In geospatial vision, the retention device is not replay but a frozen ImageNet-22k teacher; the continual-pretraining loss is

$D_1, D_2, \ldots$7

where

$D_1, D_2, \ldots$8

and $D_1, D_2, \ldots$9 aligns student and teacher features by cosine similarity (Mendieta et al., 2023). In embedded-systems code, the paper argues that high-capacity PEFT with RSLoRA is enough to keep degradation on FullStackBench C++ modest, with compile rate moving from 44.9% to 41.1% and test pass rate from 15.9% to 13.1%, which it describes as minimal degradation rather than catastrophic forgetting (Singh et al., 11 Mar 2026). In instruction-tuned LLM adaptation, IKnow’s central claim is that naive next-token continued pretraining can catastrophically damage instruction-following, as illustrated by a RepliQA score collapse to 0.2 for Llama-3.2-3B-Instruct under full NTP adaptation, while instruction-formatted MTP, MPP, and NLKG objectives avoid that failure (Zhang et al., 23 Oct 2025).

The literature therefore does not support a single universal anti-forgetting recipe. Replay, soft masking, frozen teachers, parameter isolation, internal expansion, and instruction-compatible self-supervision all appear, and their suitability depends on whether the model is encoder-only, decoder-only, already instruction tuned, or adapted under resource constraints.

6. Limitations, disagreements, and future directions

DACP research is empirically strong but methodologically heterogeneous, and several recurring limitations appear across the literature. Many studies are single-step or single-domain, not multi-stage lifelong adaptation. The dialogue paper explicitly does not examine multi-domain continual adaptation, catastrophic forgetting of general-domain knowledge, larger adaptation corpora, or longer schedules (Wu et al., 2021). The biomedical adapter paper is specific to Chinese biomedical text and BERT/RoBERTa-style encoders, does not compare vocabulary redesign, and measures forgetting only through general-domain MLM accuracy rather than downstream tasks (Yan et al., 2022). Industrial studies often rely on in-house benchmarks and partially specified post-training details, which makes it difficult to isolate the effect of DACP from later alignment or service-specific fine-tuning (Kim et al., 9 Jul 2025, Khasanova et al., 9 Oct 2025, Fu et al., 7 Oct 2025).

Several substantive disagreements also remain open. One concerns whether replay is necessary. Industrial small-LLM work presents replay as the main anti-forgetting mechanism and selects around 50% replay as the best balance (Kim et al., 9 Jul 2025), whereas the embedded-systems code paper reports that domain-only training outperformed mixed domain-plus-general-code mixtures and argues that LoRA already mitigates forgetting (Singh et al., 11 Mar 2026). Another concerns whether DACP uniformly improves downstream reasoning. The legal adaptation study argues that DACP improves legal-domain knowledge but does not reliably improve prompt-based legal reasoning and can degrade some exam-style tasks (Chen et al., 18 Apr 2025). A related disagreement concerns what counts as the correct adaptation signal: some papers stay close to the original pretraining objective, whereas DACIP-RC and IKnow argue that instruction-tuned or enterprise-oriented settings may require instruction-formatted or task-synthetic self-supervision rather than raw next-token prediction (Khasanova et al., 9 Oct 2025, Zhang et al., 23 Oct 2025).

Future directions suggested by the literature are concrete. One is stronger control of what the model is exposed to during DACP: data selection, retrieval augmentation, phrase-aware masking, and tokenizer adaptation all indicate that corpus design can matter as much as corpus size (Hiwarkhedkar et al., 2024, Zhukova et al., 28 Apr 2025, Feng et al., 2024). Another is more realistic continual evaluation over long domain sequences, where order, similarity, and saturation effects become visible (Yıldız et al., 2024). A third is adaptation of already instruction-tuned checkpoints without access to the base model, which motivates instruction-compatible continual objectives (Zhang et al., 23 Oct 2025). A fourth is stronger analysis of computational efficiency, especially for small or medium models that must operate under industrial resource constraints (Kim et al., 9 Jul 2025, Zhukova et al., 28 Apr 2025).

Taken together, the literature supports a concise but qualified characterization of DACP. Continued in-domain pretraining usually helps, often substantially, and can be effective even with only the target task’s own unlabeled data. However, the benefit is not solely a function of “more in-domain text.” It depends on the match between objective and downstream structure, the construction and mixing of corpora, the retention mechanism, the model’s initialization state, and the evaluation regime. In that sense, DACP has evolved from a simple continuation of pretraining into a broader design space of objective engineering, corpus engineering, and continual-learning control.