Domain Prompts: Bridging Pretraining and Domain Shift

• Domain prompts are specialized representations that encode domain-specific signals to mitigate performance degradation when models face shifts between pretraining and deployment data.
• They manifest as continuous prefix embeddings, generated soft prompts, and image-level adaptations, demonstrating versatility across modalities like NLP, vision, and speech.
• Parameter-efficient domain prompt methods optimize small auxiliary modules to yield measurable gains in domain generalization and unsupervised adaptation.

Domain prompts are prompt representations that encode domain information so that a pretrained model can adapt to domain shift without full fine-tuning. Across the literature, they appear as continuous prefix embeddings for LLMs, learned context tokens for CLIP-style vision-LLMs, image-level visual prompts, generated soft prompts, and domain-conditioned prompt vectors inside diffusion and detection systems. Their central function is to bridge a mismatch between pretraining and deployment distributions in settings such as domain generalization, unsupervised domain adaptation, low-resource special-domain classification, automatic speech recognition rescoring, federated learning, continual test-time adaptation, time-series generation, and document layout analysis (Zhang et al., 2021, Ge et al., 2022, Dingliwal et al., 2021).

1. Problem setting and conceptual rationale

The modern literature on domain prompts begins from a recurring empirical observation: pretrained models generalize strongly within or near their pretraining distribution, but degrade when the downstream data come from a different domain. In low-resource NLP, the stated issue is the gap between general-domain pretraining data and special-domain target data, which makes standard prompting less effective and motivates domain-oriented prompting (Goswami et al., 2023). In CLIP-based unsupervised domain adaptation, the stated criticism of conventional feature-alignment methods is that direct source-target alignment can distort semantic structure and reduce class discriminability; DAPL therefore embeds domain information into prompts rather than trying to suppress it in the feature space (Ge et al., 2022). In source-free domain generalization, the key motivation is that source images may be unavailable, whereas domain styles can still be represented textually through prompt banks and unified into robust class representations (Niu et al., 2022).

A second line of motivation concerns the structure of domain shift itself. DoPrompt argues that source domains can be diverse, that learning a single domain-invariant representation is hard, and that previous domain-specific methods inject source knowledge too weakly or aggregate experts too naively at test time (Zheng et al., 2022). MetaPrompt, at the level of its abstract, frames the problem as the difficulty of generalizing prompt learning to unseen classes and domains and proposes directly generating a domain invariant prompt with a dual-modality prompt tuning network, an asymmetric contrastive loss, and a meta-learning-based prompt tuning algorithm (Zhao et al., 2022). Taken together, these works distinguish two broad strategies: preserving domain information and making the prompt domain-aware, or constraining prompt learning so that the prompt itself becomes more transferable across unseen domains. This suggests that “domain prompt” is not a single algorithmic object but a family of adaptation interfaces between pretrained representations and domain-shifted inputs.

2. Representational forms of domain prompts

The simplest representational form is the learned continuous prefix. In ASR domain adaptation, domain-prompts are a small set of learned, continuous, domain-specific embedding vectors $$\Phi_D = [x_{\phi_D^1}, x_{\phi_D^2}, \dots, x_{\phi_D^k}]$$ that are prefixed to GPT-2 token embeddings and optimized while the base model remains frozen (Dingliwal et al., 2021). In CLIP-based UDA, DAPL uses prompts of the form $$p_k^d = []_1 []_2 \ldots []_{M_1} []_1^d []_2^d \ldots []_{M_2}^d [\mathrm{CLASS}]_k$$, where domain-agnostic context is shared across all data and domain-specific context is shared only within a domain (Ge et al., 2022). In CLIP-based DG, DPL conditions the text prompt by concatenating a learned domain prompt with a class prompt, $$\mathbf{p}_k^{i} = [\mathbf{p}_{ap}^{i}; \mathbf{p}_{k}]$$, where the adaptive prompt is inferred from image features by a three-layer MLP (Zhang et al., 2021).

Later work diversified this representation space. DiPrompT explicitly separates a global prompt, domain prompts, and a query prompt, with each domain prompt including a domain-specific token in addition to shared prompt tokens, thereby decoupling shared cross-client knowledge from latent-domain knowledge in federated settings (Bai et al., 2024). FedDSPG defines domain-specific soft prompts as a concatenation of domain-invariant context tokens, domain-specific context tokens, and a class token, and then trains a generator to produce these prompts for unseen target domains (Wu et al., 25 Sep 2025). DCPL uses domain biases rather than free prompt tokens: an LSDM embedding is passed through lightweight control networks to produce a language domain bias $D_b^l$ and a visual domain bias $D_b^v$, which are added to text context tokens and CLIP image features, respectively (Cao et al., 2023).

Other lines of work push prompts beyond standard token sequences. “Decorate the Newcomers” reformulates the input itself with a domain-agnostic prompt $\omega_\phi$ and a domain-specific prompt $\psi_\delta$ added directly to the target image, $x_p^T = x^T + \omega_\phi + \psi_\delta$, thereby turning prompt learning into image-level input adaptation (Gan et al., 2022). TimeDP defines a domain prompt as a prototype-weight vector over a frozen bank of time-series semantic prototypes, with prototype assignment weights serving as the conditioning signal for a DDPM (Huang et al., 9 Jan 2025). PromptDLA generates natural-language domain prompts from document-type, language, and annotation-style cues, encodes them with a frozen text encoder, and injects the resulting prompt vector into a DLA backbone (Zhang et al., 10 Mar 2026). In compositional image synthesis, the domain prompt is a class-agnostic visual prompt $p_\phi$ that is concatenated with a source-class-distilled semantic token $\hat{t}_c$ so that style and class can be recombined for zero-shot domain-adaptive generation (Sohn et al., 2023).

3. Learning paradigms and optimization objectives

Despite their representational diversity, most domain-prompt methods are parameter-efficient: the large pretrained backbone is frozen or mostly frozen, while only prompt parameters or small auxiliary modules are optimized. DAPL freezes CLIP encoders and trains domain-agnostic and domain-specific prompt tokens with a supervised source-domain cross-entropy loss plus a pseudo-label loss on confident target samples (Ge et al., 2022). DPL also freezes CLIP’s image and text encoders and trains only the prompt generator $$p_k^d = []_1 []_2 \ldots []_{M_1} []_1^d []_2^d \ldots []_{M_2}^d [\mathrm{CLASS}]_k$$0, which averages prompt outputs over a batch of images from the same domain to form a shared domain prompt (Zhang et al., 2021). In DoPrompt, by contrast, prompt learning is combined with backbone fine-tuning, and the method jointly optimizes a domain-prompt loss, an adapted-prompt classification loss, and a prompt-weight supervision loss for a two-layer MLP prompt adapter (Zheng et al., 2022).

A distinct set of methods treat prompt learning as generation. SPG first learns a domain prompt label for each source domain, then trains a conditional GAN so that the generator produces instance-specific soft prompts conditioned on image features and noise; at inference time, the generator produces prompts for unseen target-domain images (Bai et al., 2024). DPSPG replaces single-path prompt generation with dual-path stable soft prompt generation: a positive generator $$p_k^d = []_1 []_2 \ldots []_{M_1} []_1^d []_2^d \ldots []_{M_2}^d [\mathrm{CLASS}]_k$$1 and a negative generator $$p_k^d = []_1 []_2 \ldots []_{M_1} []_1^d []_2^d \ldots []_{M_2}^d [\mathrm{CLASS}]_k$$2 are trained against positive and negative domain prompt labels, and the final class probability subtracts negative similarity from positive similarity. The paper argues theoretically that this enlarges the effective margin and reduces an upper bound on the gradient norm (Zhang et al., 24 May 2025). MetaPrompt, again only at the level of its abstract description, couples a dual-modality prompt tuning network with an asymmetric contrastive loss and a meta-learning-based prompt tuning algorithm intended to constrain prompts learned on one domain or class to perform well on another (Zhao et al., 2022).

Across these systems, the optimization target changes with the application. In ASR, domain-prompts are trained by next-token prediction on unlabeled domain text and then reused for second-pass rescoring (Dingliwal et al., 2021). In continual test-time adaptation, the visual domain prompt paper uses a teacher-student consistency objective for both prompt types and adds a homeostasis-based regularizer to the domain-agnostic prompt so that domain-sensitive parameters are suppressed during continual shift (Gan et al., 2022). In source-free DG, domain-unified prompt representations are learned entirely in prompt space, via mean pooling or a cosine autoencoder with reconstruction, intra-class, and inter-class losses over prompt embeddings, without using source images (Niu et al., 2022). A plausible implication is that domain prompts are best understood as a locus for relocating adaptation pressure: from backbone weights to a smaller, more structured parameter subspace.

4. Major application regimes

In low-resource NLP classification, SwitchPrompt combines a general-domain soft prompt, automatically extracted domain-specific keywords, and two input-dependent gates. The first gate interpolates between general and domain-specific prompt components, and the second gate chooses the order of the learned vectors and keywords, making the prompt both domain-oriented and input-dependent (Goswami et al., 2023). In ASR, domain-prompts are trained from as few as 1,000 unlabeled domain-specific sentences and cached as a constant prefix during hypothesis rescoring, making them deployment-friendly for multi-domain systems (Dingliwal et al., 2021).

In vision-language classification under domain shift, several subfamilies appear. DPL conditions CLIP’s text-side classifier on a prompt inferred from unlabeled images of the current domain (Zhang et al., 2021). DoPrompt stores source-domain knowledge in ViT prompt tokens and learns a prompt adapter that synthesizes a weighted mixture of source prompts for each test image (Zheng et al., 2022). DCPL transfers specific-domain knowledge from a large-scale specific domain foundation model, such as a remote sensing foundation model or MedSAM, into visual and language branches through directly added domain biases (Cao et al., 2023). Source-free DG uses domain banks containing prompts such as watercolor, sketch, or cartoon, then learns domain-unified prompt representations through mean pooling or a cosine autoencoder (Niu et al., 2022).

Dynamic and generative prompting dominates more recent DG work. SPG introduces source-domain prompt labels and a CGAN-based generator for instance-specific soft prompts on unseen domains (Bai et al., 2024). DPSPG adds a complementary negative prompt stream to improve stability and suppress misleading information (Zhang et al., 24 May 2025). In federated settings, FedDSPG generates domain-specific soft prompts for each client domain and reuses the generator at inference for unseen domains, while DiPrompT disentangles global prompts, domain prompts, and query prompts so that latent domain structure can be inferred without explicit domain labels (Wu et al., 25 Sep 2025, Bai et al., 2024).

Domain prompts also extend beyond standard classification. For continual test-time adaptation, image-level visual domain prompts adapt a frozen source model to a sequence of unlabeled target domains while separating current-domain knowledge from domain-shared knowledge (Gan et al., 2022). In time-series generation, TimeDP uses prototype-assignment vectors as domain prompts to condition a multi-domain diffusion model and to support few-shot generation in unseen domains (Huang et al., 9 Jan 2025). In document AI, PromptDLA converts descriptive domain knowledge about document type, language, and annotation style into prompt embeddings that guide a prompted encoder for document layout analysis (Zhang et al., 10 Mar 2026). In compositional image synthesis, source-class-distilled visual prompts learn disentangled semantic and domain prompts from as few as one image, enabling synthesis of arbitrary source classes in the style of the target domain (Sohn et al., 2023). This suggests that the operative notion of “domain” has expanded from dataset identity to include style, sensor statistics, labeling conventions, document genre, and latent temporal pattern families.

5. Empirical behavior and comparative results

The empirical record consistently emphasizes parameter efficiency together with measurable gains under domain shift. DPL raises CLIP with a template prompt from 73.7 average accuracy to 79.6 on VLCS, PACS, OfficeHome, and TerraIncognita, with the largest reported gain on TerraIncognita from 34.1 to 52.6 (Zhang et al., 2021). DoPrompt reports a ViT-Base/16 average of 73.2, compared with a ViT ERM baseline of 71.8 and a best prior method average of 72.2, and characterizes its 1.4% gain as 3.5 times the improvement of the previous state of the art on that backbone (Zheng et al., 2022). SPG reports 74.7 average accuracy with ViT-B/16 in multi-source DG and 73.8 in cross-dataset generalization, while also outperforming CoOp, CoCoOp, and DPL in single-source DG (Bai et al., 2024).

Stability-focused prompt generation reports a different empirical profile. DPSPG reaches 93.66% average accuracy on PACS with ResNet-50, beats SPG by 0.86% there, surpasses SPG by 7.89% on TerraIncognita with ResNet-50, and claims an average improvement of about 3.94% over five DG benchmarks using ResNet-50. It also reports lower intra-domain prompt distance, a better intra-to-inter domain distance ratio, lower standard deviation over the last 10 epochs, and tighter prompt clusters (Zhang et al., 24 May 2025). In low-resource NLP, SwitchPrompt reports gains of up to 10.7 accuracy points and improvements of up to 2.1 points over other state-of-the-art prompting methods, while sometimes allowing general-domain BERT with SwitchPrompt to outperform domain-specific pretrained models using standard prompting (Goswami et al., 2023).

Efficiency claims are especially strong in ASR and document AI. Prompt tuning GPT-2 with domain-prompts yields 7–13% WER reduction for a new domain using 1,000 unlabeled domain-specific sentences while updating about 0.02% of the base LM parameters; concrete low-data WERR results include 13.1 for fastfood on GPT-2-medium with vocab initialization (Dingliwal et al., 2021). PromptDLA reports 78.7 mAP on DocLayNet, 69.2 mAP on M6Doc, and 69.1 mAP on D$$p_k^d = []_1 []_2 \ldots []_{M_1} []_1^d []_2^d \ldots []_{M_2}^d [\mathrm{CLASS}]_k$$3LA, beating SwinDocSegmenter on DocLayNet by +1.8 mAP, TransDLA on M6Doc by +5.4 mAP, and VGT on D$$p_k^d = []_1 []_2 \ldots []_{M_1} []_1^d []_2^d \ldots []_{M_2}^d [\mathrm{CLASS}]_k$$4LA by +0.3 mAP (Zhang et al., 10 Mar 2026). In domain-controlled prompting, DCPL reports harmonic-mean improvements over CLIP, CoOp, CoCoOp, and MaPLe on remote sensing and medical image benchmarks, including an average HM of 83.36 on eight remote-sensing datasets with ViT and 53.08 on three medical datasets (Cao et al., 2023).

6. Limitations, misconceptions, and open questions

A common misconception is that prompt learning is inherently stable once the backbone is frozen. DPSPG explicitly identifies “Prompt Variability,” the phenomenon that the same input can yield substantially different and suboptimal prompts across random seeds, and argues that prompt stability is itself a first-class objective in domain generalization (Zhang et al., 24 May 2025). Another misconception is that stronger domain knowledge sources always imply better prompt adaptation. DCPL notes that a deeper LSDM such as ViTAE is not always better than a ViT-based LSDM for prompt learning, and it also reports that gains are not uniform across all datasets or all split settings (Cao et al., 2023).

Interpretability remains limited for several continuous-prompt methods. DPL states that its learned domain prompt is a continuous vector and therefore not directly interpretable; it also notes that the method uses only visual information and cannot capture non-visual shifts or label shift (Zhang et al., 2021). Source-free DG via a domain bank finds that richer prompt banks help, but overly large or redundant banks do not always help, and TerraIncognita remains a weak point, which the paper attributes to smaller domain gaps across camera sources and visually similar rare classes (Niu et al., 2022). In compositional image synthesis, prompt capacity must be carefully bottlenecked: too small underfits the target domain, while too large causes entanglement between style and class and breaks compositional reuse (Sohn et al., 2023).

There are also domain-specific caveats. In federated DG, FedDSPG assumes a frozen CLIP ViT-B/16 backbone shared across clients and reports that too much communication reduction or overly infrequent aggregation can hurt performance (Wu et al., 25 Sep 2025). In prompt engineering for domain vocabulary, increasing lexical specificity does not generally improve performance; the reported result is instead an optimal specificity range, with nouns roughly 17.72–19.70 and verbs roughly 8.08–10.57, while overly specific verbs can be harmful in reasoning tasks (Schreiter, 10 May 2025). This suggests that prompt quality cannot be reduced to “more domain information is better.” Across the literature, the more defensible conclusion is narrower: domain prompts are effective when their parameterization, conditioning signal, and optimization objective match the structure of the domain shift they are intended to mediate.