Domain-Specific Adapters

• Domain-specific adapters are lightweight, parameter-efficient modules that integrate with frozen backbones to enable specialized domain adaptation.
• They address the trade-off between adapting to domain-specific challenges and retaining the generalization of pretrained models, reducing storage, computation, and catastrophic forgetting.
• Recent research explores diverse architectures, training procedures, and composition strategies in NLP, vision, and multimodal systems for efficient and scalable adaptation.

Domain-specific adapter denotes a parameter-efficient module that specializes a shared pretrained model to a particular domain while preserving the backbone’s general knowledge. The adapter literature spans pretrained LLMs, ConvNets, recommendation models, vision-LLMs, diffusion models, and multilingual NMT, and it frequently frames the central trade-off as domain adaptation versus storage, computation, and catastrophic forgetting. Taken together, these works suggest that the term refers less to a single block design than to a family of modular specialization mechanisms attached to a frozen or mostly shared backbone (Fichtl et al., 2024, Diao et al., 2023, Berriel et al., 2019, Omi et al., 2022).

1. Problem formulation and rationale

The basic motivation is consistent across domains. Generic pretrained models often perform well on broad text or visual distributions but degrade under domain shift, while full continued pretraining or full-parameter fine-tuning is expensive and can distort pretrained knowledge. This motivation is stated explicitly for unsupervised domain adaptation in NLP, for sentence embedding adaptation, for multi-domain visual learning, and for domain adaptation of LLMs (Zhang et al., 2021, Schopf et al., 2023, Berriel et al., 2019, Diao et al., 2023).

In NLP, this rationale appears in several forms. Adapter-based unsupervised domain adaptation freezes a pretrained LLM and learns only small inserted modules, first with mixed-domain masked language modeling and then with source-task supervision (Zhang et al., 2021). Sentence embedding adaptation uses small domain-specific adapters on top of a fixed SimCSEsup-bert-base encoder rather than fully fine-tuning a separate encoder per target domain (Schopf et al., 2023). Multilingual NMT distinguishes language-specific and domain-specific factors and studies whether they can be modularized separately, especially when in-domain parallel data is unavailable for some languages (Stickland et al., 2021).

In vision and multimodal systems, the same motivation is expressed through compute budgets, storage budgets, or domain-specific generalization. Budget-Aware Adapters treat multi-domain learning and complexity control as one problem by learning domain-specific binary channel gates over a shared ConvNet (Berriel et al., 2019). Domain adaptation for object detection, continual test-time adaptation, point clouds, and cross-domain retrieval likewise uses lightweight modules to capture target-specific or domain-specific information without rewriting the full backbone (Li et al., 2024, Liu et al., 2023, Park et al., 2023, Jiang et al., 2024).

2. Architectural patterns

The literature suggests several recurring design families rather than one canonical adapter form. Transformer bottleneck adapters remain the most common baseline: Houlsby-style adapters place bottleneck modules after attention and feed-forward blocks, while Pfeiffer-style adapters place a single bottleneck after the feed-forward block (Schopf et al., 2023, Fichtl et al., 2024). Multilingual NMT adopts a residual bottleneck form for both language adapters and domain adapters, stacking domain adapters after language adapters (Stickland et al., 2021). Adapter-based UDA with RoBERTa and XLM-R also uses one bottleneck MLP after the feed-forward sublayer of each Transformer block (Zhang et al., 2021).

Other papers reinterpret the adapter as a domain-memory mechanism. MixDA keeps the original pretrained FFN fixed and adds a domain-specific adapter in parallel, with the standard block extended from $\mathbf{F}=\mathrm{FFN}(\mathbf{q})$ to a second branch $\mathbf{K}=\mathrm{DA}(\mathbf{q})$, then fused by a mixture-of-adapters gate (Diao et al., 2023). ViDA separates a high-rank branch for current-domain-specific information from a low-rank branch for domain-shared knowledge, then combines them as

$f_f = f_o + \lambda_h f_h + \lambda_l f_l$

with uncertainty-driven homeostatic weighting (Liu et al., 2023). DA-Ada splits visual adaptation into a Domain-Invariant Adapter and a Domain-Specific Adapter, with the latter operating on the residual difference $\mathbf{h}_{i-1}-\mathbf{h}_i^I$ and modulating the invariant path through

$$\mathbf{h}_i = \mathbf{h}_i^I + \mathbf{h}_i^I \cdot \mathbf{h}_i^S$$

(Li et al., 2024).

Several methods depart from the standard bottleneck pattern entirely. Budget-Aware Adapters insert binary channel switches into every convolution,

$$\boldsymbol{x}(i,j)=g\Big(\sum_{c=1}^{C}s_c\phi_c(i,j)\Big),$$

turning the adapter into a structured channel selector over the shared backbone (Berriel et al., 2019). HAMUR uses a bottleneck residual adapter whose weights are generated dynamically by a domain-shared hyper-network, so the adapter is domain-specific and instance-specific at once (Li et al., 2023). DFN in cross-domain few-shot segmentation uses a same-width residual $1\times1$ convolution attached to deep backbone features and is interpreted as a structural domain-information decoupler rather than a classic bottleneck (Tong et al., 9 Jun 2025). MoLoRA treats multiple LoRA modules as token-routed specialists, moving from per-sequence routing

$h_i = x_i W + x_i A_{j(s_i)} B_{j(s_i)}$

to token-level routing

$h_i = x_i W + x_i A_{r(i)} B_{r(i)}$

or mixture form over Top-$k$ selected adapters (Shah et al., 16 Mar 2026).

Pattern	Mechanism	Representative papers
Bottleneck residual adapter	Down-project, nonlinearity, up-project, residual	(Schopf et al., 2023, Zhang et al., 2021, Stickland et al., 2021)
Parallel domain memory	Frozen FFN plus parallel domain adapter and gate	(Diao et al., 2023)
Structured sparsity adapter	Binary per-channel gating in convolutions	(Berriel et al., 2019)
Hyper-generated adapter	Adapter parameters generated by shared hyper-network	(Li et al., 2023)
Shared/specific decoupling	Separate invariant and domain-specific branches	(Liu et al., 2023, Li et al., 2024, Takama et al., 5 Aug 2025)
Structural decoupler	Residual $\mathbf{K}=\mathrm{DA}(\mathbf{q})$0 domain feature navigator	(Tong et al., 9 Jun 2025)

A further pattern is that some papers use “adapter” more broadly than a bottleneck block. UCDR-Adapter is best described as a prompt-plus-adapter hybrid built from class prompts, domain prompts, and dynamic target prompt generation rather than a standard transformer bottleneck (Jiang et al., 2024). ResAdapter is primarily a resolution adapter for diffusion U-Nets, but it is explicitly designed to preserve the style domain of personalized models while adapting resolution and aspect ratio (Cheng et al., 2024).

3. Training procedures and compositionality

A central divide in the literature is between two-stage adaptation and end-to-end joint training. MixDA uses an explicitly two-stage recipe: first train domain adapters on unlabeled domain data with masked language modeling plus an old-domain alignment loss,

$\mathbf{K}=\mathrm{DA}(\mathbf{q})$1

then freeze them and train only the mixture gate and task adapter on labeled downstream data (Diao et al., 2023). Ada-TSA follows a related two-step pattern for unsupervised domain adaptation: mixed-domain MLM over source and target text, followed by source-task fine-tuning while the pretrained backbone remains frozen (Zhang et al., 2021). Multilingual NMT likewise trains language adapters and then stacks domain adapters on top, with variants such as Domain Adapter Dropout and English-centric back-translation to stabilize unseen language/domain compositions (Stickland et al., 2021).

Other systems reject the frozen-backbone, stagewise pattern. HAMUR explicitly trains backbone, adapter-related factors, and hyper-network jointly with CTR cross-entropy because recommender systems are frequently updated online and are described as unsuitable for a two-stage pretrain/fine-tune paradigm (Li et al., 2023). UCDR-Adapter uses a two-phase procedure of source prompt learning and target prompt generation, but its trainable machinery is prompt-centric rather than a conventional adapter stack (Jiang et al., 2024).

Composition is itself a major theme. The systematic review of adapter-based KELMs highlights AdapterFusion, AdapterSoup, hierarchical domain adapters, and biomedical partition-and-fuse designs such as MoP as ways to combine knowledge sources or domains without overwriting the base model (Fichtl et al., 2024). The signed-weight analysis of domain-specific adapter mixtures shows that naive weight averaging often harms in-domain generalization, and that higher Fraction of Sign Difference is negatively correlated with performance; the same paper uses this observation to motivate sparse, pruned mixtures of adapters (Nguyen et al., 2024). MoLoRA pushes composition to token granularity by letting different tokens in the same sequence invoke different LoRA specialists, while preserving one-pass execution under per-token routing (Shah et al., 16 Mar 2026).

4. Application domains

The term “domain-specific adapter” covers a wide range of application regimes. In language modeling and retrieval, it includes sentence embedding adaptation for scientific papers and technical forums, unsupervised domain adaptation from source-labeled to target-unlabeled corpora, and multilingual machine translation with separate language and domain modules (Schopf et al., 2023, Zhang et al., 2021, Stickland et al., 2021). In knowledge-enhanced language modeling, the survey reports that open-domain approaches are more common overall, but the biomedical domain is the most prominent closed-domain setting, with systems such as DAKI, MoP, and KEBLM integrating UMLS, biomedical ontologies, and multiple biomedical knowledge sources through adapter-based architectures (Fichtl et al., 2024).

In recommendation and retrieval, HAMUR uses domain-specific adapters plus a shared hyper-network for multi-domain CTR prediction (Li et al., 2023). DM-Adapter adapts frozen CLIP vision and language branches to text-based person retrieval through sparse mixtures of adapters and a domain-aware router that augments routing logits from $\mathbf{K}=\mathrm{DA}(\mathbf{q})$2 to $\mathbf{K}=\mathrm{DA}(\mathbf{q})$3 (Liu et al., 6 Mar 2025). UCDR-Adapter specializes CLIP for universal cross-domain retrieval through source domain/class prompts and dynamic prompt generation for unseen domains and classes (Jiang et al., 2024).

In visual recognition and adaptation, domain-specific adapters appear in several distinct roles. Budget-Aware Adapters derive domain-specific subnetworks from a shared ConvNet under explicit complexity budgets (Berriel et al., 2019). The action-recognition MDL framework inserts domain-specific adapters between shared layers and switches both intermediate adapters and final heads by dataset/domain (Omi et al., 2022). The NAS-based MDL framework searches both what adapter structure to use and where to place adapters for each domain (Zhao et al., 2020). PC-Adapter uses a source-domain shape-aware adapter and a target-domain locality-aware graph adapter for unsupervised point-cloud adaptation (Park et al., 2023). DA-Ada injects domain-specific information into a frozen RegionCLIP-style detector, while ViDA separates current-domain-specific and domain-shared knowledge for continual test-time adaptation (Li et al., 2024, Liu et al., 2023).

Diffusion and generative modeling use the term in still broader ways. ResAdapter learns “pure resolution priors” through low-rank modifications to downsampler and upsampler convolutions and selective normalization, so that personalized Stable Diffusion or SDXL checkpoints can generate outside their native resolution domain while preserving style identity (Cheng et al., 2024). Att-Adapter adds a decoupled cross-attention branch for continuous multi-attribute control in a specific visual domain such as faces or cars, with a CVAE to prevent the attribute code from collapsing to a nearly unique identity (Cho et al., 15 Mar 2025).

5. Efficiency, scalability, and empirical behavior

A defining property of domain-specific adapters is their efficiency relative to full fine-tuning. MixDA adds about 2.6M parameters per domain on RoBERTa-large, or only 0.7% additional parameters relative to the backbone, excluding task adapters, and reports an overall average score of 52.2 versus 48.7 for Pfeiffer and 48.9 for full fine-tuning (Diao et al., 2023). Sentence-embedding adapters reach performance within about 1% of full in-domain SimCSE while training about 3.6% of the parameters, with the best Houlsby configuration at 81.53 average MAP versus 81.94 for full fine-tuning (Schopf et al., 2023).

In multi-domain vision, Budget-Aware Adapters are unusual because they also reduce inference FLOPs. On Visual Decathlon, BA at $\mathbf{K}=\mathrm{DA}(\mathbf{q})$4 uses 1.03× total parameters and 0.646 FLOP, and at $\mathbf{K}=\mathrm{DA}(\mathbf{q})$5 reaches 0.325 FLOP; the paper emphasizes that only BA reduces FLOPs below the backbone baseline among compared methods (Berriel et al., 2019). UCDR-Adapter reports 2.36M trainable parameters versus 87.84M for full fine-tuning, while DA-Ada trains only 1.794M learnable parameters and DFN increases parameters by about 2.1% and FLOPs by about 0.02% over its baseline (Jiang et al., 2024, Li et al., 2024, Tong et al., 9 Jun 2025).

In retrieval and multimodal transfer, DM-Adapter occupies a middle point between full tuning and prior PETL: it trains 16M parameters, compared with 195M for IRRA and 205M for CFine, and improves over the prior PETL baseline CSKT on CUHK-PEDES, ICFG-PEDES, and RSTPReid (Liu et al., 6 Mar 2025). ResAdapter uses only about 0.5M trainable parameters, reports a 44% speedup when combined with ElasticDiffusion, and is summarized as roughly 9× faster than ElasticDiffusion on average in the paper’s latency measurement (Cheng et al., 2024).

These empirical results also support a broader inference. Taken together, they suggest that domain-specific adapters are most competitive when a strong generic backbone already exists and the remaining gap is primarily one of domain-specific memory, routing, or control rather than wholesale architectural mismatch. MoLoRA states this argument most strongly: Qwen3-1.7B equipped with four specialist LoRAs and per-token routing exceeds Qwen3-8B on GSM8K, MATH, BBH, and GPQA while being 4.7× smaller (Shah et al., 16 Mar 2026).

6. Limitations, failure modes, and unresolved issues

The literature is equally clear that modular specialization does not automatically guarantee robust composition. MixDA is two-stage rather than end-to-end, assumes suitable unlabeled domain corpora, and notes that the multi-domain gate can make suboptimal choices (Diao et al., 2023). Budget-Aware Adapters require post-training budget compliance checks because SGD does not guarantee exact satisfaction of the KKT constraints, and different budgets require separate BN statistics (Berriel et al., 2019). HAMUR leaves some matrix-shape details inconsistent in the text and does not provide a dedicated experiment on transfer to entirely new unseen domains (Li et al., 2023).

Composition can fail more sharply in multilingual generation. In multilingual NMT, naive stacking of language-specific and domain-specific adapters often causes catastrophic forgetting of missing languages, especially on unseen target-language/domain combinations, with severe off-target translation; the paper’s main remedies are side-specific domain adaptation, Domain Adapter Dropout, and back-translation (Stickland et al., 2021). Adapter mixing can also fail in weight space: the signed-weight analysis paper finds that mixing many domain-specific adapters usually hurts in-domain test performance and ties this degradation to higher sign conflict between adapter weights (Nguyen et al., 2024).

Several papers also point to under-specification or conceptual drift in the term itself. UCDR-Adapter uses the word “adapter” but is best interpreted as a prompt-driven adaptation framework whose exact standard adapter-block design is not specified, and its target-prompt equations contain notational inconsistencies (Jiang et al., 2024). ResAdapter is domain-consistent rather than domain-specific in the usual semantic sense, because it preserves a personalized model’s existing style domain while adapting resolution (Cheng et al., 2024). The survey of adapter-based KELMs concludes that adapter gains vary strongly across models, tasks, and domains, that reasoning improvements are often smaller than gains on knowledge-intensive QA or classification tasks, and that the field still lacks precise universal guidance on method selection (Fichtl et al., 2024).

A plausible synthesis is that “domain-specific adapter” now names a design principle more than a fixed layer type: isolate domain-sensitive computation in lightweight, swappable modules; preserve a shared backbone wherever possible; and decide explicitly whether specialization should reside in weights, routes, prompts, subspaces, or residual branches. The diversity of current methods suggests both the flexibility of the idea and the absence of a settled canonical form.