Within-Domain Adapters for Neural Networks

• Within-domain adapters are lightweight, trainable modules injected into frozen neural network layers to enable efficient, domain-specific adaptation.
• They employ a bottleneck architecture with down-projection, nonlinearity, and up-projection, typically integrated after key transformer sub-layers.
• These adapters facilitate multi-domain generalization, continual learning, and robust performance while tuning only a small percentage of model parameters.

Within-domain adapters are lightweight, trainable modules incorporated into neural network architectures—most commonly transformers—for efficient, domain-specialized adaptation. Rather than large-scale fine-tuning of all model parameters, within-domain adapters target the integration or isolation of domain-specific characteristics while preserving strong performance on original or related domains. Their mechanisms, parameterizations, and deployment span language, vision, and multi-modal models, enabling parameter-efficient transfer, multi-domain generalization, continual learning, and compositional inference.

1. Architectural Fundamentals of Within-Domain Adapters

Adapters universally involve injecting domain-specific parameterized transformations into frozen layers of a backbone model. In transformer-based architectures, this typically takes the form of a bottleneck design: a low-dimensional projection (down-projection), nonlinearity, and re-projection (up-projection), added to the layer output via a residual connection. For example, the generic form is

$$h' = h + W_{\mathrm{up}}\;\sigma\left( W_{\mathrm{down}} h + b_{\mathrm{down}} \right) + b_{\mathrm{up}}$$

where $W_{\mathrm{down}}\in\mathbb R^{r \times d}$, $W_{\mathrm{up}}\in\mathbb R^{d \times r}$, $r\ll d$ controls adapter capacity, and $\sigma$ is a nonlinearity (typically ReLU or GeLU) (Schopf et al., 2023).

Crucial variants include:

• Domain-specific adapters: Each domain $i$ is assigned a unique set of bottleneck adapter parameters, which are swapped in at inference (Stickland et al., 2021, Omi et al., 2022, Presta et al., 2024).
• Low-rank/LoRA adapters: Utilize low-rank matrices for updates, e.g., $\Delta W = A B$, $A\in\mathbb R^{d \times r}$, $B\in\mathbb R^{r \times d}$, to reduce parameter overhead for each domain (Liu et al., 2024, Shah et al., 16 Mar 2026, Hedjazi et al., 11 Apr 2025).
• Residual MLP adapters: Used in many vision-language settings, these MLPs are attached on top of encoder outputs and combined via a residual ratio, as in CLIP-Adapters and Soup-Adapters (Roschkowski, 8 Jul 2025).

Insertion points are architecture-specific but common strategies include:

• After multi-head attention and/or after the feed-forward sub-layer in transformer blocks (Schopf et al., 2023, Stickland et al., 2021, Liu et al., 2024).
• Within convolutional/decoder layers for image compression or video models (Omi et al., 2022, Presta et al., 2024).

2. Training Protocols and Parameter Efficiency

Within-domain adapters are trained to specialize on domain-specific data while keeping the backbone frozen. Only the small set of adapter parameters are updated during domain adaptation. For sentence embedding models (e.g., SimCSE on BERT-base), adapting with Houlsby-style adapters (two per layer, $r=48$) leads to $W_{\mathrm{down}}\in\mathbb R^{r \times d}$0M tunable parameters ($W_{\mathrm{down}}\in\mathbb R^{r \times d}$1 of the total), with a negligible performance loss ($W_{\mathrm{down}}\in\mathbb R^{r \times d}$2) compared to full fine-tuning (Schopf et al., 2023). In DARA for visual grounding, DA adapters use just $W_{\mathrm{down}}\in\mathbb R^{r \times d}$3M parameters on top of a 151M backbone, yielding substantial gains over a frozen model (Liu et al., 2024).

The training loss is typically task-specific (cross-entropy for classification, contrastive for embeddings, MSE for image compression) and only gradients for the adapter layers are propagated. Adapters can be trained sequentially (one per domain), jointly (multi-domain adaptation), or in more elaborate hierarchical or compositional regimes (Chronopoulou et al., 2021, Shah et al., 16 Mar 2026).

3. Strategies for Mixing, Merging, and Routing Domain Adapters

Efficient use of multiple within-domain adapters involves either averaging, hierarchical aggregation, or dynamic selection.

• Adapter mixtures: Uniform or weighted averaging of adapter weights, e.g.,

$W_{\mathrm{down}}\in\mathbb R^{r \times d}$4

AdapterSoup mixes only a subset of domain-specific adapters, with practical recommendations to limit $W_{\mathrm{down}}\in\mathbb R^{r \times d}$5 due to interference measured via the fraction of sign difference (FSD) between adapters (Nguyen et al., 2024).

• Hierarchical adapters: Arrange domains into a tree; activate adapters along a root-to-leaf path. This supports parameter-sharing for related domains and specialization for leaves (Chronopoulou et al., 2021):

| Domain Adapter Hierarchy | Mixing Method | Empirical Effect | |-------------------------|---------------|------------------------| | Root, parent, leaf | Path average | Improved in-domain and OOD generalization |

• Compositional/Per-token routing: MoLoRA implements a learned or deterministic router $W_{\mathrm{down}}\in\mathbb R^{r \times d}$6 that selects (possibly multiple) domain adapters on a per-token basis, supporting mixed-modality or mixed-domain input. Per-token dispatch achieves $W_{\mathrm{down}}\in\mathbb R^{r \times d}$7 compute (for $W_{\mathrm{down}}\in\mathbb R^{r \times d}$8 tokens) rather than $W_{\mathrm{down}}\in\mathbb R^{r \times d}$9 for $W_{\mathrm{up}}\in\mathbb R^{d \times r}$0 domains with per-sequence routing (Shah et al., 16 Mar 2026).
• Zero-overhead merging: Methods such as TaDA merge a task LoRA and a domain LoRA into a single adapter with calibrated, probe-guided, per-layer weights and subspace filtering, restoring a standard rank-$W_{\mathrm{up}}\in\mathbb R^{d \times r}$1 adapter with no inference overhead and carefully respecting depth-dependent dominance (To et al., 3 Jun 2026).
• Feature gating and output heads: In continual and sequential learning, feature gating (learned sigmoid applied to previous domain’s LoRA outputs) and per-domain output heads mitigate catastrophic forgetting and allow domain-isolated representations (Hedjazi et al., 11 Apr 2025).

4. Robustness, Interference, and Generalization

Adapter composition introduces the risk of negative interference due to conflicting parameter directions. Empirical evaluation of FSD between adapters reveals—

• High FSD between adapters predicts large drops in in-domain accuracy for mixtures; Pearson $W_{\mathrm{up}}\in\mathbb R^{d \times r}$2 for dual-adapter mixing (Nguyen et al., 2024).
• Larger mixtures ($W_{\mathrm{up}}\in\mathbb R^{d \times r}$3) amplify this, underlying the recommendation to mix at most a few adapters and select based on FSD.
• Magnitude pruning of small adapter weights (up to 90% sparsity) can reduce conflicting contributions, thus improving accuracy and robustness for adapter mixtures (Nguyen et al., 2024).

Ensembling (Soup-Adapters) also directly addresses robustness under domain and hyperparameter shift:

• Averaging $W_{\mathrm{up}}\in\mathbb R^{d \times r}$4 independently trained MLP adapters and reparameterizing into a single wide MLP improves both in-distribution and out-of-distribution (OOD) robustness compared to any individual adapter and reduces sensitivity to key hyperparameters (e.g., residual ratio $W_{\mathrm{up}}\in\mathbb R^{d \times r}$5), with performance gains saturating for $W_{\mathrm{up}}\in\mathbb R^{d \times r}$6 (Roschkowski, 8 Jul 2025).

5. Applications Across Modalities and Tasks

Within-domain adapters are prominent in a variety of domains:

• Natural language: Domain-specific adapters for NLU/NLP (GLUE, scientific QA), multilingual NMT (stacked language/domain adapters), and sentence embeddings (SimCSE, SciDocs, AskUbuntu) (Schopf et al., 2023, Stickland et al., 2021, To et al., 3 Jun 2026).
• Vision and vision-language: DA adapters in visual grounding (TransVG, DARA), low-rank adapters and gating in ViT for continual learning, and domain-specific adapters in video action recognition (Liu et al., 2024, Hedjazi et al., 11 Apr 2025, Omi et al., 2022).
• Image compression: Domain adapters as small residual conv/deconv blocks, blended via a gate network; this both enhances rate-distortion performance on target domains and prevents catastrophic forgetting on the original domain (Presta et al., 2024).
• Multi-modal and compositionality: MoLoRA for per-token routing in multi-modal (text/image) or mixed-capability inference, allowing on-the-fly modular composition of domain and capability experts (Shah et al., 16 Mar 2026).

Parameter costs are consistently minimal: 1–4% of model size for adapters per domain, sometimes <1% for vision models (Schopf et al., 2023, Liu et al., 2024, Hedjazi et al., 11 Apr 2025, Omi et al., 2022).

6. Empirical Evaluations and Practical Recommendations

Key findings and practitioner's guidelines include:

• Use small bottleneck dimensions ($W_{\mathrm{up}}\in\mathbb R^{d \times r}$7 or $W_{\mathrm{up}}\in\mathbb R^{d \times r}$8 for ViT/LoRA) to balance efficiency and adaptation quality (Schopf et al., 2023, Hedjazi et al., 11 Apr 2025).
• For combinatorial mixing, restrict to a few adapters with small FSD, prune for sparsity, and use Soup-Adapter reparameterization to mitigate robustness issues (Nguyen et al., 2024, Roschkowski, 8 Jul 2025).
• In multi-domain or hierarchical adaptation, path averaging along the domain tree yields in-domain and OOD gains while enabling scalable parameter sharing (Chronopoulou et al., 2021).
• For continual learning, freeze prior adapters, apply gating, and train domain-specific heads to prevent forgetting and maximize accuracy over task sequences (Hedjazi et al., 11 Apr 2025).
• For NMT, properly decouple language and domain adapters, regularize with domain-adapter dropout, and use back-translation to maximize zero-shot and cross-lingual transfer (Stickland et al., 2021).

Adapter architectures, composition strategies, and training protocols should be selected to match the granularity of domain difference, the requirement for robustness and transfer, and the operational constraints of inference deployment.

7. Open Challenges and Future Directions

Despite considerable successes, several limitations are noted. Adapter mixtures with large FSD remain problematic; manual tree construction in hierarchical approaches can compromise sharing; continual learning performance is sequence-dependent; and merging task and domain signals requires careful calibrated algorithms (e.g., TaDA) to avoid destructive interference (To et al., 3 Jun 2026, Chronopoulou et al., 2021, Hedjazi et al., 11 Apr 2025).

A plausible implication is that further work on automatic domain clustering, learnable adapter placement, dynamic router construction, cross-modal adapter fusion, and theoretical characterization of adapter interactions will advance the field. Emerging designs, such as per-token routing (MoLoRA) and calibrated probe gating (TaDA), suggest the potential for more granular, computation-efficient, and robust within-domain specialization across increasingly diverse and compositional real-world scenarios.