Domain-Adapted Foundation Models

• Domain-adapted foundation models are specialized large-scale neural networks that integrate in-domain data and objectives to overcome limitations of general-purpose models.
• They employ parameter-efficient tuning techniques such as adapters and LoRA to minimize catastrophic forgetting and reduce computational costs.
• Empirical studies across diverse domains like digital dermatology, time series, and speech recognition demonstrate marked improvements in performance and efficiency.

Domain-adapted foundation models are large-scale, pre-trained neural networks whose architectures and training regimes are tailored to optimize performance in specific application domains by systematically incorporating in-domain data, objectives, and adaptation strategies. Unlike general-purpose foundation models trained on broad, heterogeneous datasets, domain-adapted variants are intentionally specialized to capture domain-specific patterns, semantics, and functional requirements, thereby enabling superior accuracy, robustness, and efficiency when deployed on downstream tasks relevant to the domain. This article surveys the technical foundations, adaptation methodologies, theoretical insights, and representative results in the construction and application of domain-adapted foundation models across modalities, with an emphasis on the rigorous state-of-the-art as presented in recent research.

1. Foundations and Scope of Domain-Adapted Foundation Models

Domain-adapted foundation models arise from the need to overcome the limitations of generic models in domains exhibiting specialized data distributions, ontologies, and operational constraints. While general-purpose foundation models (FMs) such as GPT-3, BERT, CLIP, or DINOv2 supply broad representational power, their performance degrades when exposed to domain shift, label imbalance, or context-specific semantics that diverge from their pre-training corpora (Chen et al., 2024). Domain adaptation of FMs encompasses architecture-level modifications, continued pre-training on in-domain corpora, parameter-efficient tuning, or hybrid strategies designed to mitigate the performance gap, including catastrophic forgetting.

The architectural decomposition of modern FMs relevant to domain adaptation comprises five layers: (1) modality encoders that digest domain-specific raw inputs, (2) input projectors that align embeddings to a joint feature space, (3) backbone calculational layers (large transformers or CNNs), (4) output projectors, and (5) modality decoders (Chen et al., 2024). Adaptation strategies may target one or several of these components, depending on the desired scope and efficiency.

2. Adaptation Frameworks and Parameter-Efficient Techniques

Adaptation methods can be broadly categorized according to their degree of intervention into foundation model parameters:

• Full-model fine-tuning updates all weights on domain-specific data, but runs the risk of catastrophic forgetting and is typically resource-intensive (He et al., 2022).
• Parameter-efficient fine-tuning—including adapters (bottleneck MLP insertion), Low-Rank Adaptation (LoRA), prompt tuning, and block expansion—updates only a subset of parameters, preserving general-domain performance and reducing compute and storage costs (Chen et al., 2024, Gatla et al., 28 Nov 2025, Zoellin et al., 2024).
• Frozen backbone with new heads trains only additional lightweight modules (linear heads, bottlenecks, or projection decoders) on top of frozen feature extractors, relying on feature generality and potentially sacrificing task-specificity if initial representations are insufficient (Kihara et al., 10 Sep 2025).

In federated and low-resource settings, lightweight adaptation and the use of frozen, high-capacity feature extractors (e.g., DINOv2 ViT-S/ViT-B) have been empirically demonstrated to yield near-optimal performance for domain adaptation without sacrificing computational efficiency or communication bandwidth (Kihara et al., 10 Sep 2025, Gatla et al., 28 Nov 2025). In few-shot and resource-constrained domains, LoRA and related adapter strategies deliver competitive accuracy while reducing the number of trainable parameters by up to 98% (Gatla et al., 28 Nov 2025).

The following table contrasts adaptation strategies:

Method	Params Updated	Catastrophic Forgetting	Data/Compute Cost
Full-model fine-tuning	All	High	High
Adapters/LoRA/Block Expansion	<5%	Low	Moderate
Frozen backbone + new head	Head only	None	Minimal

3. Theoretical Insights and Generalization Bounds

The theoretical understanding of domain-adapted FMs is informed by PAC-Bayesian generalization theory and classical domain adaptation bounds. He & Tao (He et al., 2022) model adaptation as a two-stage diffusion process, with pre-training and fine-tuning viewed as convergence of parameter distributions (Maxwell-Boltzmann distributions) under stochastic optimization. The key result is that the generalization error in the fine-tuning stage dominates, with the excess risk dictated by a domain discrepancy term involving parameter covariances and shifts between the pre-training (PT) and fine-tuning (FT) optima:

$$R(Q_\mathrm{FT}) \leq \hat R(Q_\mathrm{FT}) + \sqrt{\frac{D(Q_\mathrm{FT}\|Q_\mathrm{PT}) + \cdots }{4 N_\mathrm{FT} - 2}}$$

where $D(Q_\mathrm{FT}\|Q_\mathrm{PT})$ comprises terms measuring covariance misalignment and the parameter drift between domains. Practical remedies focus on increasing the FT dataset, regularizing to align FT and PT geometry, and employing parameter-efficient adaptations (e.g., adapters) to keep FT dynamics close to PT (He et al., 2022). Empirical results confirm that freezing large foundation model backbones and updating only adaptation heads yields more stable and robust performance under domain shift and class imbalance (Kihara et al., 10 Sep 2025).

4. Empirical Results Across Domains and Modalities

The impact of domain-adapted foundation models has been validated in multiple application domains:

• Federated and source-free adaptation: Replacing classical CNN backbones with frozen vision foundation models (e.g., DINOv2 ViT-S/ViT-B) in federated, class-imbalanced, source-free settings yields substantial MAR improvements (Office-Home: +16.6 points over ResNet-50) while reducing training and communication costs by orders of magnitude (Kihara et al., 10 Sep 2025).
• Digital dermatology: Self-supervised domain-adapted ViT-T/16 surpasses ImageNet backbones and approaches the performance of 50× larger models (e.g., MONET ViT-L/14), providing order-of-magnitude improvements in inference speed and label efficiency (Gröger et al., 2024).
• Time series analysis: Multi-domain self-supervised pre-training (e.g., TimeCLR) on a mixture of time series datasets enables Transformer-based foundation models to generalize across diverse temporal domains, outperforming domain-specific pre-training in ≈93% of downstream tasks (Yeh et al., 2023).
• Speech recognition: Large speech FMs (Conformer-XL) adapted with minimal in-domain data via residual adapters and decoders approach the performance of full fine-tuning at a fraction of the parameter and data cost (Li et al., 2023).
• Medical imaging: Parameter-efficient domain adaptation strategies, such as block expansion (BE DINORET) and LoRA-integrated multi-scale alignment (MFM-DA), yield competitive or superior few-shot performance while mitigating catastrophic forgetting and minimizing the number of updated parameters relative to specialized medical models (Zoellin et al., 2024, Jiang et al., 2 Mar 2025).

5. Design Principles, Best Practices, and Practical Guidelines

Recent research converges on a set of best practices in building and deploying domain-adapted foundation models:

• Adopt strong, frozen backbone encoders: Leverage large, self-supervised FMs (e.g., ViT-B/14, Conformer-XL) as feature extractors to maximize cross-domain invariance (Kihara et al., 10 Sep 2025, Jiang et al., 2 Mar 2025).
• Utilize parameter-efficient tuning: Apply bottleneck adapters, LoRA, or block expansion to adapt deeply without overwriting pre-trained features, ensuring data and compute efficiency and avoiding catastrophic forgetting (Gatla et al., 28 Nov 2025, Zoellin et al., 2024, Gröger et al., 2024).
• Design lightweight heads for adaptation: Restrict domain adaptation to bottleneck or classifier heads, which reduces both the size of client updates in federated or distributed settings and computational overhead (Kihara et al., 10 Sep 2025).
• Balance in-domain sampling: Use balanced or oversampled batches during (source) pre-training and adaptation to mitigate imbalance and label shift (Kihara et al., 10 Sep 2025).
• Feature storage and inference optimization: Precompute and store feature-bank outputs from frozen models, facilitating rapid deployment and minimizing redundant computation (Kihara et al., 10 Sep 2025, Gröger et al., 2024).
• Assessment and regularization: Evaluate adaptation on domain-specific benchmarks and employ regularization to reduce the discrepancy term in the theoretical generalization bound (He et al., 2022).

6. Challenges, Limitations, and Future Directions

Operational challenges persist for domain-adapted foundation models:

• Data privacy and distribution drift: Sensitive domains (healthcare, legal) may necessitate federated adaptation, differential privacy, and continual monitoring for domain drift (Chen et al., 2024).
• Catastrophic forgetting: Even parameter-efficient methods can induce forgetting of out-of-domain knowledge if adaptation is not appropriately regularized; block expansion and freezing original modules mitigate but do not eliminate this risk (Zoellin et al., 2024).
• Resource constraints: Deploying large models remains challenging in edge or low-resource settings; compact domain-specific models or plug-and-play adaptation techniques are favored (Gröger et al., 2024).
• Scalability and label efficiency: Generative and few-shot adaptation, alongside synthetic data augmentation and large-scale pseudo-labeling pipelines, are active areas of research for maximizing downstream efficiency (Chen et al., 2024, Roschkowski, 8 Jul 2025).
• Unified modularity: Architectures supporting dynamic adapter fusion, multi-domain training, and robust continual adaptation are anticipated evolutions for future foundation models (Chen et al., 2024).

Advanced research directions emphasize modular, dynamically-composable adaptation layers, multi-modal domain alignment losses, and robust domain shift diagnostics, setting the stage for routine, scalable deployment of specialized foundation models across scientific and industrial settings.

---

References:

• (Kihara et al., 10 Sep 2025) Rethinking the Backbone in Class Imbalanced Federated Source Free Domain Adaptation
• (Gröger et al., 2024) Towards Scalable Foundation Models for Digital Dermatology
• (Gatla et al., 28 Nov 2025) Tourism Question Answer System in Indian Language using Domain-Adapted Foundation Models
• (He et al., 2022) Super-model ecosystem: A domain-adaptation perspective
• (Yeh et al., 2023) Toward a Foundation Model for Time Series Data
• (Chen et al., 2024) An overview of domain-specific foundation model: key technologies, applications and challenges
• (Jiang et al., 2 Mar 2025) MFM-DA: Instance-Aware Adaptor and Hierarchical Alignment for Efficient Domain Adaptation in Medical Foundation Models
• (Zoellin et al., 2024) Block Expanded DINORET: Adapting Natural Domain Foundation Models for Retinal Imaging Without Catastrophic Forgetting