Federated Foundation Models
- Federated Foundation Models are large-scale pretrained models (e.g., GPT-4, CLIP) adapted using federated learning to preserve data privacy.
- These models employ techniques like federated fine-tuning, adapter tuning, and modular parameter sharing to balance global generalization with local specialization.
- Key applications include biomedical imaging, time-series forecasting, and recommendation systems, showcasing their potential in managing heterogeneous data.
Federated Foundation Models (FFMs) are foundation models—especially LLMs and vision-LLMs—whose training, continual updating, or fine-tuning is carried out collaboratively across decentralized data holders without sharing raw data. In the current literature, the term denotes the convergence of two developments: large, broadly pretrained foundation models with strong transfer ability, and federated learning as a privacy-preserving mechanism for distributed adaptation over private, fragmented, and often heterogeneous data. The concept is used most explicitly in biomedical healthcare, but it also extends to recommendation, time-series forecasting, wireless edge systems, and embodied AI (Li et al., 2024, Yu et al., 2023).
1. Conceptual foundations and scope
Foundation models are described as large-scale deep learning models with broad transfer ability, usually based on modern architectures such as Transformers and trained on massive corpora before downstream adaptation. The literature repeatedly attributes their effectiveness to large-scale pretraining plus flexible adaptation via unsupervised pretraining, self-supervised learning, instruction fine-tuning, and reinforcement learning from human feedback. Representative language-side examples include ChatGPT, GPT-4, LLaMA, BERT, T5, PaLM, and Bard; representative multimodal examples include CLIP, DALL·E 2, Stable Diffusion, and Segment Anything (Li et al., 2024).
Federated learning, by contrast, is the conventional decentralized training setup in which multiple clients collaboratively train or adapt a model under server coordination while keeping local data decentralized. Within this frame, FFMs are not a single algorithmic recipe but a lifecycle-level paradigm: federated pre-training, federated fine-tuning, federated prompt tuning, deployment, and continual or lifelong learning all appear as legitimate stages at which federation can be inserted into the lifespan of a foundation model (Yu et al., 2023).
A central conceptual clarification in the survey literature is that “federating an FM” is not one thing. It can mean full distributed training, federated fine-tuning of a pretrained backbone, federated adapter tuning, federated prompt learning, federated distillation, or multimodal feature-extractor federation. This distinction matters because the systems burden, privacy surface, and personalization capacity differ sharply across these variants (Li et al., 2024).
The field also rejects a narrow reading of FFMs as merely “FL with a larger model.” Once the model is a foundation model, the design space changes: full-parameter sharing often becomes unrealistic, parameter-efficient adaptation becomes central, multimodal models become more important, and deployment questions expand to include prompt learning, zero-shot transfer, continual adaptation, and governance. In that sense, FFMs denote both a technical class of distributed systems and a socio-technical response to data fragmentation and privacy constraints (Yu et al., 2023).
2. Architectural patterns and adaptation mechanisms
The architectural literature organizes FFMs around both model type and adaptation mechanism. In biomedical work, the dominant taxonomy divides models into LLMs and vision-LLMs, and further separates biomedical vision-LLMs into dual-encoder, fusion-encoder, and hierarchical structures. Dual encoders separately encode image and text before alignment, as in ConVIRT, GLoRIA, MedCLIP, and CheXZero; fusion encoders combine modalities early for deeper multimodal reasoning, as in VisualBERT, MedViLL, ARL, and Llava-med; hierarchical structures align multiple abstraction levels and include some diffusion-style systems such as RoentGen (Li et al., 2024).
Concrete FFM systems instantiate different points in this design space. Resource-aware Federated Foundation Models (RaFFM) introduce salient parameter prioritization and high-performance subnetwork extraction so that a single transformer FM can be scaled to heterogeneous client resource constraints during both optimization and deployment (Yu et al., 2023). Federated Proxy Fine-Tuning (FedPFT) constructs a proxy sub-FM through layer-wise compression and then aligns the proxy with the full FM via layer-level distillation before federated fine-tuning and neuron-level distillation during training (Peng et al., 2024). Over wireless networks, split federated LoRA places the computationally intensive encoder at the edge server while retaining the embedding and task modules at edge devices, thereby coupling foundation-model adaptation to device scheduling and bandwidth allocation (Wang et al., 2024).
Parameter-efficient adaptation is the dominant practical motif. FedMSA freezes the pretrained SAM encoder, inserts two bottleneck adapters into each ViT block, and federates the adapter weights together with decoder parameters for medical segmentation (Liu et al., 2024). FedMS combines a global expert and a local expert into a Mixture of Foundation Models and uses Sparsely Activated LoRA to progressively activate low-rank modules during training (Wu et al., 2023). Time-FFM learns global encoders and local prediction heads for time-series forecasting with a frozen GPT-2 backbone and a prompt adaptation module that selects prompt tokens dynamically from LM-derived prototypes (2405.14252).
Other systems shift the FM entirely to the server side. FedD2P never deploys the FM on resource-constrained clients; instead, a server-hosted CLIP model is adapted through a prompt generator trained from aggregated class-wise client knowledge and linguistic class descriptions, and the resulting FM outputs are distilled back into lightweight client CNNs (Atapour et al., 2024). For semi-supervised object detection, FedMox freezes a DINOv2/ViT-Adapter backbone, trains only the task head, and enlarges head capacity through a sparse Mixture-of-Experts architecture with a spatial router that can handle server/client resolution mismatch (Sun et al., 22 Aug 2025).
Taken together, these systems show that FFMs are typically modular rather than monolithic. Encoders, adapters, prompts, low-rank matrices, experts, and task heads are the units of communication and personalization. This suggests that the defining problem in FFMs is less “how to federate a model” than “which parts of a pretrained model should be shared, frozen, specialized, compressed, or routed.”
3. Application domains
Biomedical healthcare is the most fully developed application domain. The motivation is unusually strong because medical data are highly sensitive, fragmented across institutions, and intrinsically multimodal: electronic health records, clinical reports, radiology reports, chest X-rays, MRIs, CT scans, pathology images, genomic sequences, protein data, and social-media-derived health text all appear in the same ecosystem. The literature discusses disease prediction, patient phenotyping, triage or discharge recommendation, clinical note summarization, medical text simplification, medical question answering, clinical trial matching, future medical event prediction, medical image classification, report-guided diagnosis, zero-shot pathology detection, segmentation, and multimodal report generation as target FFM use cases (Li et al., 2024).
Medical segmentation provides one of the clearest empirical case studies. FedFMS adapts SAM in two forms—FedSAM and adapter-based FedMSA—and reports performance comparable to centralized training on four non-IID federated datasets. On Brain Tumor, for example, average Dice/IoU were $0.872/0.832$ for FedSAM and $0.875/0.835$ for FedMSA, compared with $0.857/0.817$ for FedU-Net; on Prostate Cancer, FedSAM reached $0.819/0.784$, while FedMSA-L reached $0.829/0.796$ (Liu et al., 2024).
Time-series forecasting is another domain where FFMs serve as cross-domain knowledge-sharing systems rather than merely privacy wrappers. Time-FFM converts time series into token-like patch embeddings, learns a prompt adaptation module over a frozen GPT-2 backbone, and uses personalized federated training with global encoders and local prediction heads. Its reported average performance across datasets was and , slightly better than UniTime at $0.559/0.437$, while also supporting few-shot and zero-shot settings (2405.14252).
Recommendation is treated differently: not as a benchmark where FFMs have already converged to a standard algorithm, but as a conceptual testbed for the generalization–personalization tension. In that literature, FFMs decompose a recommender into a pretrained FM backbone, shared federated adaptation, and user-specific adaptation, with recommendation data viewed as inherently decentralized, highly sensitive, and deeply personalized (Li et al., 13 Jun 2025).
Edge vision and embodied systems extend FFMs toward deployment realism. FedMox adapts a frozen visual foundation model for federated object detection under Practical Semi-Supervised Federated Learning, where the server has limited labeled high-resolution data and clients have only unlabeled low-resolution data; on BDD100K with , it reported total , compared with $0.875/0.835$0 for FedAvg and $0.875/0.835$1 for backbone-frozen centralized supervised learning (Sun et al., 22 Aug 2025). For embodied AI, conceptual work proposes multi-modal multi-task FFMs with modality encoders, mixture-of-modality experts, mixture-of-task experts, task heads, prompts, and adapters, all coordinated at the wireless edge; parallel work on hierarchical FFM systems maps modality encoders, prompts, MoEs, adapters, and task heads onto fog/edge hierarchies with optional D2D/P2P exchange (Borazjani et al., 16 May 2025, Abdisarabshali et al., 3 Sep 2025).
4. Mathematical and systems formulations
The mathematical treatment of FFMs remains uneven. A notable feature of the biomedical survey literature is that it does not provide a standard federated objective such as $0.875/0.835$2 or an explicit FedAvg formula; instead, its only equations describe the Transformer backbone: $0.875/0.835$3 with self-attention written as
$0.875/0.835$4
The survey explicitly states that it does not supply formal FL or FFM optimization objectives (Li et al., 2024).
Framework papers fill part of that gap with generic federated updates. One widely reused formulation is the FedAvg-style aggregation
$0.875/0.835$5
which is presented as the core update in a general FFM optimization process spanning pre-training, fine-tuning, prompt tuning, and continual learning (Yu et al., 2023).
Task-specific systems then introduce more structured decompositions. In FedFMS, the federated state is written either as
$0.875/0.835$6
for full SAM fine-tuning or as
$0.875/0.835$7
for adapter-based tuning, with communication per round quantified as $0.875/0.835$8 when $0.875/0.835$9 is the number of learnable parameters (Liu et al., 2024). In recommendation, FFM personalization is formalized by separating the pretrained backbone $0.857/0.817$0, shared federated adaptation $0.857/0.817$1, and user-specific adaptation $0.857/0.817$2, so that optimization explicitly trades off shared knowledge transfer and local personality-aware specialization (Li et al., 13 Jun 2025).
Wireless systems introduce yet another layer of formalism. In split federated LoRA, the uplink rate for device $0.857/0.817$3 at round $0.857/0.817$4 is
$0.857/0.817$5
and the per-round delay is governed by
$0.857/0.817$6
That work couples the convergence of shared LoRA parameters to wireless device scheduling and bandwidth allocation, making communication control a first-class part of the FFM optimization problem rather than a mere implementation detail (Wang et al., 2024).
A consistent theme across these formulations is that FFMs rarely reduce to a single global parameter vector. They are usually decomposed into frozen pretrained weights, shared trainable adaptation modules, and local or personalized components. This modularization is not only an efficiency device; it is the mathematical expression of how FFMs separate reusable foundation knowledge from local specialization.
5. Privacy, security, fairness, and trustworthiness
FFMs are routinely described as privacy-preserving relative to centralized raw-data pooling, but the literature is careful not to treat federation as a complete privacy solution. Model updates can leak sensitive information, and federated systems remain exposed to reconstruction attacks, poisoning, adversarial manipulation, robustness failures under heterogeneity, automation bias, and bias amplification inherited from both local datasets and pretrained models (Li et al., 2024).
Security risks become especially acute when a foundation model is used upstream in the federated pipeline. In heterogeneous federated learning, Fed-EBD shows that a compromised FM can inject a backdoor through the synthetic public dataset used for initialization and distillation, so that all clients behave honestly yet still learn the trigger. On SST-2 under cross-device IID heterogeneous settings, Fed-EBD achieved $0.857/0.817$7 ACC and $0.857/0.817$8 ASR, compared with $0.857/0.817$9 ACC and $0.819/0.784$0 ASR for vanilla FL and $0.819/0.784$1 ACC and $0.819/0.784$2 ASR for CBD-FL (Li et al., 2023). This reframes the threat surface of FFMs as a supply-chain problem involving FM provenance, prompt integrity, and synthetic-data curation.
Fairness work argues that group fairness in FFMs is not separable across sensitive attributes. A CLIP-based federated vision-language system extends fairness regularization to multiple sensitive attributes through
$0.819/0.784$3
with demographic parity and equalized odds defined per attribute and interpreted through causal discovery and intervention. Empirically, the paper shows that mitigating one attribute can worsen another; in smiling detection on CelebA, mitigating gender only gave $0.819/0.784$4 while worsening age fairness to $0.819/0.784$5 (Yang et al., 23 Jun 2025).
The recommendation literature extends privacy concerns from record confidentiality to semantic privacy. There, “preserving personality” means not only keeping $0.819/0.784$6 local but designing the architecture and aggregation process so that user-specific modules encode long-term traits without making those traits easily inferable from $0.819/0.784$7 or $0.819/0.784$8. This position explicitly argues that standard differential privacy may be insufficient because the leakage target is a latent semantic property rather than a single record (Li et al., 13 Jun 2025).
Embodied AI work adds a deployment-centered trustworthiness layer. Proposed safeguards include module-aware differential privacy, where different modules receive different noise levels, and federated behavioral shadow buffers, in which updated models run in shadow mode or digital twins before affecting live control (Borazjani et al., 16 May 2025). A plausible implication is that trustworthiness in FFMs is not reducible to one mechanism: privacy, fairness, robustness, interpretability, and safe rollout are entangled because the model is simultaneously large, modular, adaptive, and distributed.
6. Research agenda and unresolved questions
The current literature portrays FFMs as an emerging systems field rather than a finished algorithmic family. In healthcare, the most persistent open problems are model scaling, communication efficiency, non-IID data, multimodal alignment, domain adaptation, privacy leakage, robustness, fairness, and regulation. The same survey argues that FM-scale deployment demands infrastructures better tailored to large models than current frameworks such as Flower, FedML, FATE, and FederatedScope, and that benchmark suites and evaluation protocols remain largely agenda items rather than completed community standards (Li et al., 2024).
More general framework papers identify the same pattern across the FM lifecycle. They point toward federated pre-training, federated fine-tuning, federated prompt tuning, collaborative model compression, robust model fusion, federated multi-task learning, Federated Retrieval Augmented Generation, and continual or lifelong adaptation as core directions, while emphasizing that the future feasibility of FFMs depends heavily on edge hardware and communication efficiency (Yu et al., 2023).
Wireless and austere-environment work broadens the agenda further. In harsh environments, FFMs are proposed as communication-aware, decentralized or hierarchical systems that must survive intermittent connectivity, noisy sensing, dynamic topology, and strict energy budgets; suggested directions include contact-triggered aggregation, age-of-update weighting, hierarchical or semi-decentralized topologies, and even unconventional communication media such as acoustic or pressure-wave links (Chen et al., 2 Sep 2025). Hierarchical FFM work for multi-modal multi-task systems adds two overlooked heterogeneity dimensions—heterogeneity in collected modalities and heterogeneity in executed tasks—and proposes non-uniform module circulation, localized cooperative training, and optional D2D/P2P relaying across fog/edge layers (Abdisarabshali et al., 3 Sep 2025).
Embodied AI research suggests that these pressures intensify when agents differ in morphology, sensor suites, and user context. Its EMBODY framework—Embodiment heterogeneity, Modality richness and imbalance, Bandwidth and compute constraints, On-device continual learning, Distributed control and autonomy, and Yielding safety, privacy, and personalization—treats FFMs as edge systems that must balance cross-embodiment generalization against local adaptation, with evaluation spanning task success, safety violations, energy-to-performance ratio, forward and backward transfer, behavior divergence, and privacy-preserving utility loss (Borazjani et al., 16 May 2025).
Taken together, the literature supports a broad conclusion: FFMs are not merely centralized foundation models deployed at the edge, nor standard federated learning with larger parameter counts. They are modular distributed systems that attempt to reconcile pretrained generality with decentralized adaptation, usually through frozen backbones plus selective trainable components, and their unresolved questions are as much about system design, governance, and evaluation as about optimization.