Federated Multimodal Learning

• Federated Multimodal Learning is a collaborative paradigm that trains models on diverse data types across distributed clients while preserving data privacy.
• It addresses challenges like modality heterogeneity, incomplete data, and communication inefficiency via specialized aggregation and tailored learning architectures.
• Recent systems leverage pretrained foundation models with parameter-efficient fine-tuning and meta-learning to enhance performance and scalability.

Federated Multimodal Learning (FML), also called Multimodal Federated Learning (MFL), studies collaborative learning from multimodal data under a federated protocol in which raw data remain distributed across clients rather than being centrally collected. In the literature, the multimodal side spans images, text, audio, sensor streams, video, medical records, and graph-associated modalities, while the federated side is motivated by privacy, data sovereignty, and cross-institutional collaboration. The field is not simply federated learning applied to richer inputs: modality heterogeneity, privacy heterogeneity, and communication inefficiency become first-class design constraints, and, in the “big model era,” the objective often shifts from training a narrow global model from scratch to collaboratively fine-tuning a pretrained multimodal foundation model into a vertical-domain model (Peng et al., 27 May 2025, Li et al., 2023).

1. Conceptual scope and taxonomic structure

A common organizing principle treats FML through the standard federated learning paradigms. In multimodal horizontal federated learning (HFL), clients share the same feature space or modality set but hold different subsets of samples; in multimodal vertical federated learning (VFL), clients share the same samples but each party holds different feature subsets, which in the multimodal case correspond to different modalities; and in hybrid FL, both sample space and feature space are partitioned. The 2025 survey argues that the dominant challenge changes across these regimes: modality heterogeneity is central in HFL, privacy leakage is central in VFL, and efficiency is central in hybrid FL (Peng et al., 27 May 2025).

Paradigm	Partitioning assumption	Dominant challenge
HFL	Same feature space / modality set, different samples	Modality heterogeneity
VFL	Same samples, different feature subsets / modalities	Privacy leakage
Hybrid FL	Sample and feature space both partitioned	Efficiency

A second taxonomy, emphasized in healthcare, classifies clients by modality completeness. Client Modal-Complete MMFL assumes every client has all modalities; Client Unimodal MMFL assumes each client has exactly one modality; Client Modal-Incomplete MMFL allows each client to hold an arbitrary subset of modalities. This suggests that modality completeness is orthogonal to the HFL/VFL/hybrid axis and is often the more operational description in applied deployments such as hospitals, wearables, or multimodal IoT systems (Thrasher et al., 2023).

A recurrent source of ambiguity is terminological rather than algorithmic. The acronym “FML” is also used by “Federated Mutual Learning,” a 2020 heterogeneity-oriented paradigm with a global “meme model” and a private personalized model on each client. That work addresses data, objective, and model heterogeneity, but it is not a multimodal learning framework in the sense used in later MFL literature (Shen et al., 2020).

2. Learning formulations and architectural families

At the level of optimization, a standard multimodal HFL formulation writes the global objective as

$$f(\theta) := \frac{1}{N} \sum_{m=1}^{M} N_m f_m(\theta),$$

where each client minimizes a local multimodal empirical risk over its own data and the server aggregates local parameters in a FedAvg-style manner. This formulation is conceptually simple, but the multimodal setting immediately complicates it because local models may update only modality-specific submodules, local inputs may be incomplete, and fusion architectures may not be uniform across clients (Peng et al., 27 May 2025).

Early FML systems were built around modality-specific encoders and shared latent spaces rather than foundation models. “Multimodal Federated Learning on IoT Data” proposed a semi-supervised framework in which unimodal and multimodal clients train autoencoders locally, the server aggregates modality-specific encoder/decoder components through a multimodal FedAvg rule, and a server-side classifier is trained using auxiliary labeled data from one modality. The paper introduced SplitAE- and DCCAE-based alternatives to learn shared or correlated latent representations and reported that mixed unimodal + multimodal clients could improve convergence or final $F_1$ relative to only multimodal clients (Zhao et al., 2021).

A closely related but more transfer-oriented design appears in “Federated Transfer Learning with Multimodal Data.” There, users are first grouped by identical modality availability; federated learning is then performed within groups, using supervised learning for unimodal groups and self-supervised contrastive learning for multimodal groups; finally, corresponding network parts are aggregated across groups so that multimodal users transfer cross-modal structure to unimodal users. In the reported image-audio scene-classification setting, the framework improved the audio-only case from 88.16% under an FL baseline to 92.87% under balanced distribution, while also remaining effective under unbalanced distributions (Sun, 2022).

Meta-learning introduced another architectural family. The 3FM framework, also called FedMeta-Multi-Modal, treats each client as a task with a support set $D_S^u$ and a query set $D_Q^u$, performs an inner adaptation step

$$\theta^u \leftarrow \theta - \alpha \nabla L_{D_S^u}(\theta),$$

and updates the global initialization at the server through

$$\theta \leftarrow \theta - \beta \sum_{u \in U_t} g_u.$$

This recasts multimodal federation as learning an initialization that adapts quickly to missing-modality configurations rather than merely averaging locally optimized models (Tran et al., 2023).

The “big model era” shifts the architectural center of gravity again. In “Federated Learning in Big Model Era: Domain-Specific Multimodal Large Models,” the central object is a pretrained general multimodal foundation model hosted or coordinated by the server, while each enterprise fine-tunes only selected components such as LoRA adapters, expert modules, feature-map distillation targets, bridge networks, or aligned representation layers. The server aggregates these heterogeneous contributions and redistributes a fused model, so federation becomes a model-fusion-and-redistribution workflow for turning general multimodal world knowledge into vertical-domain intelligence (Li et al., 2023).

3. Modality heterogeneity, incomplete data, and alignment

The most persistent technical problem in FML is that multimodal heterogeneity is not reducible to ordinary non-IIDness. Clients may differ in which modalities exist, in how many modalities are present, in whether modalities are aligned, and in whether features are missing inside an available modality. The 3FM experiments expose how severe this can be even in a small tri-modal benchmark: a naive baseline reached 100% accuracy in the img/sign, img/spectrogram, and img only settings, but only 8.495%, 13.349%, and 20.603% in spectrogram only, sign only, and sp/sign, respectively. Under careful meta-learning-rate tuning, 3FM improved especially in the spectrogram/sign, spectrogram only, and sign only cases, with the best reported setting at 3 clients, outer learning rate 0.001, and inner learning rate 0.00001 (Tran et al., 2023).

FedMLLM generalizes this issue to multimodal LLMs and formalizes four multimodal scenarios: Aligned Modal, Missing Modal, Cross Modal, and Hybrid Modal. Its experiments show that non-aligned settings usually degrade performance relative to aligned data, that the cross-modal scenario is the hardest and most variable, and that simple modality-agnostic prompting plus middle-layer adaptive regularization help stabilize training. A representative result reported in the paper is that, in Hybrid mode on CrisisMMD, the combined strategies improve FedAdagrad by +5.01 (Xu et al., 2024).

Two later lines of work address incompleteness more directly. FedMobile targets mobile sensing and Web of Things settings with incomplete modalities by reconstructing missing modalities in a shared latent feature space via a local generator $\omega_G^k$, aligning generated and real features with KL-based losses, and weighting client updates by contribution-aware aggregation. The paper reports robust learning even when up to 90% of modality information is missing or when data from two modalities are randomly missing, and it reports an average 4.3% improvement over AutoFed in the two-modality-missing ADM setting (Liu et al., 20 Feb 2025).

PEPSY considers a still more general setting in which each client observes a different subset of modalities and may also have missing input features within each available modality. Its solution is to learn a client-side data-missing profile $\Psi$ composed of embedding controls, which are queried by the local representation and used as reconfiguration signals during fusion. The server does not naively average these profile embeddings; instead, it aggregates them via probabilistic synchronization across clients with similar missingness patterns. The method is supported by an explicit performance bound relating output discrepancy to the data-specific contrastive loss and reports up to 36.45% performance improvement under severe data incompleteness (Nguyen et al., 27 Oct 2025).

4. Efficiency, personalization, and system optimization

Because multimodal models require dedicated encoders, projection heads, and fusion modules, resource constraints are unusually acute in federated deployment. LW-FedMML addresses this by decomposing training into stages and training only the newly added layer blocks at each stage while freezing previous layers. Across its experiments, the method reduces memory usage by up to 2.7×, computational operations by 2.4×, and total communication cost by 2.3×, while remaining competitive with end-to-end FedMML. Its progressive variant, Prog-FedMML, is less efficient but can exceed end-to-end FedMML on some tasks, including COCO retrieval and ADVANCE audiovisual scene recognition (Tun et al., 2024).

FlexMod treats local multimodal training as a resource-allocation problem rather than a uniform schedule over modalities. It estimates modality quality via prototype learning, modality importance via Shapley values, and uses DDPG to adapt the trade-off weight $\beta^r$ in the per-round utility. The method schedules modality combinations rather than single modalities, reflecting the paper’s observation that training combinations in parallel is faster than sequentially training separate modalities. On UCI-HAR, FlexMod reaches test accuracy 0.6 in 12 rounds, whereas the Entire Update baseline needs 20 rounds (Bian et al., 2024).

Personalization can also be moved into aggregation itself. In “Aggregation Design for Personalized Federated Multi-Modal Learning over Wireless Networks,” the server maintains a personalized model for each device and learns modality-specific aggregation coefficients $\xi^m_{k,k',t}$, then uses these coefficients together with wireless channel state to decide which parameters are uploaded. On CREMA-D under Non-IID-1, the proposed method reports 65.31%, compared with 48.90% for FedAvg, 51.02% for FedProx, 57.53% for FedFomo, and 58.65% for FedAMP; with $F_1$0, training time is 5.47×10$F_1$1 s versus 6.50×10$F_1$2 s for FedAvg (Yin et al., 2024).

Wireless multimodal federation has also been studied directly under latency and bandwidth constraints. JCSBA uses decision-level fusion, adds unimodal loss functions to both the training objective and local update loss, derives a closed-form upper bound related to client and modality scheduling, and jointly optimizes client selection and bandwidth allocation. The paper reports improvements of 4.06% in multimodal accuracy and 2.73% in unimodal accuracy over conventional algorithms (Han et al., 16 Sep 2025). A related UAV-assisted framework optimizes sensing scheduling, power control, trajectory, and BS resource management, and reports latency reductions of 29.39% versus UAV-SS-PC, 11.96% versus UAV-T-RA, and 42.49% versus BS-RA, while also improving training performance under IID and non-IID data (Shaon et al., 2 Oct 2025).

For unified multimodal models and multimodal LLMs, efficiency is typically achieved through PEFT. FedUMM instantiates federation for BLIP3o on NVIDIA FLARE, freezes the backbone, trains only LoRA adapters with rank $F_1$3 and scaling factor $F_1$4, and aggregates only adapter parameters with a quality-aware factor. On VQA v2, it reports 0.094 GB per-round communication versus 28.6 GB for full fine-tuning-based FL, a 99.7% reduction, while achieving 80.2 accuracy against 79.5 for full FedAvg and retaining about 97.3% of centralized LoRA performance (Su et al., 21 Jan 2026).

5. Privacy, security, traceability, and post-training control

In FML, privacy protection is partly architectural and partly procedural. A recurring design rule is that raw multimodal data remain on local devices or enterprise infrastructure and only model-related information is exchanged. The big-model-era framework makes this explicit and lists several protections: parameter-efficient fine-tuning with selective upload, feature-map or embedding transfer instead of sample sharing, TLS for transmission, identity and permission management, secure aggregation via multi-party secure computation, and possible use of MPC, homomorphic encryption, differential privacy, and trusted execution environments. It also highlights inference-time leakage risks specific to generative models and recommends filtering sensitive training data plus output controls such as blacklists, post-processing, and conditional generation constraints (Li et al., 2023).

This does not imply that federation alone yields formal privacy. One later unified-multimodal-model study states explicitly that FL provides only implicit privacy and that no formal differential privacy guarantee is built into the method (Su et al., 21 Jan 2026). That position is consistent with the survey’s treatment of multimodal VFL and hybrid FL, where exchanged embeddings and gradients can leak raw features, labels, relation structure, or model behavior; the survey therefore identifies privacy leakage, rather than ordinary statistical heterogeneity, as the defining VFL problem in multimodal federation (Peng et al., 27 May 2025).

Beyond confidentiality, recent FML work introduces traceability as a model property. FedLAB addresses federated multimodal graph foundation learning by replacing opaque fused embeddings with typed hierarchical codebooks for modality evidence, node semantics, and topology context. Clients upload only aggregate posterior statistics—usage counts and posterior-weighted local centers—and the server refines global codebooks through semantic barycenters rather than raw features or graph structure. Across 10 benchmarks and 6 downstream tasks, the paper reports an average improvement of 4.38% over the strongest baseline and a maximum gain of 7.53%, while preserving what it calls a native semantic trace interface (Chen et al., 30 Jun 2026).

A further extension of post-training control is unlearning. EASE studies federated multimodal contrastive unlearning for CLIP-style image-text models and argues that forgotten knowledge persists through three residual anchors: bilinear cross-modal coupling, principal-angle subspace entanglement, and continued federated updates. It answers these with bilateral knowledge excision, gradient subspace decomposition, and a direction-selective Forget Lock. Under client unlearning on Flickr30K with CLIP-B/32, EASE is reported to match the retrain reference to within 0.2 R@1 on the forget side and 4.2 R@1 on the retain side (Ding et al., 1 May 2026).

6. Domain-specific deployments and research trajectory

A concrete large-scale deployment scenario appears in the smart-city case study of “Federated Learning in Big Model Era: Domain-Specific Multimodal Large Models.” Three organizations collaborate: ENN Group contributes energy security data with more than 260k visual-textual samples; Unicom (Shanghai) contributes urban management data with more than 150k visual-textual samples, partially annotated; and WiNDAKA contributes community security data with about 110k image samples, with annotations available. The system is deployed on the ENNEW Federated Learning platform, supports distributed scheduling, data processing, algorithm execution, security, and performance monitoring, and iterates through a train–filter–retrain data-quality loop in which the federated model itself is used to identify low-quality samples and generate or expand captions. Preliminary results are described qualitatively rather than through a finalized benchmark table, but the paper claims better image understanding from multiple perspectives and more professional and informative captions for city safety operation management (Li et al., 2023).

Healthcare provides a second major application frontier. The review of multimodal federated learning in healthcare treats diagnosis, segmentation, MRI reconstruction and synthesis, survival analysis, and personalized monitoring as core task families, with data spanning CT, MRI, X-ray, ultrasound, PET, histopathology, EHRs, genomics, ECG, wearable signals, and IoT sensors. It also shows that healthcare methods occupy all three client-modality regimes—modal-complete, unimodal, and modal-incomplete—and that model designs range from straightforward FedAvg-based fusion to modality normalization, pseudo-modality generation, representation sharing, and prediction stacking (Thrasher et al., 2023).

The current research trajectory points toward increasingly heterogeneous and increasingly modular systems. The survey highlights lightweight modality-aware models, stronger cryptographic and privacy mechanisms, self-supervised and semi-supervised learning, personalization, cross-modal knowledge transfer, prompt-based learning, and interpretability as principal future directions (Peng et al., 27 May 2025). A plausible implication is that FML is evolving from a narrowly defined aggregation problem into a broader systems discipline spanning multimodal representation learning, foundation-model adaptation, wireless scheduling, semantic auditability, and deletion control under persistent data isolation.