Multimodal Federated Learning

• Multimodal Federated Learning is a framework that trains models over distributed clients holding various modalities like image, text, and audio without sharing raw data.
• It addresses key challenges including modality heterogeneity, privacy leakage, and communication inefficiency through horizontal, vertical, and hybrid FL paradigms.
• Recent research highlights modular encoder–fusion–classifier designs, parameter-efficient adaptations, and representation alignment to optimize multimodal performance.

Searching arXiv for recent multimodal federated learning papers and benchmarks to ground the article. Multimodal Federated Learning (MFL) combines multimodal learning with federated learning to train models over distributed clients holding modalities such as image, audio, text, and sensor signals without centralizing raw data. Relative to unimodal FL and centralized multimodal learning, MFL must cope with fragmented cross-modal structure, modality heterogeneity, privacy heterogeneity, and communication inefficiency, and it is commonly organized through horizontal FL (HFL), vertical FL (VFL), and hybrid FL paradigms rather than through a single canonical training setup (Peng et al., 27 May 2025).

1. Paradigms, objectives, and formal scope

A useful starting point is the paradigm-oriented taxonomy that separates MFL by how samples and features are partitioned. In multimodal HFL, clients share the same nominal feature space but own different samples, and the standard global objective remains

$$f(\theta) := \frac{1}{N} \sum_{m=1}^{M} N_m f_m(\theta), \qquad f_m(\theta) := \frac{1}{N_m} \sum_{i=1}^{N_m} \ell(\theta; x_m^i, y_m^i).$$

In multimodal VFL, different parties hold different feature subsets or modalities of the same samples, and the objective becomes

$$f(\Theta) := \frac{1}{N} \sum_{i=1}^{N} \ell\left( \theta_0 \circ \{h_k(\theta_k; x_k^i)\}_{k=1}^K; y^i \right).$$

Hybrid FL composes silo-local VFL with inter-silo HFL through

$$f(\Theta) := \frac{1}{N} \sum_{m=1}^M N_m f_m(\Theta).$$

The survey associates these three regimes with three dominant challenge classes: modality heterogeneity in HFL, privacy leakage in VFL, and efficiency in hybrid FL (Peng et al., 27 May 2025).

Paradigm	Partition pattern	Prominent challenge
HFL	Same feature space, different samples	Modality heterogeneity
VFL	Same samples, different modalities/features	Privacy leakage
Hybrid FL	Both sample and feature partitioning	Efficiency

This taxonomy is not merely organizational. It distinguishes settings in which clients all possess multimodal examples from settings in which clients hold only partial modalities, or even different tasks, while still contributing to a common federated objective. Benchmarks such as FedMultimodal instantiate the HFL case with local multimodal encoder–fusion–classifier models and server-side aggregation via FedAvg, FedProx, FedRS, or FedOpt (Feng et al., 2023). By contrast, transfer vision–language FL in FedCola uses three client types—image clients, text clients, and image–text clients—and trains a shared global multimodal transformer under explicit task and modality mismatch (Sun et al., 2024).

2. Heterogeneity, missing modalities, and modality imbalance

MFL inherits classical non-IID difficulty and adds modality-specific heterogeneity. FedMultimodal emphasizes label skew, missing modalities, missing labels, and erroneous labels as distinct corruption modes for multimodal FL, with synthetic non-IID partitions generated by Dirichlet distributions using $\alpha \in \{0.1, 5.0\}$ and missing modalities simulated by a Bernoulli process with rate $q \in \{0.1,0.2,0.3,0.4,0.5\}$ (Feng et al., 2023). In CLIP-based personalized FL, FedDLP simulates non-IID clients through Dirichlet partitions with concentration parameter $\beta\in\{0.1,0.01\}$, where small $\beta$ produces strong label imbalance and missing classes per client (Nguyen et al., 10 Mar 2025).

A central difficulty is modality heterogeneity in the strict sense: different clients may have different subsets of modalities. The 3FM framework makes this explicit with aligned triples $$(x^{\text{img}}, x^{\text{spect}}, x^{\text{sign}}, y)$$ and six missing-modality configurations—img+sign, spect+sign, img+spect, img only, spect only, and sign only—while evaluating on full modality at test time (Tran et al., 2023). FedMFS and mmFedMC likewise assume client-specific modality sets $$\mathbb{D}^k = \{\mathcal{D}^k_1,\dots,\mathcal{D}^k_{M_k}\}$$, so global learning must proceed even when some clients lack sensors or modalities that others possess (Yuan et al., 2023, Yuan et al., 2024).

A second difficulty is modality imbalance. FedCMI defines modality imbalance as the phenomenon in which different modalities learn at different speeds and a dominant modality suppresses a weak modality during multimodal optimization; its gradient analysis shows that larger correct-class logits from the dominant modality can dominate the fused cross-entropy gradient (Fan et al., 2023). The balanced modality-selection work reports the same problem at the federation level as global modality-level bias: on CREMA-D, audio substantially outperforms visual, and diversity-based client selection improves the dominant audio modality far more than the weak visual modality, while randomly discarding one modality on some clients can even improve multimodal global accuracy when it reduces the audio–visual gap (Fan et al., 2023). Together these results indicate that MFL difficulty is not exhausted by client-level non-IIDness; the modality axis itself can become a source of optimization bias.

3. Architectural patterns and alignment mechanisms

A recurrent architectural pattern is the modular encoder–fusion–classifier design. FlexMod formulates the global model as $\Theta = \{\theta^0,\theta^1,\dots,\theta^M\}$, where $$f(\Theta) := \frac{1}{N} \sum_{i=1}^{N} \ell\left( \theta_0 \circ \{h_k(\theta_k; x_k^i)\}_{k=1}^K; y^i \right).$$0 are modality-specific encoders producing $$f(\Theta) := \frac{1}{N} \sum_{i=1}^{N} \ell\left( \theta_0 \circ \{h_k(\theta_k; x_k^i)\}_{k=1}^K; y^i \right).$$1, and $$f(\Theta) := \frac{1}{N} \sum_{i=1}^{N} \ell\left( \theta_0 \circ \{h_k(\theta_k; x_k^i)\}_{k=1}^K; y^i \right).$$2 is a shared header that fuses them into $$f(\Theta) := \frac{1}{N} \sum_{i=1}^{N} \ell\left( \theta_0 \circ \{h_k(\theta_k; x_k^i)\}_{k=1}^K; y^i \right).$$3 (Bian et al., 2024). FedMultimodal instantiates this with Conv+RNN or RNN-only encoders, followed by either concatenation-based fusion or a lightweight attention mechanism

$$f(\Theta) := \frac{1}{N} \sum_{i=1}^{N} \ell\left( \theta_0 \circ \{h_k(\theta_k; x_k^i)\}_{k=1}^K; y^i \right).$$4

before a classifier head (Feng et al., 2023). In 3FM, separate image, spectrogram, and sign-language branches are flattened, concatenated, and passed through two fully connected layers, with missing branches muted to zeros (Tran et al., 2023).

A second pattern is parameter-efficient adaptation of large multimodal foundation models. FedDLP freezes the CLIP backbone $$f(\Theta) := \frac{1}{N} \sum_{i=1}^{N} \ell\left( \theta_0 \circ \{h_k(\theta_k; x_k^i)\}_{k=1}^K; y^i \right).$$5, retains the vision–language alignment learned during pretraining, and fine-tunes either the text encoder or the image encoder with two LoRA adapters per client: a larger local adapter $$f(\Theta) := \frac{1}{N} \sum_{i=1}^{N} \ell\left( \theta_0 \circ \{h_k(\theta_k; x_k^i)\}_{k=1}^K; y^i \right).$$6 for personalization and a smaller global adapter $$f(\Theta) := \frac{1}{N} \sum_{i=1}^{N} \ell\left( \theta_0 \circ \{h_k(\theta_k; x_k^i)\}_{k=1}^K; y^i \right).$$7 for federated aggregation (Nguyen et al., 10 Mar 2025). FedCola extends transformer-based vision–language FL by mixing local and out-modality transformer blocks on uni-modal clients through learnable gates, effectively using

$$f(\Theta) := \frac{1}{N} \sum_{i=1}^{N} \ell\left( \theta_0 \circ \{h_k(\theta_k; x_k^i)\}_{k=1}^K; y^i \right).$$8

inside transformer layers while preserving a standard FL communication interface at upload time (Sun et al., 2024).

A third pattern replaces parameter sharing with representation alignment. CreamFL avoids parameter averaging across heterogeneous multimodal, image-only, and text-only clients by exchanging representations on a public image–text dataset and performing global–local contrastive aggregation; its local objective combines task loss with inter-modal and intra-modal contrastive regularization (Yu et al., 2023). FedMEKT instead uses a small multimodal proxy dataset and a distillation-based multimodal embedding knowledge transfer pipeline built from local multimodal autoencoders, a generalized multimodal autoencoder on the server, and a generalized classifier (Le et al., 2023). CoMFed further compresses this idea into class-wise latent-space consensus by learning client-specific projection matrices $$f(\Theta) := \frac{1}{N} \sum_{i=1}^{N} \ell\left( \theta_0 \circ \{h_k(\theta_k; x_k^i)\}_{k=1}^K; y^i \right).$$9 and communicating only projected class means $$f(\Theta) := \frac{1}{N} \sum_{m=1}^M N_m f_m(\Theta).$$0 (Badi et al., 19 Mar 2026).

4. Personalization and communication-efficient optimization

Personalization in MFL is often implemented by asymmetric sharing. FedDLP separates personalization and generalization through a larger local LoRA adapter, a smaller global LoRA adapter, alternating freezing, bi-directional distillation, and SoRA-style pruning on the local adapter; only the global adapter is aggregated by FedAvg, while the local adapter remains private (Nguyen et al., 10 Mar 2025). EPFL in FHBench exploits LoRA asymmetry differently: it aggregates only $$f(\Theta) := \frac{1}{N} \sum_{m=1}^M N_m f_m(\Theta).$$1 matrices in a similarity-weight