---
title: Multimodal Federated Learning
url: https://www.emergentmind.com/topics/multimodal-federated-learning-fl
type: topic
---

# Multimodal Federated Learning

Searching arXiv for recent multimodal federated learning papers and benchmarks to ground the article.
Multimodal Federated Learning (MFL) combines multimodal learning with federated learning to train models over distributed clients holding modalities such as image, audio, text, and sensor signals without centralizing raw data. Relative to unimodal FL and centralized multimodal learning, MFL must cope with fragmented cross-modal structure, modality heterogeneity, privacy heterogeneity, and communication inefficiency, and it is commonly organized through horizontal FL (HFL), vertical FL (VFL), and hybrid FL paradigms rather than through a single canonical training setup [2505.21792].

## 1. Paradigms, objectives, and formal scope

A useful starting point is the paradigm-oriented taxonomy that separates MFL by how samples and features are partitioned. In multimodal HFL, clients share the same nominal feature space but own different samples, and the standard global objective remains
\[
f(\theta) := \frac{1}{N} \sum_{m=1}^{M} N_m f_m(\theta),
\qquad
f_m(\theta) := \frac{1}{N_m} \sum_{i=1}^{N_m} \ell(\theta; x_m^i, y_m^i).
\]
In multimodal VFL, different parties hold different feature subsets or modalities of the same samples, and the objective becomes
\[
f(\Theta) := \frac{1}{N} \sum_{i=1}^{N} \ell\left( \theta_0 \circ \{h_k(\theta_k; x_k^i)\}_{k=1}^K; y^i \right).
\]
Hybrid FL composes silo-local VFL with inter-silo HFL through
\[
f(\Theta) := \frac{1}{N} \sum_{m=1}^M N_m f_m(\Theta).
\]
The survey associates these three regimes with three dominant challenge classes: modality heterogeneity in HFL, privacy leakage in VFL, and efficiency in hybrid FL [2505.21792].

| Paradigm | Partition pattern | Prominent challenge |
|---|---|---|
| HFL | Same feature space, different samples | Modality heterogeneity |
| VFL | Same samples, different modalities/features | Privacy leakage |
| Hybrid FL | Both sample and feature partitioning | Efficiency |

This taxonomy is not merely organizational. It distinguishes settings in which clients all possess multimodal examples from settings in which clients hold only partial modalities, or even different tasks, while still contributing to a common federated objective. Benchmarks such as FedMultimodal instantiate the HFL case with local multimodal encoder–fusion–classifier models and server-side aggregation via FedAvg, FedProx, FedRS, or FedOpt [2306.09486]. By contrast, transfer vision–language FL in FedCola uses three client types—image clients, text clients, and image–text clients—and trains a shared global multimodal transformer under explicit task and modality mismatch [2404.12467].

## 2. Heterogeneity, missing modalities, and modality imbalance

MFL inherits classical non-IID difficulty and adds modality-specific heterogeneity. FedMultimodal emphasizes label skew, missing modalities, missing labels, and erroneous labels as distinct corruption modes for multimodal FL, with synthetic non-IID partitions generated by Dirichlet distributions using \(\alpha \in \{0.1, 5.0\}\) and missing modalities simulated by a Bernoulli process with rate \(q \in \{0.1,0.2,0.3,0.4,0.5\}\) [2306.09486]. In CLIP-based personalized FL, FedDLP simulates non-IID clients through Dirichlet partitions with concentration parameter \(\beta\in\{0.1,0.01\}\), where small \(\beta\) produces strong label imbalance and missing classes per client [2503.07552].

A central difficulty is modality heterogeneity in the strict sense: different clients may have different subsets of modalities. The 3FM framework makes this explicit with aligned triples \((x^{\text{img}}, x^{\text{spect}}, x^{\text{sign}}, y)\) and six missing-modality configurations—img+sign, spect+sign, img+spect, img only, spect only, and sign only—while evaluating on full modality at test time [2312.10179]. FedMFS and mmFedMC likewise assume client-specific modality sets \(\mathbb{D}^k = \{\mathcal{D}^k_1,\dots,\mathcal{D}^k_{M_k}\}\), so global learning must proceed even when some clients lack sensors or modalities that others possess [2310.07048], [2401.16685].

A second difficulty is modality imbalance. FedCMI defines modality imbalance as the phenomenon in which different modalities learn at different speeds and a dominant modality suppresses a weak modality during multimodal optimization; its gradient analysis shows that larger correct-class logits from the dominant modality can dominate the fused cross-entropy gradient [2401.00894]. The balanced modality-selection work reports the same problem at the federation level as global modality-level bias: on CREMA-D, audio substantially outperforms visual, and diversity-based client selection improves the dominant audio modality far more than the weak visual modality, while randomly discarding one modality on some clients can even improve multimodal global accuracy when it reduces the audio–visual gap [2401.00403]. Together these results indicate that MFL difficulty is not exhausted by client-level non-IIDness; the modality axis itself can become a source of optimization bias.

## 3. Architectural patterns and alignment mechanisms

A recurrent architectural pattern is the modular encoder–fusion–classifier design. FlexMod formulates the global model as \(\Theta = \{\theta^0,\theta^1,\dots,\theta^M\}\), where \(\theta^m\) are modality-specific encoders producing \(Z_n^m = \theta^m(X_n^m)\), and \(\theta^0\) is a shared header that fuses them into \(\hat{Y}_n = \theta^0(Z_n^1,\dots,Z_n^M)\) [2408.06549]. FedMultimodal instantiates this with Conv+RNN or RNN-only encoders, followed by either concatenation-based fusion or a lightweight attention mechanism
\[
u = \tanh(Wh+b),\quad a=\mathrm{softmax}(u^Tc),\quad v=\sum_i a_i h_i,
\]
before a classifier head [2306.09486]. In 3FM, separate image, spectrogram, and sign-language branches are flattened, concatenated, and passed through two fully connected layers, with missing branches muted to zeros [2312.10179].

A second pattern is parameter-efficient adaptation of large multimodal foundation models. FedDLP freezes the CLIP backbone \(\Delta\), retains the vision–language alignment learned during pretraining, and fine-tunes either the text encoder or the image encoder with two LoRA adapters per client: a larger local adapter \(\theta_l^i\) for personalization and a smaller global adapter \(\theta_g^i\) for federated aggregation [2503.07552]. FedCola extends transformer-based vision–language FL by mixing local and out-modality transformer blocks on uni-modal clients through learnable gates, effectively using
\[
W_{\text{eff}} = W_{\text{local}} + g\,W_{\text{out}}
\]
inside transformer layers while preserving a standard FL communication interface at upload time [2404.12467].

A third pattern replaces parameter sharing with representation alignment. CreamFL avoids parameter averaging across heterogeneous multimodal, image-only, and text-only clients by exchanging representations on a public image–text dataset and performing global–local contrastive aggregation; its local objective combines task loss with inter-modal and intra-modal contrastive regularization [2302.08888]. FedMEKT instead uses a small multimodal proxy dataset and a distillation-based multimodal embedding knowledge transfer pipeline built from local multimodal autoencoders, a generalized multimodal autoencoder on the server, and a generalized classifier [2307.13214]. CoMFed further compresses this idea into class-wise latent-space consensus by learning client-specific projection matrices \(\mathbf{P}_i\) and communicating only projected class means \(\widehat{\mathbf{u}_{i,m}^{(t)}} = \mathbf{P}_i^{(t)} \widehat{\bar{\mathbf{v}_{i,m}^{(t)}}\) [2603.19067].

## 4. Personalization and communication-efficient optimization

Personalization in MFL is often implemented by asymmetric sharing. FedDLP separates personalization and generalization through a larger local LoRA adapter, a smaller global LoRA adapter, alternating freezing, bi-directional distillation, and SoRA-style pruning on the local adapter; only the global adapter is aggregated by FedAvg, while the local adapter remains private [2503.07552]. EPFL in FHBench exploits LoRA asymmetry differently: it aggregates only \(A\) matrices in a similarity-weight

Source: https://www.emergentmind.com/topics/multimodal-federated-learning-fl