---
title: Mix-Modal Federated Learning (MixMFL)
url: https://www.emergentmind.com/topics/mix-modal-federated-learning-mixmfl
type: topic
---

# Mix-Modal Federated Learning (MixMFL)

Mix-Modal Federated Learning (MixMFL) denotes a federated learning regime in which clients possess distinct, possibly incomplete subsets of modalities under heterogeneous data distributions, so modality heterogeneity and data heterogeneity must be handled simultaneously rather than in isolation [2509.02541]. Closely related formulations in the recent literature include multimodal federated learning with modality heterogeneity in computational pathology, incomplete-modality federated learning for image-text data, transfer federated learning with unpaired uni-modal and multimodal clients, and modality-incongruity settings with client-modality selection [2402.15858], [2406.11048], [2404.12467], [2401.00403]. Across these formulations, the standard assumption behind homogeneous FL—that clients expose compatible input structures and can therefore participate in a unified parameter-averaging loop—fails structurally once modality subsets differ.

## 1. Paradigm boundaries and terminology

A precise distinction among neighboring paradigms was made for MRI segmentation, where MixMFL was separated from multimodal federated learning (MulMFL) and cross-modal federated learning (CroMFL) [2509.02541].

| Paradigm | Client modality pattern | Main challenge |
|---|---|---|
| MulMFL | All clients have all modalities | Data heterogeneity |
| CroMFL | One modality per client from a shared data distribution | Modality heterogeneity |
| MixMFL | Varied, incomplete subsets under heterogeneous data distributions | Data heterogeneity and modality heterogeneity |

Within this taxonomy, MixMFL is the most general case: each client may have only a subset of modalities, different clients may hold different subsets, and the underlying local data distributions may also differ. This generality explains why techniques designed for homogeneous multimodal federation, such as training a single unified fusion model with standard aggregation, are repeatedly reported as fragile or inapplicable once modalities are missing, unpaired, or unevenly distributed [2509.02541], [2402.15858].

A separate source of terminological confusion is **mixed FL**, which is not a modality taxonomy at all. Mixed FL jointly optimizes a federated loss and a centralized server-side loss, with algorithms such as Parallel Training, 1-way Gradient Transfer, and 2-way Gradient Transfer; its focus is joint decentralized and centralized learning rather than client-wise modality composition [2205.13655]. Thus, MixMFL concerns *what modalities are available where*, whereas mixed FL concerns *where the loss is computed*.

## 2. Core problem structure

The defining systems problem in MixMFL is architectural incompatibility induced by modality heterogeneity. FedMM formalizes this in computational pathology: client \(C_i\) may hold only \(M_i \leq M\) modalities, local embeddings are extracted by modality-specific feature extractors \(f_i^{(k)}\), and the client-specific classifier \(g_i(\cdot)\) is never aggregated because heterogeneous modality subsets induce heterogeneous feature spaces [2402.15858]. The same incompatibility recurs in transformer-based transfer MFL, where image-only, text-only, and image-text clients optimize different objectives, creating both a cross-modality gap and an in-modality gap [2404.12467].

A second difficulty is incomplete modality. FedMVP studies the case where clients may lose part of the modalities in local datasets and uses foundation-model-based completion before federated training [2406.11048]. FedMobile studies mobile sensing systems in which some modalities are unavailable or only partially captured, emphasizing that node-side interpolation overlooks shared latent feature structure across nodes [2502.15839]. In 3FM, each client is treated as a task with its own subset of available modalities, and meta-learning is used to adapt to new or missing modality combinations [2312.10179]. In UTMP, the restriction is even stronger: clients may train only on unimodal data, yet the global model is expected to support multimodal prediction [2303.15486].

A third difficulty is *modal bias*. BMSFed empirically reports that local training with a certain single modality may contribute more to the global model than training with all local modalities, and therefore frames the problem as one of both local modal imbalance and global modality-level bias [2401.00403]. This is distinct from simple missingness: even when several modalities exist locally, naive joint training may still overfit to the dominant or faster-learning modality.

A fourth difficulty is systems heterogeneity beyond modalities. FlexMod focuses on IoT settings where uniform training frequencies across modality encoders are inefficient under limited computational budgets, and FedMFS focuses on network-constrained devices that cannot upload all local modality models every round [2408.06549], [2310.07048]. AproMFL further extends the scope to heterogeneous tasks, where mixed modalities and heterogeneous downstream objectives jointly weaken unified representation learning [2502.04400].

## 3. Architectural patterns in MixMFL

A dominant design pattern is **modality-wise decomposition**. FedMM trains multiple single-modal feature extractors federatedly, aggregates each extractor only across clients sharing that modality, and leaves downstream fusion and classification local to each client [2402.15858]. This removes the requirement that all clients instantiate the same multimodal fusion model. In MRI segmentation, MDM-MixMFL pushes this logic further by explicitly decoupling each modality into modality-tailored and modality-shared information; modality-tailored encoders are updated modality-wise across relevant clients, while the modality-shared encoder is updated across all clients [2509.02541].

A second pattern is **prototype or embedding transfer**. FedMEKT employs local multimodal autoencoder learning, generalized multimodal autoencoder construction, and generalized classifier learning, with the server and clients exchanging joint embedding knowledge distilled on a small multimodal proxy dataset [2307.13214]. AproMFL also uses prototypes, but adapts construction to client task type: labeled unimodal clients use label-guided class prototypes, whereas unlabeled or multimodal clients use clustering-based pseudo-labels and fused image-text embeddings; only mapping modules are aggregated, not full encoders [2502.04400]. FedMobile similarly works in latent space, reconstructing missing modality features via local generators and then aggregating generator knowledge across nodes [2502.15839].

A third pattern is **completion or compensation for missing modalities**. FedMVP performs prompt-augmented modality completion on the client side using large-scale pre-trained models: missing images are regenerated using DALLE-style text-to-image generation, and missing textual descriptions are synthesized using image captioning and visual question-answering prompts; ViT and BERT then encode the completed modalities, and a cross-modal attention joint encoder supports downstream classification [2406.11048]. MDM-MixMFL instead uses a modality memorizing mechanism: a global memory bank stores modality prototypes refreshed from modality-tailored encoders, and missing modality representations are compensated by retrieving prototypes most similar to the available modality-shared features [2509.02541].

A fourth pattern is **pre-trained alignment with parameter-efficient adaptation**. FedAlign places a text encoder on the server and image encoders on clients, initializes both from pre-trained CLIP, freezes the backbones, and updates only LoRA parameters, with global text features broadcast as modality anchors [2411.15837]. FedCola also leverages transformer modularity, but in a different regime: image-only, text-only, and image-text clients share a unified transformer architecture; uni-modal clients can incorporate blocks from the other modality through gated complementary local training, while aggregation selectively mixes only layers deemed transferable across modality-specific or task-specific objectives [2404.12467].

## 4. Aggregation, scheduling, and communication control

Once model factorization is introduced, aggregation itself becomes modality-aware or relation-aware rather than globally uniform. FedMM aggregates extractor weights separately for each modality and also computes per-modality global prototypes [2402.15858]. HA-Fedformer replaces vanilla averaging with hierarchical aggregation: Posterior-based Encoder Aggregation uses layer-wise means and covariances estimated via MCMC to downweight uncertain local encoders, while Cross-modal Decoder Aggregation uses attention-like parameter correlation to aggregate decoders trained on different modalities without requiring aligned features [2303.15486].

Several systems explicitly optimize communication. FedMFS introduces a selective modality communication rule in which each client computes a priority score
\[
P_m^k = \alpha_s \tilde{\varphi}_m^k + \alpha_c \bigl(1 - |\tilde{\theta}_m^k|\bigr),
\]
combining Shapley-value modality impact and model-size communication cost, then uploads only the top-\(\gamma\) modality models [2310.07048]. FlexMod addresses a different resource axis: it allocates local training frequencies across modality combinations under a fixed time budget \(T\), using prototype-based encoder quality, Shapley-based modality importance, and a DDPG agent that sets the trade-off coefficient \(\beta^r\) round by round [2408.06549].

Other methods alter aggregation weights using structural similarity. TACTFL computes pairwise representational consistency on server-held labeled data and assigns client aggregation weights according to similarity-guided model aggregation, so semantically misaligned self-supervised client models have less influence on the global update [2509.17532]. AproMFL constructs a client relationship graph from cosine similarities between mapping modules and uses that graph to adapt aggregation weights dynamically [2502.04400]. FedAlign uses a query-based server-side method to combine global LoRA updates for lower layers with client-personalized LoRA initialization for higher layers, explicitly separating general representation transfer from personalized semantics [2411.15837].

Selection can also be moved from the client level to the client-modality level. BMSFed introduces modality selection rather than conventional client selection, uses a facility-location-type submodular objective to select representative and diverse modalities, and biases the selection toward weaker modalities through imbalance ratios and modal enhancement loss [2401.00403]. This directly targets the global modal bias that ordinary participant sampling leaves unresolved.

At the personalization boundary, FedMix is not a multimodal method per se, but it remains relevant as a heterogeneity-management template. It trains an ensemble of experts, lets each client adaptively select the expert subset it uses, and aggregates expert-specific gradients with weights proportional to client usage rather than global participation [2107.06724]. This suggests a general compatibility between MixMFL and expert-specialization strategies, although the cited work addresses non-i.i.d. heterogeneity more broadly than modalities.

## 5. Representative optimization formulations

FedMM provides a canonical modality-wise formulation. For client \(i\), local embeddings are
\[
\mathbf{h}_i^{(k)} = f_i^{(k)}(\mathbf{x}_i^{(k)}), \qquad
\hat{y}_i = g_i(\{\mathbf{h}_i^{(k)}\}_{k=1}^{M_i}),
\]
and the global prototype for modality \(k\) is
\[
\bar{\mathbf{p}}^{(k)} = \frac{1}{N_k}\sum_{i=1}^{N_k}\mathbf{h}_i^{(k)}.
\]
Feature extractor training uses a dynamic loss that combines distance-to-prototype and classification:
\[
\min \left\{ \lambda(t)\frac{\beta}{D}\left\|\mathbf{h}_i^{(k)}-\bar{\mathbf{p}}^{(k)}\right\|_2 + [1-\lambda(t)]\,\mathrm{BCE}(\hat{y}_i,y_i)\right\},
\]
with \(\lambda(t)\) scheduled by a sigmoid transition over communication rounds [2402.15858].

FlexMod formalizes local computation allocation as an integer optimization problem over all non-empty modality combinations:
\[
U^r(\mathbf{a}^r)=\sum_{s=1}^{S} [\beta^r \Omega^s + (1-\beta^r)\Gamma^s]\, a_s^r,
\qquad
\sum_{s=1}^{S} t_s a_s^r \leq T,\quad a_s^r \in \mathbb{Z}_{\geq 0}.
\]
Here \(\Omega^s\) summarizes prototype-based encoder quality, \(\Gamma^s\) summarizes Shapley-based modality importance, and the DDPG agent selects \(\beta^r \in [0,1]\) from the current quality and importance state [2408.06549].

BMSFed formalizes weak-modality enhancement through a prototypical cross-entropy term:
\[
\mathcal{B}_k^{\mathcal{C}_h}(V_I) =
-\frac{1}{|D_k|}\sum_{(x_j,y_j)\in D_k}
\log
\frac{\exp(-d(z_j,c_{y_j}))}
{\sum_{i=1}^{Y}\exp(-d(z_j,c_i))}.
\]
When global prototypes are used for uni-modal clients, the loss aligns local weak-modality embeddings to server-aggregated class prototypes, reducing heterogeneity introduced by selective modality training [2401.00403].

In 3FM, the adaptation objective is explicitly meta-learned:
\[
\theta^u \leftarrow \theta - \alpha \nabla L_{D_S^u}(\theta), \qquad
g_u \leftarrow \nabla_\theta L_{D_Q^u}(\theta^u), \qquad
\theta \leftarrow \theta - \beta \sum_{u\in U_t} g_u.
\]
Because each client is treated as a task defined by its available modality subset, the support-query split forces the global model to learn how to adapt to different missing-modality patterns rather than merely average across them [2312.10179].

TACTFL exemplifies the recent shift toward self-supervised MixMFL. Local temporal contrastive learning is coupled with similarity-guided aggregation:
\[
W(i)=\frac{\sum_{j=0}^{C-1} S(i,j)}{\sum_{i=0}^{C-1}\sum_{j=0}^{C-1} S(i,j)},
\qquad
m'=\sum_{i=0}^{C-1} W(i)\, m_i,
\]
where \(S(i,j)\) is the cosine similarity between client summary representations on server-held labeled data [2509.17532]. In effect, aggregation is conditioned on representational consensus rather than dataset size alone.

## 6. Empirical behavior, misconceptions, and unresolved trade-offs

Reported empirical results consistently show that naive homogenization strategies are weak baselines in mixed-modality settings. In FedMM, hospital 2 on TCGA-NSCLC obtained an AUC increase of 0.023 (2.79%) over Local and 0.065 (8.31%) over Multi-FedAvg; on TCGA-RCC the corresponding gains were 0.027 (2.97%) and 0.069 (7.96%) [2402.15858]. In FlexMod, on UCI-HAR with non-IID data \((\alpha = 10)\), reaching test accuracy 0.6 required 12 rounds versus 20 rounds for Entire Update [2408.06549]. In FedMFS, ActionSense results reported 97.34% accuracy at 0.72 MB/round, with communication overhead reduced by over 4x [2310.07048]. In HA-Fedformer++, CMU-MOSEI \(Acc_7\) reached 48.6 versus 43.8 for FedMSplit [2303.15486]. In MDM-MixMFL, BraTS21 average mDice reached 58.60% versus 55.78% for the next-best baseline and approached the i.i.d. upper bound of 59.56% [2509.02541]. In TACTFL, UCF101 with only 10% labelled data reached 68.48% top-1 accuracy versus 35.35% for FedOpt [2509.17532]. FedMobile reported robust learning even when up to 90% of modality information was missing or when data from two modalities were randomly missing [2502.15839]. FedMVP reported that on CUB-200 at \(\beta = 0.8\), accuracy remained in the 68–71% range under severe incompleteness [2406.11048].

A recurring misconception is that MixMFL can be handled adequately by zero-masking or placeholder filling. The comparative setups themselves argue otherwise: Multi-FedAvg in FedMM sets missing modalities to zero; BMSFed criticizes ordinary client selection for global modal bias; FedMVP explicitly contrasts foundation-model-powered completion with dummy sentences, dummy images, attention-masking, and shallow imputation [2402.15858], [2401.00403], [2406.11048]. Another misconception is that MixMFL is merely multimodal FL with non-i.i.d. data. The MixMFL taxonomy in MRI segmentation treats it as a distinct regime because both client-wise modality subsets and data distributions vary simultaneously [2509.02541].

The main unresolved trade-offs are assumption-related rather than purely algorithmic. FedMEKT assumes a small shared multimodal proxy dataset; FlexMod estimates modality importance on a small validation set of approximately 1% of total data; TACTFL uses a small labelled server dataset; FedAlign assumes server-side text descriptions and a server text encoder; FedMobile uses a proxy dataset for local/global contribution assessment; AproMFL explicitly avoids a prior public dataset [2307.13214], [2408.06549], [2509.17532], [2411.15837], [2502.15839], [2502.04400]. This suggests that current MixMFL methods differ substantially in how much shared side information they assume, even when all preserve the basic FL restriction that raw client data are not centralized.

A related trade-off concerns what is actually communicated. Some methods exchange only model weights or LoRA parameters; others exchange aggregate prototype embeddings, distilled proxy-data embeddings, synthesized feature sets, or global text features [2402.15858], [2307.13214], [2411.15837]. A plausible implication is that privacy in MixMFL is usually operationalized as *raw-data non-disclosure*, not as an absolute prohibition on representation sharing. That distinction is increasingly important as the field moves from homogeneous FedAvg-style models toward prototype transfer, similarity-guided aggregation, foundation-model completion, and adaptive cross-client alignment.

Source: https://www.emergentmind.com/topics/mix-modal-federated-learning-mixmfl