Mix-Modal Federated Learning (MixMFL)

• Mix-Modal Federated Learning (MixMFL) is a regime where clients hold varied, incomplete modality subsets, simultaneously addressing data and modality heterogeneity.
• It employs architectural patterns like modality-wise decomposition, prototype transfer, and completion strategies to overcome incompatibility and modal bias.
• Practical implementations show improved performance over standard methods, with demonstrated gains in computational pathology, mobile sensing, and multimodal optimization tasks.

Mix-Modal Federated Learning (MixMFL) denotes a federated learning regime in which clients possess distinct, possibly incomplete subsets of modalities under heterogeneous data distributions, so modality heterogeneity and data heterogeneity must be handled simultaneously rather than in isolation (Hu et al., 2 Sep 2025). Closely related formulations in the recent literature include multimodal federated learning with modality heterogeneity in computational pathology, incomplete-modality federated learning for image-text data, transfer federated learning with unpaired uni-modal and multimodal clients, and modality-incongruity settings with client-modality selection (Peng et al., 2024, Che et al., 2024, Sun et al., 2024, Fan et al., 2023). Across these formulations, the standard assumption behind homogeneous FL—that clients expose compatible input structures and can therefore participate in a unified parameter-averaging loop—fails structurally once modality subsets differ.

1. Paradigm boundaries and terminology

A precise distinction among neighboring paradigms was made for MRI segmentation, where MixMFL was separated from multimodal federated learning (MulMFL) and cross-modal federated learning (CroMFL) (Hu et al., 2 Sep 2025).

Paradigm	Client modality pattern	Main challenge
MulMFL	All clients have all modalities	Data heterogeneity
CroMFL	One modality per client from a shared data distribution	Modality heterogeneity
MixMFL	Varied, incomplete subsets under heterogeneous data distributions	Data heterogeneity and modality heterogeneity

Within this taxonomy, MixMFL is the most general case: each client may have only a subset of modalities, different clients may hold different subsets, and the underlying local data distributions may also differ. This generality explains why techniques designed for homogeneous multimodal federation, such as training a single unified fusion model with standard aggregation, are repeatedly reported as fragile or inapplicable once modalities are missing, unpaired, or unevenly distributed (Hu et al., 2 Sep 2025, Peng et al., 2024).

A separate source of terminological confusion is mixed FL, which is not a modality taxonomy at all. Mixed FL jointly optimizes a federated loss and a centralized server-side loss, with algorithms such as Parallel Training, 1-way Gradient Transfer, and 2-way Gradient Transfer; its focus is joint decentralized and centralized learning rather than client-wise modality composition (Augenstein et al., 2022). Thus, MixMFL concerns what modalities are available where, whereas mixed FL concerns where the loss is computed.

2. Core problem structure

The defining systems problem in MixMFL is architectural incompatibility induced by modality heterogeneity. FedMM formalizes this in computational pathology: client $C_i$ may hold only $M_i \leq M$ modalities, local embeddings are extracted by modality-specific feature extractors $f_i^{(k)}$, and the client-specific classifier $g_i(\cdot)$ is never aggregated because heterogeneous modality subsets induce heterogeneous feature spaces (Peng et al., 2024). The same incompatibility recurs in transformer-based transfer MFL, where image-only, text-only, and image-text clients optimize different objectives, creating both a cross-modality gap and an in-modality gap (Sun et al., 2024).

A second difficulty is incomplete modality. FedMVP studies the case where clients may lose part of the modalities in local datasets and uses foundation-model-based completion before federated training (Che et al., 2024). FedMobile studies mobile sensing systems in which some modalities are unavailable or only partially captured, emphasizing that node-side interpolation overlooks shared latent feature structure across nodes (Liu et al., 20 Feb 2025). In 3FM, each client is treated as a task with its own subset of available modalities, and meta-learning is used to adapt to new or missing modality combinations (Tran et al., 2023). In UTMP, the restriction is even stronger: clients may train only on unimodal data, yet the global model is expected to support multimodal prediction (Zhang et al., 2023).

A third difficulty is modal bias. BMSFed empirically reports that local training with a certain single modality may contribute more to the global model than training with all local modalities, and therefore frames the problem as one of both local modal imbalance and global modality-level bias (Fan et al., 2023). This is distinct from simple missingness: even when several modalities exist locally, naive joint training may still overfit to the dominant or faster-learning modality.

A fourth difficulty is systems heterogeneity beyond modalities. FlexMod focuses on IoT settings where uniform training frequencies across modality encoders are inefficient under limited computational budgets, and FedMFS focuses on network-constrained devices that cannot upload all local modality models every round (Bian et al., 2024, Yuan et al., 2023). AproMFL further extends the scope to heterogeneous tasks, where mixed modalities and heterogeneous downstream objectives jointly weaken unified representation learning (Gai et al., 6 Feb 2025).

3. Architectural patterns in MixMFL

A dominant design pattern is modality-wise decomposition. FedMM trains multiple single-modal feature extractors federatedly, aggregates each extractor only across clients sharing that modality, and leaves downstream fusion and classification local to each client (Peng et al., 2024). This removes the requirement that all clients instantiate the same multimodal fusion model. In MRI segmentation, MDM-MixMFL pushes this logic further by explicitly decoupling each modality into modality-tailored and modality-shared information; modality-tailored encoders are updated modality-wise across relevant clients, while the modality-shared encoder is updated across all clients (Hu et al., 2 Sep 2025).

A second pattern is prototype or embedding transfer. FedMEKT employs local multimodal autoencoder learning, generalized multimodal autoencoder construction, and generalized classifier learning, with the server and clients exchanging joint embedding knowledge distilled on a small multimodal proxy dataset (Le et al., 2023). AproMFL also uses prototypes, but adapts construction to client task type: labeled unimodal clients use label-guided class prototypes, whereas unlabeled or multimodal clients use clustering-based pseudo-labels and fused image-text embeddings; only mapping modules are aggregated, not full encoders (Gai et al., 6 Feb 2025). FedMobile similarly works in latent space, reconstructing missing modality features via local generators and then aggregating generator knowledge across nodes (Liu et al., 20 Feb 2025).

A third pattern is completion or compensation for missing modalities. FedMVP performs prompt-augmented modality completion on the client side using large-scale pre-trained models: missing images are regenerated using DALLE-style text-to-image generation, and missing textual descriptions are synthesized using image captioning and visual question-answering prompts; ViT and BERT then encode the completed modalities, and a cross-modal attention joint encoder supports downstream classification (Che et al., 2024). MDM-MixMFL instead uses a modality memorizing mechanism: a global memory bank stores modality prototypes refreshed from modality-tailored encoders, and missing modality representations are compensated by retrieving prototypes most similar to the available modality-shared features (Hu et al., 2 Sep 2025).

A fourth pattern is pre-trained alignment with parameter-efficient adaptation. FedAlign places a text encoder on the server and image encoders on clients, initializes both from pre-trained CLIP, freezes the backbones, and updates only LoRA parameters, with global text features broadcast as modality anchors (Ma et al., 2024). FedCola also leverages transformer modularity, but in a different regime: image-only, text-only, and image-text clients share a unified transformer architecture; uni-modal clients can incorporate blocks from the other modality through gated complementary local training, while aggregation selectively mixes only layers deemed transferable across modality-specific or task-specific objectives (Sun et al., 2024).

4. Aggregation, scheduling, and communication control

Once model factorization is introduced, aggregation itself becomes modality-aware or relation-aware rather than globally uniform. FedMM aggregates extractor weights separately for each modality and also computes per-modality global prototypes (Peng et al., 2024). HA-Fedformer replaces vanilla averaging with hierarchical aggregation: Posterior-based Encoder Aggregation uses layer-wise means and covariances estimated via MCMC to downweight uncertain local encoders, while Cross-modal Decoder Aggregation uses attention-like parameter correlation to aggregate decoders trained on different modalities without requiring aligned features (Zhang et al., 2023).

Several systems explicitly optimize communication. FedMFS introduces a selective modality communication rule in which each client computes a priority score

$$P_m^k = \alpha_s \tilde{\varphi}_m^k + \alpha_c \bigl(1 - |\tilde{\theta}_m^k|\bigr),$$

combining Shapley-value modality impact and model-size communication cost, then uploads only the top-$\gamma$ modality models (Yuan et al., 2023). FlexMod addresses a different resource axis: it allocates local training frequencies across modality combinations under a fixed time budget $T$, using prototype-based encoder quality, Shapley-based modality importance, and a DDPG agent that sets the trade-off coefficient $\beta^r$ round by round (Bian et al., 2024).

Other methods alter aggregation weights using structural similarity. TACTFL computes pairwise representational consistency on server-held labeled data and assigns client aggregation weights according to similarity-guided model aggregation, so semantically misaligned self-supervised client models have less influence on the global update (Sun et al., 22 Sep 2025). AproMFL constructs a client relationship graph from cosine similarities between mapping modules and uses that graph to adapt aggregation weights dynamically (Gai et al., 6 Feb 2025). FedAlign uses a query-based server-side method to combine global LoRA updates for lower layers with client-personalized LoRA initialization for higher layers, explicitly separating general representation transfer from personalized semantics (Ma et al., 2024).

Selection can also be moved from the client level to the client-modality level. BMSFed introduces modality selection rather than conventional client selection, uses a facility-location-type submodular objective to select representative and diverse modalities, and biases the selection toward weaker modalities through imbalance ratios and modal enhancement loss (Fan et al., 2023). This directly targets the global modal bias that ordinary participant sampling leaves unresolved.

At the personalization boundary, FedMix is not a multimodal method per se, but it remains relevant as a heterogeneity-management template. It trains an ensemble of experts, lets each client adaptively select the expert subset it uses, and aggregates expert-specific gradients with weights proportional to client usage rather than global participation (Reisser et al., 2021). This suggests a general compatibility between MixMFL and expert-specialization strategies, although the cited work addresses non-i.i.d. heterogeneity more broadly than modalities.

5. Representative optimization formulations

FedMM provides a canonical modality-wise formulation. For client $i$, local embeddings are

$$\mathbf{h}_i^{(k)} = f_i^{(k)}(\mathbf{x}_i^{(k)}), \qquad \hat{y}_i = g_i(\{\mathbf{h}_i^{(k)}\}_{k=1}^{M_i}),$$

and the global prototype for modality $M_i \leq M$0 is

$M_i \leq M$1

Feature extractor training uses a dynamic loss that combines distance-to-prototype and classification: $M_i \leq M$2 with $M_i \leq M$3 scheduled by a sigmoid transition over communication rounds (Peng et al., 2024).

FlexMod formalizes local computation allocation as an integer optimization problem over all non-empty modality combinations: $M_i \leq M$4 Here $M_i \leq M$5 summarizes prototype-based encoder quality, $M_i \leq M$6 summarizes Shapley-based modality importance, and the DDPG agent selects $M_i \leq M$7 from the current quality and importance state (Bian et al., 2024).

BMSFed formalizes weak-modality enhancement through a prototypical cross-entropy term: $M_i \leq M$8 When global prototypes are used for uni-modal clients, the loss aligns local weak-modality embeddings to server-aggregated class prototypes, reducing heterogeneity introduced by selective modality training (Fan et al., 2023).

In 3FM, the adaptation objective is explicitly meta-learned: $M_i \leq M$9 Because each client is treated as a task defined by its available modality subset, the support-query split forces the global model to learn how to adapt to different missing-modality patterns rather than merely average across them (Tran et al., 2023).

TACTFL exemplifies the recent shift toward self-supervised MixMFL. Local temporal contrastive learning is coupled with similarity-guided aggregation: $f_i^{(k)}$0 where $f_i^{(k)}$1 is the cosine similarity between client summary representations on server-held labeled data (Sun et al., 22 Sep 2025). In effect, aggregation is conditioned on representational consensus rather than dataset size alone.

6. Empirical behavior, misconceptions, and unresolved trade-offs

Reported empirical results consistently show that naive homogenization strategies are weak baselines in mixed-modality settings. In FedMM, hospital 2 on TCGA-NSCLC obtained an AUC increase of 0.023 (2.79%) over Local and 0.065 (8.31%) over Multi-FedAvg; on TCGA-RCC the corresponding gains were 0.027 (2.97%) and 0.069 (7.96%) (Peng et al., 2024). In FlexMod, on UCI-HAR with non-IID data $f_i^{(k)}$2, reaching test accuracy 0.6 required 12 rounds versus 20 rounds for Entire Update (Bian et al., 2024). In FedMFS, ActionSense results reported 97.34% accuracy at 0.72 MB/round, with communication overhead reduced by over 4x (Yuan et al., 2023). In HA-Fedformer++, CMU-MOSEI $f_i^{(k)}$3 reached 48.6 versus 43.8 for FedMSplit (Zhang et al., 2023). In MDM-MixMFL, BraTS21 average mDice reached 58.60% versus 55.78% for the next-best baseline and approached the i.i.d. upper bound of 59.56% (Hu et al., 2 Sep 2025). In TACTFL, UCF101 with only 10% labelled data reached 68.48% top-1 accuracy versus 35.35% for FedOpt (Sun et al., 22 Sep 2025). FedMobile reported robust learning even when up to 90% of modality information was missing or when data from two modalities were randomly missing (Liu et al., 20 Feb 2025). FedMVP reported that on CUB-200 at $f_i^{(k)}$4, accuracy remained in the 68–71% range under severe incompleteness (Che et al., 2024).

A recurring misconception is that MixMFL can be handled adequately by zero-masking or placeholder filling. The comparative setups themselves argue otherwise: Multi-FedAvg in FedMM sets missing modalities to zero; BMSFed criticizes ordinary client selection for global modal bias; FedMVP explicitly contrasts foundation-model-powered completion with dummy sentences, dummy images, attention-masking, and shallow imputation (Peng et al., 2024, Fan et al., 2023, Che et al., 2024). Another misconception is that MixMFL is merely multimodal FL with non-i.i.d. data. The MixMFL taxonomy in MRI segmentation treats it as a distinct regime because both client-wise modality subsets and data distributions vary simultaneously (Hu et al., 2 Sep 2025).

The main unresolved trade-offs are assumption-related rather than purely algorithmic. FedMEKT assumes a small shared multimodal proxy dataset; FlexMod estimates modality importance on a small validation set of approximately 1% of total data; TACTFL uses a small labelled server dataset; FedAlign assumes server-side text descriptions and a server text encoder; FedMobile uses a proxy dataset for local/global contribution assessment; AproMFL explicitly avoids a prior public dataset (Le et al., 2023, Bian et al., 2024, Sun et al., 22 Sep 2025, Ma et al., 2024, Liu et al., 20 Feb 2025, Gai et al., 6 Feb 2025). This suggests that current MixMFL methods differ substantially in how much shared side information they assume, even when all preserve the basic FL restriction that raw client data are not centralized.

A related trade-off concerns what is actually communicated. Some methods exchange only model weights or LoRA parameters; others exchange aggregate prototype embeddings, distilled proxy-data embeddings, synthesized feature sets, or global text features (Peng et al., 2024, Le et al., 2023, Ma et al., 2024). A plausible implication is that privacy in MixMFL is usually operationalized as raw-data non-disclosure, not as an absolute prohibition on representation sharing. That distinction is increasingly important as the field moves from homogeneous FedAvg-style models toward prototype transfer, similarity-guided aggregation, foundation-model completion, and adaptive cross-client alignment.