Multimodal Online Federated Learning

• Multimodal Online Federated Learning (MMO-FL) is a framework for dynamic, decentralized learning on continuously streamed IoT data with heterogeneous modality availability.
• It employs strategies like prototype substitution, latent reconstruction, and selective aggregation to mitigate missing data, modality imbalance, and resource constraints.
• Key contributions include adapting online gradient methods and cross-modal fusion to achieve robust real-time performance under diverse and shifting IoT conditions.

Searching arXiv for the cited MMO-FL and related multimodal federated learning papers. Multimodal Online Federated Learning (MMO-FL) denotes federated learning over continually arriving multimodal data streams on distributed clients, typically under IoT or WoT constraints, where clients differ in modality availability, data distributions, resource budgets, and connectivity. In the cited literature, the term is explicitly introduced for dynamic and decentralized multimodal learning in IoT environments with limited local storage and real-time data collection, while closely related work addresses its constituent problems through multimodal federated meta-learning, missing-modality mitigation, latent reconstruction, modality/client selection, adaptive scheduling, and multimodal transformer aggregation (Wang et al., 22 May 2025, Wang et al., 15 Aug 2025, Tran et al., 2023, Liu et al., 20 Feb 2025, Yuan et al., 2024, Sun et al., 2024, Bian et al., 2024).

1. Formal scope and notation

A recurrent formalization models a cloud server and $K$ clients over global rounds $t=1,\dots,T$, with modalities indexed by $m\in\{1,\dots,M\}$. In the IoT-oriented MMO-FL formulation, the global model is $\Theta=\{\theta^0,\theta^1,\dots,\theta^M\}$, where $\theta^m$ are modality-specific encoders and $\theta^0$ is a shared head that fuses modality features. If $Z_k^{t,m}=\theta^m(X_k^{t,m})$, the per-round multimodal loss is written as

$$F_t(\Theta,\mathcal{D}^t)=\frac{1}{K}\sum_{k=1}^K f_t(\theta^0(Z_k^{t,1},\dots,Z_k^{t,M}),Y_k^t),$$

and online performance is assessed by the cumulative regret

$$\mathrm{Reg}_T=\sum_{t=1}^T F_t(\Theta^t;\mathcal{D}^t)-\sum_{t=1}^T F_t(\Theta^*;\mathcal{D}^t),$$

where $\Theta^*$ is the best fixed model in hindsight (Wang et al., 22 May 2025).

A more granular formulation for modality heterogeneity defines a global modality set $t=1,\dots,T$0, client-specific subsets $t=1,\dots,T$1, and missing-modality masks $t=1,\dots,T$2. In 3FM, each sample from client $t=1,\dots,T$3 is $t=1,\dots,T$4 with label $t=1,\dots,T$5, each modality encoder $t=1,\dots,T$6 produces $t=1,\dots,T$7, and late fusion concatenates masked features,

$t=1,\dots,T$8

with missing branches contributing zero vectors (Tran et al., 2023).

The online dimension is explicit in the MMO-FL formulations for IoT: data arrive continually, clients perform local online gradient descent over current streaming datasets, and the server aggregates local models after each round. Related papers are often synchronous rather than fully streaming, but they preserve the same central objects—per-round updates, modality masks, incomplete observations, and round-adaptive aggregation or scheduling—which makes them directly relevant to MMO-FL even when they are not “online” in the strict sense (Wang et al., 22 May 2025, Liu et al., 20 Feb 2025, Yuan et al., 2024, Sun et al., 2024, Bian et al., 2024).

2. Heterogeneity, missingness, and imbalance

The literature treats modality incompleteness as a primary structural challenge. In 3FM, each client has a subset $t=1,\dots,T$9 and a missing-modality mask $m\in\{1,\dots,M\}$0, so robustness requires adaptation across different modality subsets. FedMobile uses an analogous mask $m\in\{1,\dots,M\}$1, motivated by mobile sensing systems in which sensors fail, channels are intermittently unavailable, or only partial measurements are captured. The IoT MMO-FL formulation similarly allows client- and round-dependent available modality sets $m\in\{1,\dots,M\}$2, with missing-modality gradients set to zero for unavailable branches (Tran et al., 2023, Liu et al., 20 Feb 2025, Wang et al., 22 May 2025).

A second layer of difficulty is that missingness is not the only form of instability. QQR formalizes modality quantity and quality imbalance (QQI) through per-sample missingness indicators $m\in\{1,\dots,M\}$3, round-level quality indicators $m\in\{1,\dots,M\}$4, and the imbalance parameters $m\in\{1,\dots,M\}$5, $m\in\{1,\dots,M\}$6, and $m\in\{1,\dots,M\}$7. Here, $m\in\{1,\dots,M\}$8 denotes the intra-round quantity imbalance ratio, $m\in\{1,\dots,M\}$9 the fraction of rounds with quantity imbalance, and $\Theta=\{\theta^0,\theta^1,\dots,\theta^M\}$0 the fraction of rounds with low-quality data. This extends MMO-FL beyond absent modalities to degraded modalities, noisy sensing, and intermittent quality collapse (Wang et al., 15 Aug 2025).

Communication and computation constraints constitute a third axis of heterogeneity. mmFedMC assumes that clients cannot upload all locally trained modality models every round, and therefore separates per-modality global training from local decision-level fusion while jointly selecting modalities and clients. FlexMod addresses a complementary problem: even when every client has all modalities, uniformly allocating local training frequencies across modality encoders is inefficient under resource constraints, so scheduling must account for modality importance, encoder quality, and per-round time budgets (Yuan et al., 2024, Bian et al., 2024).

A common misconception is that MMO-FL is reducible to multimodal imputation. The cited work suggests a broader problem class: modality subset variation, streaming data, non-IID client distributions, resource-constrained participation, modality quality degradation, asynchronous or stale updates, and objective mismatch between uni-modal and multimodal clients all appear as first-order design variables in current formulations (Wang et al., 15 Aug 2025, Liu et al., 20 Feb 2025, Sun et al., 2024).

3. Algorithmic families

One line of work treats MMO-FL as a rapid adaptation problem. 3FM uses a MAML-style bilevel objective in a federated protocol: each client splits data into support and query sets, performs an inner update

$\Theta=\{\theta^0,\theta^1,\dots,\theta^M\}$1

computes a meta-gradient on $\Theta=\{\theta^0,\theta^1,\dots,\theta^M\}$2, and the server updates the shared initialization by summing client meta-gradients. Missing modalities are handled by masking or zeroing, and robustness arises from meta-training over clients with different modality subsets $\Theta=\{\theta^0,\theta^1,\dots,\theta^M\}$3. In the reported system, fusion is late concatenation, no explicit regularizer for missingness is introduced, and personalization arises implicitly from the inner-loop adaptation $\Theta=\{\theta^0,\theta^1,\dots,\theta^M\}$4 rather than from separate client heads (Tran et al., 2023).

A second family uses prototype substitution for missing or degraded modalities. PMM constructs class-wise modality prototypes online, maintains persistent global prototypes through a running average, and substitutes missing features with the appropriate class prototype during local training. QQR extends this prototype logic to both quantity and quality imbalance through three components: Online Global Prototype Construction (OGPC), Prototypical Quantity Rebalancing (PNR), and Prototypical Quality Rebalancing (PLR). PNR replaces absent modality features with cumulative global class prototypes, while PLR adds a prototype cross-entropy term,

$\Theta=\{\theta^0,\theta^1,\dots,\theta^M\}$5

so that low-quality modality features are pulled toward class prototypes (Wang et al., 22 May 2025, Wang et al., 15 Aug 2025).

A third family reconstructs missing information in a shared latent space rather than in input space. FedMobile introduces a conditional generator $\Theta=\{\theta^0,\theta^1,\dots,\theta^M\}$6 trained by knowledge distillation to produce modality-consistent latent features from labels, aligns generated and observed features with KL divergence, and reconstructs missing modality latents as $\Theta=\{\theta^0,\theta^1,\dots,\theta^M\}$7 when $\Theta=\{\theta^0,\theta^1,\dots,\theta^M\}$8. Server-side generator aggregation is not naive averaging: generators are clustered with K-means over proxy-label latent outputs, representative generators are scored by Shapley values, and both the generator and predictor are aggregated with contribution-aware weighting so that low-quality or harmful nodes are down-weighted (Liu et al., 20 Feb 2025).

A fourth family optimizes communication or computation by selecting what is trained and what is transmitted. mmFedMC keeps per-modality global models at the server and a local decision-level fusion model $\Theta=\{\theta^0,\theta^1,\dots,\theta^M\}$9 at the client. Each client scores modalities using a composite priority

$\theta^m$0

where $\theta^m$1 is a Shapley-based impact estimate, $\theta^m$2 encodes model size, and $\theta^m$3 encodes recency. The server then selects, for each modality, the $\theta^m$4 clients with the smallest local losses before FedAvg aggregation of that modality model (Yuan et al., 2024). FlexMod addresses the training side rather than the uplink side: it assesses encoder quality from global prototypes, computes modality importance by Shapley values on a validation set, and uses DDPG to choose a round-specific tradeoff parameter $\theta^m$5 for solving a time-budgeted allocation problem over modality combinations (Bian et al., 2024).

A fifth family adapts multimodal transformers to federated heterogeneity. FedCola targets settings with image-only, text-only, and paired vision-language clients. Uni-modal clients temporarily mix complementary transformer blocks through gated linear sublayers of the form

$\theta^m$6

compress the result to $\theta^m$7 before upload, and the server performs layer-wise collaborative aggregation that preferentially mixes self-attention layers while preserving MLP specialization. This design explicitly addresses both the cross-modality gap and the in-modality gap among clients with different objectives (Sun et al., 2024).

Taken together, these methods imply no single algorithmic consensus. Zero-masked late fusion, prototype substitution, latent-space reconstruction, modality/client selection, reinforcement-learned scheduling, and layer-wise transformer collaboration all solve different slices of the MMO-FL problem, and their assumptions are not interchangeable.

4. Theory and online properties

The strongest explicit MMO-FL theory in the cited set concerns online regret under missingness and quality degradation. For the IoT MMO-FL formulation without missing modalities, the regret remains sublinear under convexity, differentiability, Lipschitz partial gradients, bounded coordinates, and a step size $\theta^m$8: with $\theta^m$9 local updates, the bound remains $\theta^0$0 up to an additional drift term reflecting multi-step local updates. With missing modalities, however, the bound becomes

$\theta^0$1

where $\theta^0$2 lower-bounds the proportion of available modalities. The linear term in $\theta^0$3 is the explicit performance degradation induced by missing modalities (Wang et al., 22 May 2025).

QQR generalizes this analysis from missingness alone to combined quantity and quality imbalance. Under Assumptions 1–6, the paper shows that quality imbalance adds a linear term governed by $\theta^0$4, yielding

$\theta^0$5

while combined quantity and quality imbalance yields

$\theta^0$6

The interpretation given in the paper is that low-quality data increase gradient bias and variance through $\theta^0$7, whereas missing modalities reduce effective per-modality sample support through $\theta^0$8; PLR is designed to reduce $\theta^0$9 and PNR to increase effective $Z_k^{t,m}=\theta^m(X_k^{t,m})$0 (Wang et al., 15 Aug 2025).

Other frameworks are more cautious theoretically. FedCola sketches a FedAvg-style convergence bound under smoothness, bounded variance, and a positive semidefinite server collaboration matrix, supporting the stability of collaborative aggregation for transformer blocks (Sun et al., 2024). By contrast, mmFedMC explicitly states that it does not provide formal convergence or generalization guarantees, and FedMobile states that it does not present formal convergence theorems, although it argues that stability is improved by latent-space operation, contribution-aware weighting, and clustered Shapley aggregation (Yuan et al., 2024, Liu et al., 20 Feb 2025).

The online status of the different systems is uneven. PMM and QQR are explicitly online, with sliding windows or fixed-size buffers and round-by-round local OGD on continually refreshed data (Wang et al., 22 May 2025, Wang et al., 15 Aug 2025). 3FM is not an online algorithm in the strict sense, but its MAML initialization is presented as directly relevant to MMO-FL because it adapts quickly across changing modality subsets (Tran et al., 2023). FedMobile, FedCola, mmFedMC, and FlexMod are synchronous round-based systems; each paper nevertheless outlines natural extensions to streaming mini-batches, asynchronous aggregation, staleness-aware weights, or drift-aware recency and scheduling (Liu et al., 20 Feb 2025, Sun et al., 2024, Yuan et al., 2024, Bian et al., 2024).

5. Experimental domains and empirical findings

The empirical literature spans wearable sensing, vision-language learning, medical and mobility applications, and sentiment analysis. 3FM evaluates multimodal federated meta-learning on a hand-aligned three-modality digit benchmark built from Sign-Language-Digits, MNIST, and Free-Spoken-Digit. Under six missing-modality scenarios, the best reported configuration with $Z_k^{t,m}=\theta^m(X_k^{t,m})$1, $Z_k^{t,m}=\theta^m(X_k^{t,m})$2, and $Z_k^{t,m}=\theta^m(X_k^{t,m})$3 achieved test accuracies of spectrogram/sign $Z_k^{t,m}=\theta^m(X_k^{t,m})$4, img $Z_k^{t,m}=\theta^m(X_k^{t,m})$5, sign $Z_k^{t,m}=\theta^m(X_k^{t,m})$6, img/spect $Z_k^{t,m}=\theta^m(X_k^{t,m})$7, img/sign $Z_k^{t,m}=\theta^m(X_k^{t,m})$8, and spect $Z_k^{t,m}=\theta^m(X_k^{t,m})$9; with $$F_t(\Theta,\mathcal{D}^t)=\frac{1}{K}\sum_{k=1}^K f_t(\theta^0(Z_k^{t,1},\dots,Z_k^{t,M}),Y_k^t),$$0 and $$F_t(\Theta,\mathcal{D}^t)=\frac{1}{K}\sum_{k=1}^K f_t(\theta^0(Z_k^{t,1},\dots,Z_k^{t,M}),Y_k^t),$$1, several scenarios were also strong, including spect $$F_t(\Theta,\mathcal{D}^t)=\frac{1}{K}\sum_{k=1}^K f_t(\theta^0(Z_k^{t,1},\dots,Z_k^{t,M}),Y_k^t),$$2 and spect/sign $$F_t(\Theta,\mathcal{D}^t)=\frac{1}{K}\sum_{k=1}^K f_t(\theta^0(Z_k^{t,1},\dots,Z_k^{t,M}),Y_k^t),$$3 (Tran et al., 2023).

FedMobile evaluates WoT-relevant multimodal FL under incomplete modalities on USC-HAD, MHAD, ADM, C-MHAD, and FLASH. The paper reports robustness up to $$F_t(\Theta,\mathcal{D}^t)=\frac{1}{K}\sum_{k=1}^K f_t(\theta^0(Z_k^{t,1},\dots,Z_k^{t,M}),Y_k^t),$$4 missing modality information and in two-modality-missing settings. Selected results include MHAD at $$F_t(\Theta,\mathcal{D}^t)=\frac{1}{K}\sum_{k=1}^K f_t(\theta^0(Z_k^{t,1},\dots,Z_k^{t,M}),Y_k^t),$$5, where FedMobile reached $$F_t(\Theta,\mathcal{D}^t)=\frac{1}{K}\sum_{k=1}^K f_t(\theta^0(Z_k^{t,1},\dots,Z_k^{t,M}),Y_k^t),$$6 versus AutoFed’s $$F_t(\Theta,\mathcal{D}^t)=\frac{1}{K}\sum_{k=1}^K f_t(\theta^0(Z_k^{t,1},\dots,Z_k^{t,M}),Y_k^t),$$7; USC-HAD at $$F_t(\Theta,\mathcal{D}^t)=\frac{1}{K}\sum_{k=1}^K f_t(\theta^0(Z_k^{t,1},\dots,Z_k^{t,M}),Y_k^t),$$8, where FedMobile reached $$F_t(\Theta,\mathcal{D}^t)=\frac{1}{K}\sum_{k=1}^K f_t(\theta^0(Z_k^{t,1},\dots,Z_k^{t,M}),Y_k^t),$$9 versus $$\mathrm{Reg}_T=\sum_{t=1}^T F_t(\Theta^t;\mathcal{D}^t)-\sum_{t=1}^T F_t(\Theta^*;\mathcal{D}^t),$$0; and ADM with audio/radar missing at $$\mathrm{Reg}_T=\sum_{t=1}^T F_t(\Theta^t;\mathcal{D}^t)-\sum_{t=1}^T F_t(\Theta^*;\mathcal{D}^t),$$1, where FedMobile reached $$\mathrm{Reg}_T=\sum_{t=1}^T F_t(\Theta^t;\mathcal{D}^t)-\sum_{t=1}^T F_t(\Theta^*;\mathcal{D}^t),$$2 versus $$\mathrm{Reg}_T=\sum_{t=1}^T F_t(\Theta^t;\mathcal{D}^t)-\sum_{t=1}^T F_t(\Theta^*;\mathcal{D}^t),$$3. The added communication cost from generator transmission is reported as approximately $$\mathrm{Reg}_T=\sum_{t=1}^T F_t(\Theta^t;\mathcal{D}^t)-\sum_{t=1}^T F_t(\Theta^*;\mathcal{D}^t),$$4 MB per round, or about $$\mathrm{Reg}_T=\sum_{t=1}^T F_t(\Theta^t;\mathcal{D}^t)-\sum_{t=1}^T F_t(\Theta^*;\mathcal{D}^t),$$5 more than FedProx under the USC setting (Liu et al., 20 Feb 2025).

mmFedMC evaluates communication-efficient multimodal FL on ActionSense, UCI-HAR, PTB-XL, MELD, and DFC23. Its main claim is that it can achieve accuracy comparable to several baselines while reducing communication overhead by over $$\mathrm{Reg}_T=\sum_{t=1}^T F_t(\Theta^t;\mathcal{D}^t)-\sum_{t=1}^T F_t(\Theta^*;\mathcal{D}^t),$$6. At a cumulative $$\mathrm{Reg}_T=\sum_{t=1}^T F_t(\Theta^t;\mathcal{D}^t)-\sum_{t=1}^T F_t(\Theta^*;\mathcal{D}^t),$$7 MB per client, the default configuration reports, for example, ActionSense IID accuracy $$\mathrm{Reg}_T=\sum_{t=1}^T F_t(\Theta^t;\mathcal{D}^t)-\sum_{t=1}^T F_t(\Theta^*;\mathcal{D}^t),$$8 at $$\mathrm{Reg}_T=\sum_{t=1}^T F_t(\Theta^t;\mathcal{D}^t)-\sum_{t=1}^T F_t(\Theta^*;\mathcal{D}^t),$$9 MB per iteration and Natural accuracy $\Theta^*$0 at $\Theta^*$1 MB per iteration, while several baseline fusion strategies consume substantially more communication. The paper also reports that lower-loss client selection consistently outperforms higher-loss selection in its multimodal, heterogeneous setting (Yuan et al., 2024).

FedCola studies multimodal transformers with uni-modal and paired clients on CIFAR-100, AG NEWS, Flickr30k subsets, COCO Captions, and medical-domain stress tests. In the default setting with non-IID Dirichlet $\Theta^*$2 and participation $\Theta^*$3, FedCola achieved $\Theta^*$4 on Flickr versus FedAvg’s $\Theta^*$5 and $\Theta^*$6 on COCO versus FedAvg’s $\Theta^*$7. The paper further reports robustness under stronger heterogeneity, lower participation, and modality-imbalanced participation, while preserving or slightly improving uni-modal classification performance (Sun et al., 2024).

The explicitly online IoT systems, PMM and QQR, evaluate on UCI-HAR and MVSA-Single with sliding-window data generation. PMM reports that prototype substitution substantially outperforms Partial Modality and Zero Filling baselines under missing modalities, may surpass Full Modality in later rounds as prototypes become more representative, and remains effective under high missing rates, non-IID splits, quantized prototype uploads with $\Theta^*$8 bits, and delayed prototype updates (Wang et al., 22 May 2025). QQR reports that PNR consistently outperforms Zero Padding under quantity imbalance, that PLR improves over using low-quality data directly under quality imbalance, and that low-bit prototype quantization causes only minor accuracy reduction while substantially reducing communication (Wang et al., 15 Aug 2025).

6. Assumptions, limitations, and future directions

Several assumptions recur across the literature. 3FM assumes a fixed global modality set, label-aligned multimodal samples, simple concatenation-based late fusion, and synchronous federated rounds; it does not support dynamic discovery of brand-new modalities in the reported experiments (Tran et al., 2023). PMM and QQR are also synchronous in their reported form, although both are explicitly online in the sense of streaming local data and fixed-size local buffers (Wang et al., 22 May 2025, Wang et al., 15 Aug 2025). FedCola is synchronous and does not implement online or asynchronous arrivals; FlexMod likewise assumes full participation, comparable client compute capabilities, and no missing modalities at clients (Sun et al., 2024, Bian et al., 2024).

The mitigation mechanisms themselves embody different tradeoffs. Zero masking, as in 3FM, is simple and avoids explicit imputation, but it does not attempt cross-modal reconstruction (Tran et al., 2023). Prototype-based substitution, as in PMM and QQR, is lightweight and communication-conscious, but it presumes reliable class semantics and can be affected by prototype staleness or class-inference errors (Wang et al., 22 May 2025, Wang et al., 15 Aug 2025). FedMobile’s latent generator avoids raw-space reconstruction and adds contribution-aware aggregation, but it relies on a shared latent space and a proxy dataset for contribution estimation; the paper notes that a misaligned or too-small proxy set can make aggregation weights noisy (Liu et al., 20 Feb 2025). FlexMod improves local resource allocation but does not address missing modalities, asynchronous aggregation, or partial participation, and it does not analyze the scalability of exact Shapley computation as the number of modalities grows (Bian et al., 2024).

Privacy and security remain incomplete. FedMobile emphasizes that only model weights and generator parameters are transmitted and that reconstruction occurs in latent space rather than raw space, but it still recommends secure aggregation and differential privacy as additional protections (Liu et al., 20 Feb 2025). PMM and QQR note that prototype sharing may raise privacy concerns and identify differentially private prototype sharing or secure aggregation as natural future extensions (Wang et al., 22 May 2025, Wang et al., 15 Aug 2025). FedCola similarly notes that model inversion and membership inference are not directly addressed (Sun et al., 2024).

Future directions are remarkably consistent. The literature repeatedly points toward streaming meta-updates, continual or drift-aware prototype maintenance, adaptive modality selection, uncertainty-aware reconstruction, asynchronous and fault-tolerant aggregation, robust aggregation against adversarial or low-utility updates, and mechanisms for continual modality discovery rather than operation over a fixed modality set (Tran et al., 2023, Liu et al., 20 Feb 2025, Wang et al., 15 Aug 2025, Wang et al., 22 May 2025, Yuan et al., 2024, Sun et al., 2024). This suggests that MMO-FL is best understood not as a single settled algorithm, but as a research program centered on online multimodal learning under federated privacy constraints, with missingness, modality quality, communication, and continual adaptation treated as coequal system variables.