Modality QQR: Rebalancing for MMO-FL

• Modality QQR is a prototype-driven algorithm that addresses both missing and noisy modalities by rebalancing quantity and quality in online federated learning.
• It integrates online global prototype construction, prototypical quantity rebalancing, and quality rebalancing to substitute missing features and correct degraded representations.
• The method achieves improved learning performance with theoretical regret bounds and empirical validations under varying modality imbalances and noise conditions.

Modality Quantity and Quality Rebalanced (QQR) is a prototype learning-based algorithm for multimodal online federated learning (MMO-FL) that mitigates modality quantity imbalance and modality quality imbalance by maintaining cumulative global prototypes and using them for missing-modality feature substitution and low-quality representation regularization (Wang et al., 15 Aug 2025). In adjacent multimodal literature, the term also invites a broader interpretation: modality imbalance is not only a problem of missing or noisy inputs, but also a problem of unequal informational contribution, unequal sensitivity, and unequal reliability across modalities. Related work operationalizes these factors through sampling discrepancy, gradient sensitivity, uncertainty-derived reliability, or quality estimators, rather than through a single universal definition of “quality” (Jiang et al., 5 Mar 2025, Li et al., 2024, Zhu et al., 7 Apr 2026, Mai et al., 3 Mar 2026).

1. Conceptual scope and terminology

In the strict formulation introduced for MMO-FL, modality quantity imbalance refers to unequal sample availability across modalities, modeled by modality-availability indicators $p_{k,n}^{t,m}\in\{0,1\}$, while modality quality imbalance refers to integrity or noise differences across modalities, modeled by modality-quality indicators $q_k^{t,m}\in\{0,1\}$ (Wang et al., 15 Aug 2025). The motivating setting is IoT instability: sensors may fail intermittently, sample asynchronously, or generate degraded observations. Under the basic multimodal objective, only samples containing all modalities can directly contribute to joint multimodal training, so partially observed samples become a direct source of inefficiency and bias.

Related multimodal papers use the same vocabulary differently. "Rebalanced Multimodal Learning with Data-aware Unimodal Sampling" defines a neglected imbalance source at the sampling level: even equal unimodal sample counts can contain different informational content, so quantity itself can be imbalanced by sampling policy; its quality-related signal is only implicit, derived from a cumulative confidence-based discrepancy score (Jiang et al., 5 Mar 2025). "MBQ: Modality-Balanced Quantization for Large Vision-LLMs" defines imbalance through modality-dependent sensitivity during post-training quantization calibration, where language and vision tokens should not receive equal optimization weight (Li et al., 2024). "QA-MoE: Towards a Continuous Reliability Spectrum with Quality-Aware Mixture of Experts for Robust Multimodal Sentiment Analysis" places missingness and degradation on a single continuous reliability axis $r_m\in(0,1]$ (Zhu et al., 7 Apr 2026). "Addressing Missing and Noisy Modalities in One Solution: Unified Modality-Quality Framework for Low-quality Multimodal Data" treats missing modality as a special type of noisy modality and predicts per-modality quality scores $\alpha_m$ (Mai et al., 3 Mar 2026). Taken together, these works show that “quality” may denote corruption severity, training usefulness, downstream sensitivity, or estimated reliability, depending on the framework.

2. Formal MMO-FL setting and theoretical characterization

The QQR algorithm is defined in an online federated regime with $K$ clients, $M$ modalities, and $T$ global rounds. At round $t$, client $k$ observes a multimodal dataset $\mathcal D_k^t$ whose entries contain modality samples $q_k^{t,m}\in\{0,1\}$0, labels $q_k^{t,m}\in\{0,1\}$1, availability indicators $q_k^{t,m}\in\{0,1\}$2, and modality-quality indicators $q_k^{t,m}\in\{0,1\}$3 (Wang et al., 15 Aug 2025). The global model is $q_k^{t,m}\in\{0,1\}$4, where $q_k^{t,m}\in\{0,1\}$5 is the encoder for modality $q_k^{t,m}\in\{0,1\}$6 and $q_k^{t,m}\in\{0,1\}$7 is the shared head. For modality $q_k^{t,m}\in\{0,1\}$8, the feature representation is $q_k^{t,m}\in\{0,1\}$9, and the round loss is

$r_m\in(0,1]$0

Because data arrives online, the objective is cumulative regret against the best fixed model in hindsight: $r_m\in(0,1]$1 Under convexity, Lipschitzness, bounded parameter components, and bounded gradient variation across local iterations, the imbalance-free MMO-FL system admits an $r_m\in(0,1]$2 regret rate when $r_m\in(0,1]$3 (Wang et al., 15 Aug 2025).

The paper then separates the two imbalance sources analytically. Quality imbalance is modeled as bounded distortion between clean and degraded gradients: $r_m\in(0,1]$4 where $r_m\in(0,1]$5 is the maximum gradient deviation induced by noise in modality $r_m\in(0,1]$6. This yields a regret order of

$r_m\in(0,1]$7

Quantity imbalance is modeled through bounded deviation between a per-sample gradient and the full-batch gradient,

$r_m\in(0,1]$8

and enters the regret via the minimum actually available modality sample count

$r_m\in(0,1]$9

With both imbalance types present, the regret order becomes

$\alpha_m$0

The theoretical interpretation is explicit: quality imbalance harms learning through gradient distortion, whereas quantity imbalance harms learning through reduced usable multimodal coverage (Wang et al., 15 Aug 2025).

3. QQR algorithmic structure

QQR operates in parallel with standard federated training and consists of three components: Online Global Prototype Construction (OGPC), Prototypical Quantity Rebalancing (PNR), and Prototypical Quality Rebalancing (PLR) (Wang et al., 15 Aug 2025).

OGPC constructs class- and modality-specific prototypes from trustworthy local features. A local prototype $\alpha_m$1 is the mean modality-$\alpha_m$2 embedding of client-$\alpha_m$3, round-$\alpha_m$4 samples that satisfy

$\alpha_m$5

Thus prototype construction uses only samples for which the modality is present, the modality is normal quality, and the class label is $\alpha_m$6. The server first aggregates these into instantaneous global prototypes,

$\alpha_m$7

and then maintains cumulative global prototypes by running average,

$\alpha_m$8

These cumulative prototypes are stored server-side and broadcast to clients.

PNR addresses quantity imbalance through feature-level substitution rather than raw-data imputation. For sample $\alpha_m$9 at client $K$0, round $K$1, modality $K$2,

$K$3

where $K$4 is the sample’s class label. If the modality is present, QQR uses the encoder output; if it is missing, QQR inserts the class-conditioned cumulative global prototype. This is the quantity-rebalancing step: it increases the effective usable modality count by making partially observed samples available to the multimodal head.

PLR addresses quality imbalance with a prototype-based regularizer. For modality $K$5, the paper defines a Prototype Cross Entropy loss

$K$6

where $K$7 is Euclidean distance. The full quality-rebalancing loss is

$K$8

The multiplicative factor $K$9 is decisive: if the modality is normal quality, the prototype regularizer vanishes; if the modality is low quality, the feature is pulled toward its class prototype and away from other class prototypes. Quantity rebalancing is therefore substitution for missingness, while quality rebalancing is representation correction for degradation.

4. Comparative operationalizations of quantity and quality

The broader multimodal literature does not converge on a single implementation of quantity–quality rebalance. Instead, it provides several operational families.

Method	Quantity mechanism	Quality or reliability mechanism
QQR (Wang et al., 15 Aug 2025)	Class-conditioned prototype substitution for missing modality features	Prototype regularization activated by binary quality indicator
DUS (Jiang et al., 5 Mar 2025)	Adaptive per-modality sample counts	Cumulative discrepancy from correct-class confidence
MBQ (Li et al., 2024)	Splits modality losses and counters uniform token aggregation indirectly	Gradient-weighted reconstruction via modality sensitivity
QA-MoE (Zhu et al., 7 Apr 2026)	Missingness treated as $M$0	Aleatoric-uncertainty reliability with expert suppression
UMQ (Mai et al., 3 Mar 2026)	Missing modality handled as extreme low quality	Rank-guided quality estimation, enhancement, and MQ-MoE

DUS is a direct quantity-rebalancing method and only a partial quality-rebalancing method. Its central monitored quantity is the cumulative modality discrepancy

$M$1

derived from the average confidence assigned to the correct class by modality $M$2; this signal drives heuristic or RL-based per-iteration sample counts $M$3 (Jiang et al., 5 Mar 2025). MBQ is the opposite asymmetry: it is primarily quality-aware. It weights vision and language reconstruction errors during quantization calibration by gradient magnitudes $M$4 and $M$5, thereby prioritizing the more loss-sensitive modality (Li et al., 2024). QA-MoE and UMQ are closer to a unified treatment. QA-MoE defines a scalar modality reliability

$M$6

and interpolates between routed expert computation and a learnable prior,

$M$7

so missing and degraded modalities are different points on a single spectrum (Zhu et al., 7 Apr 2026). UMQ similarly unifies noisy and missing modalities under low quality, learns modality scores $M$8, enhances low-quality representations with cross-modal and modality-specific cues, and routes by modality-quality pattern (Mai et al., 3 Mar 2026).

A common misconception is that every reliability-aware multimodal method is a QQR method. The nearby "Multi-QuAD" summary explicitly describes QADM-Net as not a modality quantity–quality rebalance framework in the literal sense; it is a quality-adaptive dynamic inference method that adjusts network depth and parameters according to estimated confidence, rather than explicitly rebalancing modality quantity (Shen et al., 2024).

5. Empirical behavior and systems implications

The QQR paper evaluates on two real-world multimodal datasets adapted into online streams: UCI-HAR, using accelerometer and gyroscope modalities across five clients, and MVSA-Single, using text and image modalities across five clients (Wang et al., 15 Aug 2025). Quantity imbalance is simulated with intra-round and inter-round ratios $M$9 and $T$0; quality imbalance is controlled by $T$1, with low-quality data generated by additive white Gaussian noise at SNR $T$2 dB. For quantity imbalance, the reported ranking is

$T$3

where FC is Full Collection, PNR is prototype-based quantity rebalancing, ZP is Zero Padding, and IS is Incomplete Subset training. For quality imbalance, the ranking is

$T$4

where PQ is Pristine Quality, PLR is prototype-based quality rebalancing, and BQ is uncorrected degraded training. The paper also reports that prototype upload can be quantized with $T$5 bits per component, causing only minor performance loss while reducing communication cost (Wang et al., 15 Aug 2025).

Adjacent work reports similar empirical patterns once the rebalancing signal is aligned with the operational definition of imbalance. DUS improves Kinetics-Sounds from $T$6 Acc / $T$7 MAP under the baseline to $T$8 Acc / $T$9 MAP in its full discrepancy-plus-RL form, and shows that cumulative monitoring is better than noisy per-batch discrepancy alone (Jiang et al., 5 Mar 2025). MBQ reports task-accuracy improvements of up to $t$0 and $t$1 under W3 and W4A8 quantization, respectively, and also implements a fused W3 GPU kernel yielding a $t$2 decode speedup on LLaVA-onevision-7B on RTX 4090 (Li et al., 2024). QA-MoE reports $t$3 on clean CMU-MOSI and an average $t$4 under the random missing protocol, compared with $t$5 for SAM-LML, while retaining a one-checkpoint-for-all property across reliability conditions (Zhu et al., 7 Apr 2026). UMQ remains best or near-best across missing-rate and Gaussian-noise experiments on MOSI and MOSEI, with average missing-modality results of $t$6 on MOSI and $t$7 on MOSEI for Acc2 / F1 / Acc7 (Mai et al., 3 Mar 2026).

These results do not establish a single dominant mechanism, but they do indicate a consistent pattern: multimodal systems improve when imbalance is treated as a structured modality-level phenomenon rather than as undifferentiated regularization noise.

6. Assumptions, limitations, and open questions

QQR makes several explicit assumptions. It requires class labels $t$8, modality-availability indicators $t$9, modality-quality indicators $k$0, server-side prototype memory $k$1, and additional communication for prototype exchange (Wang et al., 15 Aug 2025). Prototype substitution uses the sample’s class label $k$2, which is natural in supervised training but less straightforward in semi-supervised or test-time settings. The regret analysis is derived under convexity and boundedness assumptions, whereas the empirical system uses deep multimodal models. The paper also notes that cumulative prototypes may lag behind rapidly shifting data distributions, and that prototype management adds logic and communication beyond standard federated averaging (Wang et al., 15 Aug 2025).

The broader literature exposes a deeper conceptual debate: what exactly counts as modality quality? DUS does not score corruption, missingness, or annotation reliability directly; it uses cumulative confidence on the ground-truth class as a proxy for current informational usefulness (Jiang et al., 5 Mar 2025). MBQ measures downstream loss sensitivity, not input fidelity, and only indirectly addresses raw token-count effects (Li et al., 2024). QA-MoE depends critically on uncertainty estimation quality, and its fusion stage is described only as a standard fusion layer, so reliability-aware reweighting is concentrated before fusion rather than explicitly carried through every layer (Zhu et al., 7 Apr 2026). UMQ absorbs missingness into a low-quality formulation rather than preserving separate missingness indicators in the core method (Mai et al., 3 Mar 2026). QADM-Net, finally, shows that a system can be strongly quality-adaptive without being a literal quantity–quality rebalance framework at all (Shen et al., 2024).

A plausible implication is that “QQR” names both a specific algorithmic design and a broader research question. As a specific method, QQR is prototype-driven MMO-FL with distinct mechanisms for missingness and degradation (Wang et al., 15 Aug 2025). As a broader agenda suggested by adjacent work, it concerns how multimodal systems should redistribute data, computation, precision, or trust when modalities differ in availability, noise level, confidence, or task sensitivity (Jiang et al., 5 Mar 2025, Li et al., 2024, Zhu et al., 7 Apr 2026, Mai et al., 3 Mar 2026). The unresolved issue is not whether rebalancing matters, but which signal—availability, confidence, uncertainty, sensitivity, or learned quality score—should govern it in a given multimodal regime.