QA-MoE: Towards a Continuous Reliability Spectrum with Quality-Aware Mixture of Experts for Robust Multimodal Sentiment Analysis

Abstract: Multimodal Sentiment Analysis (MSA) aims to infer human sentiment from textual, acoustic, and visual signals. In real-world scenarios, however, multimodal inputs are often compromised by dynamic noise or modality missingness. Existing methods typically treat these imperfections as discrete cases or assume fixed corruption ratios, which limits their adaptability to continuously varying reliability conditions. To address this, we first introduce a Continuous Reliability Spectrum to unify missingness and quality degradation into a single framework. Building on this, we propose QA-MoE, a Quality-Aware Mixture-of-Experts framework that quantifies modality reliability via self-supervised aleatoric uncertainty. This mechanism explicitly guides expert routing, enabling the model to suppress error propagation from unreliable signals while preserving task-relevant information. Extensive experiments indicate that QA-MoE achieves competitive or state-of-the-art performance across diverse degradation scenarios and exhibits a promising One-Checkpoint-for-All property in practice.

• The paper presents a novel Continuous Reliability Spectrum that unifies noise intensity and modality missing rate into a unified evaluation framework.
• QA-MoE leverages probabilistic feature modeling and quality-aware routing to dynamically suppress unreliable signals without retraining.
• Experimental validation across diverse benchmarks demonstrates state-of-the-art performance and robust generalization with a one-checkpoint-for-all approach.

Robust Multimodal Sentiment Analysis via Quality-Aware Mixture-of-Experts and the Continuous Reliability Spectrum

Introduction

The paper "QA-MoE: Towards a Continuous Reliability Spectrum with Quality-Aware Mixture of Experts for Robust Multimodal Sentiment Analysis" (2604.05704) addresses critical limitations in Multimodal Sentiment Analysis (MSA) under real-world noisy and incomplete modality conditions. Traditional MSA models predominantly assume ideal, noise-free, and complete modality input, resulting in brittle deployment behaviors when facing unpredictable signal degradation or missing modalities. The authors propose a unified theoretical and practical framework: the Continuous Reliability Spectrum, alongside a Quality-Aware Mixture-of-Experts (QA-MoE) architecture that explicitly quantifies modality reliability using aleatoric uncertainty. This approach robustly routes information through the model, suppressing unreliable or missing signals without retraining or fine-tuning for each degradation scenario.

Figure 1: The Continuous Reliability Spectrum unifies three evaluation protocols defined by noise intensity ($\lambda$) and missing rate ($\eta$); inputs from Text, Audio, and Vision are processed as imperfect multimodal signals.

Continuous Reliability Spectrum and Stochastic Imperfection Modeling

The central theoretical contribution is the Continuous Reliability Spectrum, which provides a unified axis for characterizing both noise intensity ($\lambda$) and missing rate ($\eta$) across modalities. This spectrum subsumes existing discrete evaluations (e.g., only missing, only noisy) into a continuous two-dimensional plane, enabling realistic modeling of real-world reliability dynamics.

The transformation function for corrupted input is rigorously defined, integrating additive noise and stochastic modality dropping. Three core evaluation protocols are thus unified:

• Modality Missingness (vary $\eta$; $\lambda=0$)
• Quality Degradation (vary $\lambda$; $\eta=0$)
• Stochastic Mixture (sample both $\lambda$ and $\eta$ jointly)

This formalism allows for comprehensive benchmarking across all possible corruption intensities, contrasting with previous work that evaluates models on isolated defect types and fixed ratios.

QA-MoE: Quality-Aware Mixture-of-Experts Framework

QA-MoE innovates both in representation and computation flow. The architecture relies on three key mechanisms:

• Probabilistic Feature Modeling encodes input features as distributions $\eta$0, where the mean captures semantic content and the variance quantifies aleatoric uncertainty for each modality.
• Quality-Aware Routing computes a reliability score $\eta$1 as a closed-form function of variance, softly controlling the gating of expert outputs. The expert aggregation is modulated by $\eta$2, ensuring unreliable signals (high uncertainty) contribute less or are bypassed entirely.
• Dual-Branch Prediction produces both the sentiment prediction and its associated total uncertainty, optimized via heteroscedastic regression (negative log-likelihood with sample-dependent variance).

This enables the model to dynamically modulate computation in response to reliability estimates, a principal limitation of all prior deterministic or semantically-routed MoE approaches.

Figure 2: Overview of QA-MoE: (A) Probabilistic Feature Modeling for uncertainty; (B) Quality-Aware Routing guided by quality scores; (C) Dual-Branch Prediction for downstream optimization with uncertainty weighting.

Experimental Validation and Numerical Results

Evaluation covers four high-profile benchmarks (CMU-MOSI, CMU-MOSEI, IEMOCAP, MIntRec) and exposes models to all spectrum regimes—clean, modality-missing, noise, and arbitrary stochastic mixtures.

Notably, QA-MoE achieves state-of-the-art top-line metrics in both ACC$\eta$3 and F1 across all protocols. In several high-corruption and missing rate settings, QA-MoE maintains performance well above prior art:

• In random missing experiments on CMU-MOSI, the average ACC$\eta$4 is 42.0%, exceeding the strongest baseline by 6.1%.
• Under severe noise scenarios ($\eta$5), QA-MoE leads SAM-LML by 0.9%, indicating superior resilience to unstructured corruption.

These results are achieved using a single checkpoint—without retraining—demonstrating robust generalization across unseen degradation settings, a property the authors term "One-Checkpoint-for-All". This is visualized directly in the performance landscape below.

Figure 3: Continuous Reliability Landscape—QA-MoE exhibits a smooth, gently degrading performance surface with no sharp failure points, validating robust generalization as the reliability context varies.

By contrast, prior architectures such as SAM-LML exhibit brittle "reliability cliffs," failing abruptly beyond the narrow training regime.

Figure 4: Reliability landscape of SAM-LML, highlighting a sharp accuracy drop outside the clean regime—demonstrating the lack of robust generalization.

Ablation, Sensitivity, and Interpretability

Ablation experiments show that dropping quality gating, uncertainty modeling (variance), or the universal fallback (global prior) result in substantial performance degradation, verifying that all QA-MoE components contribute essentially to its robustness.

Further analysis demonstrates that routing with $\eta$6 active experts optimally balances capacity and overfitting risk, as evidenced by the performance peak in parameter sensitivity studies.

Figure 5: Parameter $\eta$7 Sensitivity—performance maximizes at $\eta$8 active experts, with both lower and higher $\eta$9 leading to reduced robustness.

Visualization of expert utilization underscores that the model adaptively reallocates computational bandwidth in response to degradation type and location, rather than applying undifferentiated penalties.

Figure 6: Adaptive Routing Patterns—experts are activated selectively depending on input integrity, with some specializing in robust operation under partial signal failure.

Implications and Future Directions

QA-MoE fundamentally advances the practical deployment of MSA in-the-wild, where unpredictable and mixed-signal decomposition is the norm. The explicit uncertainty-driven routing capability is not only practically relevant for affective computing but also generalizes to any decision-making pipeline requiring multimodal trust calibration, e.g., medical imaging, human-computer interaction, and robust conversational agents.

The theoretical contribution—modeling reliability as a spectrum—will likely stimulate follow-up work on reliability-aware architectures. However, integrating explicit fine-grained uncertainty types (e.g., distinguishing between occlusion and blur) and reducing computational routing overhead are open lines for further research.

Conclusion

This work provides a unifying reliability framework for MSA and an effective quality-aware expert routing mechanism, validated on diverse benchmarks with a robust "One-Checkpoint-for-All" capability. The integration of probabilistic modeling, uncertainty-aware routing, and dynamic loss weighting yields consistent gains over deterministic and fixed-ratio approaches, both practically and theoretically. The visual evidence (continuous reliability landscape) confirms that the proposed methodology offers resilience absent in prior art. These advances mark a significant technical development toward deployment-safe multimodal affective AI.