QA-MoE: Quality-Aware Mixture-of-Experts

• The paper introduces QA-MoE, a paradigm that integrates quality signals into MoE models to dynamically route features and enhance prediction robustness.
• It employs quality-aware gating to modulate expert selection using reliability metrics, thereby improving interpretability and performance.
• Applications span speech quality, video quality, and multimodal sentiment analysis, demonstrating significant gains in correlation and robustness metrics.

Quality-Aware Mixture-of-Experts (QA-MoE) is a model design paradigm that explicitly incorporates quality or reliability signals into a Mixture-of-Experts (MoE) architecture to enable adaptive, interpretable, and robust predictions across diverse domains such as speech and video quality assessment and multimodal sentiment analysis. By integrating quality-aware gating or expert selection and, in some designs, explicit quality or artifact representations, the QA-MoE framework offers improved performance, interpretability, and adaptability under varying input degradation scenarios, granularity levels, and domain-specific challenges (Hu et al., 8 Jul 2025, Zhu et al., 7 Apr 2026, Feng et al., 1 Dec 2025).

1. Foundational Principles and Architectural Overview

At its core, QA-MoE extends the standard MoE framework by using quality or reliability indicators to modulate how feature representations are routed through a bank of expert subnetworks. A typical setup includes:

• Backbone Feature Extractor: Pre-trained encoders such as wav2vec2 for speech (Hu et al., 8 Jul 2025) or ViT-B/16 for video (Feng et al., 1 Dec 2025) generate compact, high-level representations.
• Expert Networks: Each expert (E₁, ..., E_N) is an independent subnetwork, often with small MLP or gated architectures, tailored for specific perceptual domains or semantic subspaces.
• Quality-Aware Gating: Gating networks produce a soft or sparse assignment over the expert pool, modulated by estimated reliability or quality metrics, e.g., via self-supervised uncertainty (Zhu et al., 7 Apr 2026) or domain-specific gating (Feng et al., 1 Dec 2025).
• Multi-Branch Prediction: Output heads generate scalar quality scores, artifact vectors, or regression/classification outputs, supporting both aggregate and diagnosis-oriented assessment.

In general, the QA-MoE architecture enables dynamic expert routing based on explicit or learned notions of input quality, thereby adapting inference to the signal properties and increasing robustness to corruption, missingness, or artifact diversity.

2. Mathematical Formulation of Expert Routing and Quality Gating

The QA-MoE routing adopts a mathematically rigorous approach to leverage both semantic content and estimated quality:

• Gating Function: For global feature $h\in\mathbb{R}^d$,

$$g(h) = \mathrm{Softmax}(W_g\,h + b_g) \in \mathbb{R}^N,\quad g_i = \frac{\exp(w_i^\top h + b_i)}{\sum_{j=1}^N \exp(w_j^\top h + b_j)}$$

with trainable $W_g\in\mathbb{R}^{N\times d}$, $b_g\in\mathbb{R}^N$ (Hu et al., 8 Jul 2025).

• Quality Suppression (Multimodal Extension): For modality $m$ with aleatoric uncertainty $\boldsymbol\sigma^2_m$:

$$r_m = \frac{1}{1 + \frac{1}{d}\sum_{k=1}^d \sigma^2_{m,k}} \in (0,1]$$

used to modulate expert aggregation:

$$\mathbf{y}_m = r_m \sum_{i=1}^N g_i(\boldsymbol\mu_m) E_i(\boldsymbol\mu_m) + (1 - r_m)\mathbf{y}_\text{prior}$$

where $\boldsymbol\mu_m$ is the predicted mean, $E_i$ are experts (e.g., GLUs), and $$g(h) = \mathrm{Softmax}(W_g\,h + b_g) \in \mathbb{R}^N,\quad g_i = \frac{\exp(w_i^\top h + b_i)}{\sum_{j=1}^N \exp(w_j^\top h + b_j)}$$0 is a fallback embedding (Zhu et al., 7 Apr 2026).

• Final Prediction: For regression quality assessment:

$$g(h) = \mathrm{Softmax}(W_g\,h + b_g) \in \mathbb{R}^N,\quad g_i = \frac{\exp(w_i^\top h + b_i)}{\sum_{j=1}^N \exp(w_j^\top h + b_j)}$$1

The gating and expert outputs may be further regularized for diversity and sparsity to encourage expert specialization and stable routing (Hu et al., 8 Jul 2025).

3. Domain-Specific Implementations

Speech Quality Assessment

The QA-MoE for speech quality uses a self-supervised pretrained encoder (e.g., wav2vec2), with experts as small MLPs and a gating network followed by dual-head outputs: one for MOS regression, one for synthesis-model classification. The overall loss combines smoothed $$g(h) = \mathrm{Softmax}(W_g\,h + b_g) \in \mathbb{R}^N,\quad g_i = \frac{\exp(w_i^\top h + b_i)}{\sum_{j=1}^N \exp(w_j^\top h + b_j)}$$2 regression, cross-entropy classification with label smoothing, and regularization for expert diversity and gating sparsity. Dynamic weighting of task losses across progressive, multi-stage training is used to balance classification and regression objectives (Hu et al., 8 Jul 2025).

Multimodal Sentiment Analysis

In QA-MoE for multimodal sentiment analysis, each modality is encoded into a probabilistic representation with a learned variance reflecting aleatoric uncertainty. The quality gating factor $$g(h) = \mathrm{Softmax}(W_g\,h + b_g) \in \mathbb{R}^N,\quad g_i = \frac{\exp(w_i^\top h + b_i)}{\sum_{j=1}^N \exp(w_j^\top h + b_j)}$$3 is learned self-supervised via heteroscedastic regression. Expert routing adapts per-modality according to the estimated reliability, and the framework supports clean, missing, and noisy input regimes in a unified manner. A dual-branch head predicts both the sentiment score and its uncertainty (Zhu et al., 7 Apr 2026).

Video Quality Assessment

In unified VQA, three experts specialize in spatial, color, and temporal domains, respectively. The model uses proxy metrics for each domain (e.g., VMAF, HDR-VDP-3, VFIPS) and a multi-proxy ranking-based loss to encourage each expert to focus on relevant artifact types. An optional gating mechanism assigns soft weights for combining expert outputs. A multi-task diagnostic head produces both a scalar quality score and a multi-dimensional artifact probability vector, trained via weak supervision using artifact labels generated during database construction (Feng et al., 1 Dec 2025).

4. Training Paradigms and Data Strategies

QA-MoE models typically employ structured, multi-stage training regimes tailored to their domain constraints:

• Stage-Wise Training: Begin with auxiliary tasks (e.g., model classification or domain artifact ranking), then proceed to joint or fine-tuning stages for the main regression or quality task.
• Weighted Loss Scheduling: Dynamic adjustment of task weights ($$g(h) = \mathrm{Softmax}(W_g\,h + b_g) \in \mathbb{R}^N,\quad g_i = \frac{\exp(w_i^\top h + b_i)}{\sum_{j=1}^N \exp(w_j^\top h + b_j)}$$4, $$g(h) = \mathrm{Softmax}(W_g\,h + b_g) \in \mathbb{R}^N,\quad g_i = \frac{\exp(w_i^\top h + b_i)}{\sum_{j=1}^N \exp(w_j^\top h + b_j)}$$5) over epochs to prioritize auxiliary or primary objectives.
• Data Augmentation: Leverage diverse synthetic datasets or on-the-fly input degradation, such as adding noise or modality dropout, to promote expert specialization and robust quality gating (Hu et al., 8 Jul 2025, Zhu et al., 7 Apr 2026).
• Regularization: Expert diversity (e.g., orthogonality, entropy maximization) and sparsity constraints on gating ensure non-trivial expert assignment and domain separation.

Optimization practices include Adam(W) optimizers, learning rate scheduling (e.g., cosine annealing), weight decay, and dropout to prevent overfitting, with early stopping based on validation performance.

5. Empirical Performance and Diagnostic Interpretability

Extensive experiments demonstrate the advantages and remaining challenges of QA-MoE across domains:

Application	Key Metrics	Outcome and Insights
Speech QA (Hu et al., 8 Jul 2025)	System MOS: MSE=0.056, LCC=0.978, SRCC=0.913, KTAU=0.758	QA-MoE attains lowest MSE (20% gain over 2nd best), SOTA correlation
	Utterance MOS: MSE=0.277, LCC=0.811	Middle-of-pack; limited utterance-level generalization
Multimodal SA (Zhu et al., 7 Apr 2026)	ACC₇=53.6% (MOSI), 58.4% (MOSEI); robust to noise/miss	QA-MoE outperforms spectrum-aware retrained SAM-LML, smooth reliability curve
Video QA (Feng et al., 1 Dec 2025)	SROCC>0.9 (Full-ref HD/UHD), Artifact F1>0.9	Beats 18+ baselines, delivers interpretable artifact vectors

A distinguishing property is QA-MoE's ability to provide interpretable attributions (e.g., artifact reason vectors in video, synthesis fingerprinting in speech) and smooth robustness across degradation spectra (e.g., missingness, noise, or domain shifts). Diagnostic heads and multi-task setups furnish actionable information beyond scalar quality, supporting practical deployment.

6. Limitations and Prospective Advances

While QA-MoE achieves state-of-the-art results in multiple settings, certain limitations remain:

• Granularity Gap: QA-MoE struggles to generalize at fine-grained (e.g., utterance-level) scales in speech, partly due to insufficient modeling of subjective raters or fine prosodic cues (Hu et al., 8 Jul 2025).
• Quality Signal Interpretability: Current quality estimates are implicit and agnostic to specific artifact types, limiting targeted interventions (Zhu et al., 7 Apr 2026).
• Computational Efficiency: MoE routing incurs increased inference cost, though sparse top-$$g(h) = \mathrm{Softmax}(W_g\,h + b_g) \in \mathbb{R}^N,\quad g_i = \frac{\exp(w_i^\top h + b_i)}{\sum_{j=1}^N \exp(w_j^\top h + b_j)}$$6 gating strategies can mitigate this overhead (Zhu et al., 7 Apr 2026).
• Generalization Across Domains: Extensions to additional modalities (e.g., LiDAR, real-world sensor streams) are ongoing research areas.

Suggested future work includes developing fine-grained, type-specific quality predictors, lightweight and structured routing architectures (such as low-rank or pruned experts), and expanding the Continuous Reliability Spectrum to new domains and modalities (Zhu et al., 7 Apr 2026).

7. Broader Significance and Outlook

The QA-MoE paradigm reframes quality assessment problems across signal domains as adaptive routing and expert selection driven by learned, data-dependent reliability cues. This approach enables robust, interpretable, and scalable models—offering actionable artifact diagnostics and graceful performance under diverse real-world distortions and missingness. Research contributions such as "Speech Quality Assessment Model Based on Mixture of Experts" (Hu et al., 8 Jul 2025), "QA-MoE: Towards a Continuous Reliability Spectrum..." (Zhu et al., 7 Apr 2026), and "Towards Unified Video Quality Assessment" (Feng et al., 1 Dec 2025) collectively suggest the viability of QA-MoE as a general strategy for quality-problem domains and motivate future generalizations into broader classes of multimodal and artifact-rich tasks.