Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis

Abstract: Multimodal sentiment analysis (MSA) integrates heterogeneous text, audio, and visual signals to infer human emotions. While recent approaches leverage cross-modal complementarity, they often struggle to fully utilize weaker modalities. In practice, dominant modalities tend to overshadow non-verbal ones, inducing modality competition and limiting overall contributions. This imbalance degrades fusion performance and robustness under noisy or missing modalities. To address this, we propose a novel model, Enhance-then-Balance Modality Collaboration framework (EBMC). EBMC improves representation quality via semantic disentanglement and cross-modal enhancement, strengthening weaker modalities. To prevent dominant modalities from overwhelming others, an Energy-guided Modality Coordination mechanism achieves implicit gradient rebalancing via a differentiable equilibrium objective. Furthermore, Instance-aware Modality Trust Distillation estimates sample-level reliability to adaptively modulate fusion weights, ensuring robustness. Extensive experiments demonstrate that EBMC achieves state-of-the-art or competitive results and maintains strong performance under missing-modality settings.

• The paper introduces EBMC, a two-stage framework that enhances shared and modality-specific features before balancing fusion through energy-guided coordination and instance-level trust distillation.
• EBMC achieves strong results on MOSI, MOSEI, and IEMOCAP, including 50.34 Acc-7 and 0.833 correlation on MOSI, plus an average IEMOCAP F1 of 86.35%.
• The method improves robustness under missing or corrupted modalities, reaching 78.92 average F1 on incomplete MOSEI and adding negligible overhead beyond the multimodal encoders.

Motivation and problem statement

Multimodal sentiment analysis (MSA) fuses text, audio, and visual signals to predict sentiment, yet most fusion architectures implicitly assume that all modalities contribute in a balanced and reliable manner. In practice, the textual modality dominates both predictive power and gradient flow, while audio and visual streams carry weaker or noisier affective cues and are progressively marginalized during joint optimization—a dynamic the authors characterize as a "Matthew effect" of modality competition (2604.12518). Prior remedies operate largely at the optimization level (gradient modulation, decoupled encoders, prototype rebalancing) but provide limited semantic enhancement of weak modalities, and robustness to noise or missingness is typically handled by separate mechanisms bolted onto the fusion pipeline.

The paper proposes EBMC (Enhance-then-Balance Modality Collaboration), a two-stage framework that addresses imbalance at both the representation level and the optimization level, while embedding instance-level reliability estimation directly into training. The design is modular: Stage I enhances unimodal representations; Stage II balances modality contributions during fusion.

Method: enhance-then-balance architecture

Stage I — representation enhancement. Modality Semantic Disentanglement (MSD) decomposes each modality representation $z_m$ into a shared component $z_m^c$ and a modality-specific component $z_m^s$ via lightweight MLPs. Three constraints enforce functional separation: an InfoNCE contrastive loss aligns shared components across modalities; batch-wise cosine-similarity minimization decorrelates specific components; and unimodal predictors on $z_m^s$ preserve per-modality predictive capability. Cross-modal Complementary Enhancement (CCE) then enriches each modality by feeding its own components together with shared and specific components of the other modalities into an enhancement network $G_m$, trained with a reconstruction term plus a task-consistency loss, with optional stochastic perturbation for feature diversity.

Stage II — balance and trust. Energy-guided Modality Coordination (EMC) is presented as the first EBM-inspired modality coordination mechanism. Each modality receives an energy potential combining representation magnitude $\|z_m\|_2^2$, task loss $\ell_m$, and predictive entropy $u_m$. A pairwise energy-gap objective $\mathcal{L}_{gap}=\sum_{i,j}(E(m_i)-E(m_j))^2$ drives all modalities toward equilibrium, and an explicit energy-descent update $\Delta z_m=-\lambda\,\partial E(m)/\partial z_m$ implements negative-feedback dynamics: low-energy (dominant) modalities are suppressed while high-energy (weak) modalities receive larger corrective gradients. A curvature penalty $z_m^c$0 smooths the energy landscape. The appendix derives that this induces implicit, energy-aware gradient weights proportional to $z_m^c$1, giving EMC a principled EBM interpretation as a single-step Langevin flow restricted to modality-specific potentials—stronger grounding than ad-hoc gradient-balancing rules.

Instance-aware Modality Trust Distillation (IMTD) operates at the sample level. Teacher predictors from MSD produce probabilistic outputs whose variance is converted into confidence scores with a soft normalization factor; normalized weights $z_m^c$2 then scale a temperature-scaled KL divergence between the fused student prediction and each teacher. Noisy or corrupted modalities are thus down-weighted per instance rather than per modality class.

The total objective combines task loss with weighted MSD, CCE, EMC, and IMTD terms ($z_m^c$3, $z_m^c$4). Notably, EMC and IMTD consist only of element-wise algebraic operations, so they add negligible overhead relative to backbone encoders.

Main results

On CMU-MOSI and CMU-MOSEI, EBMC achieves state-of-the-art or competitive results against ten strong baselines including SelfMM, ConKI, ConFEDE, GLoMo, DEVA, and Semi-IIN:

Dataset	Acc-2	F1	Acc-7	Corr
CMU-MOSI	86.26/87.84	86.20/87.79	50.34	0.833
CMU-MOSEI	86.04/88.10	86.23/88.07	57.32	0.824

The largest margins appear on fine-grained Acc-7: +1.89% over Semi-IIN on MOSI and +1.1% on MOSEI, which the authors attribute to better exploitation of subtle non-verbal cues in ambiguous samples. On IEMOCAP emotion recognition in conversation, EBMC reaches an average F1 of 86.35%, exceeding the strongest baseline DMD (85.08%) by 1.27%, with pronounced gains in Happy (92.0) and Angry (90.3) categories.

Robustness results are the strongest empirical claims in the paper. Under six incomplete-modality testing conditions on CMU-MOSEI, EBMC attains the highest average F1 (78.92 vs. 74.50 for EUAR), with particularly large gains in audio-only conditions (71.67 vs. 66.54 for GCNet). Under simulated intra-modality missingness (frame-level dropout rates up to 0.9, averaged over rates), EBMC surpasses LNLN by 1.52% F1 on MOSI and improves Corr by 0.058 on MOSEI, indicating better-calibrated intensity predictions under corruption. Ablations confirm each module contributes: removing EMC causes the largest F1 drop (−2.43% MOSI, −2.87% MOSEI), followed by MSD (−1.63%/−1.76%), IMTD (−1.02%/−0.98%), and CCE (−0.89%/−1.18%). An additional ablation shows every energy term in EMC is necessary, with removal of the loss-coupling term $z_m^c$5 degrading Acc-7 most severely (57.32 → 55.94).

Analysis sections support the mechanism claims: t-SNE visualizations show progressive cluster separation as MSD and CCE are added; contribution analysis shows text dominance above 50% without EMC is reduced with rebalancing toward audio and video alongside consistent Acc-2 gains; case studies illustrate correct handling of neutral utterances where baselines over-rely on textual polarity. Efficiency is favorable: with offline-cached features, EBMC reduces to 6.38M parameters and trains in roughly 5 minutes for 300 epochs on MOSI, versus 112M parameters and ~1 hour online.

Limitations and open questions

Several caveats bear on the reported results. First, the energy potential weights ($z_m^c$6, $z_m^c$7, $z_m^c$8) and the four loss coefficients are set empirically; sensitivity analysis shows performance peaks sharply at moderate values (0.1) and declines under both under- and over-regularization, so transfer to other tasks or architectures may require retuning. Second, the EBM interpretation rests on treating the deterministic energy-descent step as a single-step Langevin flow without the noise term; no convergence guarantees beyond stationary-point characterization are provided. Third, the missing-modality evaluation uses standard benchmark corruptions (frame-level dropout, whole-modality removal); performance under structured real-world missingness patterns remains untested. Fourth, hyperparameter sensitivity was analyzed only on CMU-MOSEI, leaving cross-dataset stability of the optimal configuration an open question. Finally, whether the energy-equilibrium objective interacts beneficially with large pretrained multimodal backbones beyond BERT-based encoders is not examined.

Conclusion

EBMC unifies weak-modality enhancement (MSD, CCE), optimization-level rebalancing via a differentiable energy-based coordination mechanism (EMC), and instance-level reliability-aware distillation (IMTD) within a single framework. It delivers state-of-the-art or competitive accuracy on MOSI, MOSEI, and IEMOCAP, with its clearest advantages appearing under missing-modality and intra-modality corruption, where it substantially outperforms dedicated robustness baselines. The framework's lightweight auxiliary modules and demonstrated cross-task generalization make it a practical candidate for broader multimodal settings, though the empirical tuning of its energy landscape and the absence of formal convergence guarantees remain points requiring further scrutiny.