SURE: Synergistic Uncertainty-aware Reasoning for Multimodal Emotion Recognition in Conversations

Abstract: Multimodal emotion recognition in conversations (MERC) requires integrating multimodal signals while being robust to noise and modeling contextual reasoning. Existing approaches often emphasize fusion but overlook uncertainty in noisy features and fine-grained reasoning. We propose SURE (Synergistic Uncertainty-aware REasoning) for MERC, a framework that improves robustness and contextual modeling. SURE consists of three components: an Uncertainty-Aware Mixture-of-Experts module to handle modality-specific noise, an Iterative Reasoning module for multi-turn reasoning over context, and a Transformer Gate module to capture intra- and inter-modal interactions. Experiments on benchmark MERC datasets show that SURE consistently outperforms state-of-the-art methods, demonstrating its effectiveness in robust multimodal reasoning. These results highlight the importance of uncertainty modeling and iterative reasoning in advancing emotion recognition in conversational settings.

• The paper presents SURE, a novel framework that integrates uncertainty-aware modules for robust multimodal emotion recognition in conversations.
• It employs a mixture-of-experts module, iterative context reasoning, and transformer gates to effectively mitigate noise and enhance contextual integration.
• Experimental results show significant improvements, with gains of +4.76% accuracy on IEMOCAP and +0.76% F1 on MELD, validating its design and methodology.

Synergistic Uncertainty-Aware Reasoning for Multimodal Emotion Recognition in Conversations

Introduction

Emotion recognition in multi-turn conversations, leveraging multimodal cues (text, acoustic, and visual), presents unique challenges stemming from the dynamic and noisy nature of real-world data and the necessity of coherent context modeling. The paper "SURE: Synergistic Uncertainty-aware Reasoning for Multimodal Emotion Recognition in Conversations" (2604.01916) introduces SURE, a framework that distinctly addresses both modality-dependent noise and the contextual reasoning necessary for robust emotion identification in conversational settings.

SURE Framework Overview

SURE integrates three primary components: an Uncertainty-Aware Mixture-of-Experts (MoE) module for modality-specific noise mitigation, an Iterative Reasoning module facilitating multi-pass contextual integration, and a Transformer Gate to model intra- and inter-modal dependencies. The end-to-end system begins with the extraction of utterance-centered unimodal features (RoBERTa-based for text, openSMILE for acoustics, and DenseNet for visual), which are then processed through the aforementioned architectural sequence.

Figure 1: The SURE framework, comprising multimodal input processing, Uncertainty-Aware MoE, Iterative Reasoning, and Transformer Gate integration modules.

A key distinction in SURE’s design is its formal approach to modeling and leveraging uncertainty estimates for each modality and data instance, which guides expert selection and fusion at each step, thereby mitigating the impact of outlier and noisy data.

Methodology

Uncertainty-Aware Mixture-of-Experts (MoE) Module

After unimodal feature extraction, each modality-specific feature is projected as the parameterization of a diagonal Gaussian $\mathcal{N}(\mu^m, \sigma^{m2}I)$ using independent fully connected projections for mean and variance. The stochastic latent representation enables the quantification of uncertainty, with higher predicted variances indicating noisier or unreliable features.

A gating network uses this uncertainty information during its top-$k$ expert selection, ensuring that representations with lower estimated uncertainty are prioritized, and unreliable experts are suppressed. This dynamic routing is critical for robust downstream fusion and context reasoning, setting SURE apart from classical MoE paradigms in multimodal systems.

Iterative Reasoning Module

Motivated by cognitive emotion theory, the module adopts a memory-augmented, multi-iteration retrieval mechanism. Utterance-level MoE outputs are encoded using an LSTM-based structure to produce contextual memories; queries are iteratively refined via attention-based retrieval and sequential LSTM updates, enabling the model to extract and aggregate context-relevant emotional cues across multiple dialogue turns. This multi-pass inference deepens the representation of emotional state transitions and dependencies, surpassing one-shot context fusion techniques.

Transformer Gate

The Transformer Gate module is designed to jointly encode intra-modal dependencies (via self-attention) and inter-modal interactions (via cross-attention). For each modality, Transformer blocks calculate attention both within and across modalities (e.g., text attending to acoustic and visual), and the resulting representations are adaptively fused with a gating mechanism. This hierarchical attention structure ensures fine-grained integration of multimodal signals necessary for accurate emotion discrimination.

The final representations are concatenated and projected onto the emotion space for utterance-level classification.

Experimental Evaluation

SURE’s evaluation is conducted on two MERC benchmarks: IEMOCAP and MELD. The comparison set includes both contemporary graph-based models (e.g., MMGCN, Joyful, MMPCGN) and advanced fusion-based architectures (e.g., DialogueTRM, DF-ERC, SDT, MM-NodeFormer). Metrics used are overall accuracy and weighted F1-score, following established protocols.

SURE consistently achieves superior performance over all evaluated baselines on both datasets. On IEMOCAP, SURE delivers a +4.76% Acc and +3.77% F1 improvement over Joyful, and on MELD, it outperforms the strongest baseline by +0.76% F1. Gains are especially pronounced in complex, multi-party conversational data, attributed to SURE's uncertainty-driven expert selection and iterative context integration.

Key findings highlight that:

• Removal of the MoE module (i.e., ablation of uncertainty modeling) results in a decrease of 0.57% (IEMOCAP, F1) and 0.34% (MELD, F1).
• Omission of the Iterative Reasoning module causes an additional 0.38–0.44% F1 drop.
• Missing modality experiments confirm that text dominates in emotion disambiguation, while acoustic and visual signals are more noise-prone; SURE’s full integration yields the highest scores, demonstrating robust multimodal generalization.

Theoretical and Practical Implications

SURE’s architecture suggests several important implications for multimodal and dialogue-based emotion recognition:

• Uncertainty estimation at the modeling level, exploited within the expert routing mechanism, is critical for handling data heterogeneity and real-world noise, with direct downstream impact on system reliability.
• Multi-pass, memory-based context reasoning outperforms static or shallow aggregation, indicating that deeper context modeling is essential for capturing emotional dynamics in conversations.
• Intra/inter-modal attention fusion enables the system to adaptively prioritize and integrate signal sources, crucial in multimodal settings with varying information quality per modality.

These architectural principles may further benefit the design of system architectures for other multimodal conversational tasks, such as dialogue state tracking, sentiment analysis, and more general multimodal sequential tasks.

Future Directions

Further research could extend SURE in several directions:

• Exploration of more adaptive, hierarchical expert-sharing policies to accommodate dynamically changing conversational contexts and speaker attributes.
• Transfer to broader multimodal dialogue understanding tasks, including those involving LLMs, multimodal retrieval, or complex event attribution.
• Deeper interpretability investigations, particularly for explainable emotion reasoning across modalities and dialogue structure.

Conclusion

SURE offers a principled framework for robust, uncertainty-aware multimodal fusion and fine-grained context reasoning in conversational emotion recognition. By orchestrating expert selection through explicit uncertainty modeling and leveraging iterative, memory-based reasoning alongside hierarchical attention fusion, SURE achieves consistent performance gains and establishes a clear direction towards robust and context-sensitive multimodal understanding (2604.01916).