MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognition (2401.03429v3)

Abstract: Multimodal emotion recognition plays a crucial role in enhancing user experience in human-computer interaction. Over the past few decades, researchers have proposed a series of algorithms and achieved impressive progress. Although each method shows its superior performance, different methods lack a fair comparison due to inconsistencies in feature extractors, evaluation manners, and experimental settings. These inconsistencies severely hinder the development of this field. Therefore, we build MERBench, a unified evaluation benchmark for multimodal emotion recognition. We aim to reveal the contribution of some important techniques employed in previous works, such as feature selection, multimodal fusion, robustness analysis, fine-tuning, pre-training, etc. We hope this benchmark can provide clear and comprehensive guidance for follow-up researchers. Based on the evaluation results of MERBench, we further point out some promising research directions. Additionally, we introduce a new emotion dataset MER2023, focusing on the Chinese language environment. This dataset can serve as a benchmark dataset for research on multi-label learning, noise robustness, and semi-supervised learning. We encourage the follow-up researchers to evaluate their algorithms under the same experimental setup as MERBench for fair comparisons. Our code is available at: https://github.com/zeroQiaoba/MERTools.

• The paper introduces MERBench, a unified framework standardizing evaluations for multimodal emotion recognition systems.
• It presents the MER2023 dataset tailored for Chinese language environments, supporting multi-label, noise, and semi-supervised learning research.
• Experimental analysis reveals the effects of fusion strategies, noise robustness, and cross-corpus variations, guiding future MER research.

Overview of "MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognition"

The paper "MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognition" addresses the challenges and inconsistencies in evaluating multimodal emotion recognition (MER) systems. It introduces MERBench, a comprehensive evaluation framework that standardizes the assessment of various multimodal the emotion recognition methodologies. By establishing a solid benchmark, the paper intends to provide transparent and fair comparisons across different approaches, which is crucial for advancing research in this field.

Key Contributions

1. Introduction of MERBench: The authors introduce MERBench, a unified evaluation tool that facilitates the comparison of multiple emotion recognition methods. This benchmark provides a consistent framework that accounts for the variations in feature extractors, evaluation manners, and experimental setups traditionally seen across different studies.
2. MER2023 Dataset: The paper presents MER2023, a novel dataset that emphasizes the Chinese language environment. This dataset is designed to support research in multi-label learning, noise robustness, and semi-supervised learning. It is broken into several subsets to evaluate various aspects of MER systems, such as punctuation robustness and noise robustness.
3. Comprehensive Analysis: By utilizing MERBench and MER2023, the authors investigate several pivotal MER components—including feature selection, multimodal fusion strategies, and cross-corpus performance under consistent experimental setups. This analysis identifies important insights into the impact of distinct features and fusion techniques on emotion recognition.
4. Guidelines for Future Research: The paper outlines several promising directions in multimodal emotion recognition. These include focusing on weakly and self-supervised models for visual feature extraction and emphasizing multilingual and expressive acoustic models for better cross-language performance.

Experimental Insights

• Unimodal vs. Multimodal Approaches: The evaluation demonstrates that different emotion recognition datasets have varying modality preferences, indicating that while multimodal data is beneficial, some datasets are more aligned with specific modalities over others, which affects the choice of an appropriate model for each dataset.
• Noise Robustness: The empirical tests show a performance drop for models subjected to increasing levels of noise, highlighting the challenges researchers face in ensuring robust MER systems. Proper data augmentation was found to mitigate noise effects, thus suggesting its relevance in model training.
• Cross-Corpus Performance: It is challenging to achieve high performance across different datasets due to the unique way each dataset expresses emotions. This underlines the necessity for datasets to integrate more uniform labeling techniques and explanatory data to enhance model generalizability.
• Fine-tuning vs. Pre-training: While fine-tuning pre-trained models can bring some improvement, it does not typically offer substantial benefits in comparison to pre-training, especially in resource-constrained scenarios like emotion recognition tasks. The use of pre-pretrained models emerges as a more pragmatic approach, considering computational efficiency.

Future Directions

The paper outlines several future avenues for research in multimodal emotion recognition. Enhancing weakly or self-supervised learning models, improving multilingual robust models, and incorporating comprehensive multimodal fusion strategies are potential research areas. Additionally, the need for cross-domain datasets to handle diverse expressions of emotions effectively remains significant.

The MERBench framework is a significant stepping stone for emotive AI, offering clarity and a consolidated platform for researchers to build consistent and comparative methodologies. Its focus on standardization in evaluation lays a strong foundation for subsequent research, guiding the development of robust and contextually aware emotion recognition systems.