M3ER: Multiplicative Multimodal Emotion Recognition Using Facial, Textual, and Speech Cues (1911.05659v2)

Abstract: We present M3ER, a learning-based method for emotion recognition from multiple input modalities. Our approach combines cues from multiple co-occurring modalities (such as face, text, and speech) and also is more robust than other methods to sensor noise in any of the individual modalities. M3ER models a novel, data-driven multiplicative fusion method to combine the modalities, which learn to emphasize the more reliable cues and suppress others on a per-sample basis. By introducing a check step which uses Canonical Correlational Analysis to differentiate between ineffective and effective modalities, M3ER is robust to sensor noise. M3ER also generates proxy features in place of the ineffectual modalities. We demonstrate the efficiency of our network through experimentation on two benchmark datasets, IEMOCAP and CMU-MOSEI. We report a mean accuracy of 82.7% on IEMOCAP and 89.0% on CMU-MOSEI, which, collectively, is an improvement of about 5% over prior work.

• The paper introduces a novel multiplicative fusion mechanism that dynamically weights facial, textual, and speech cues to overcome inconsistent modality reliability.
• It employs Canonical Correlational Analysis to filter out noisy inputs, achieving a roughly 5% accuracy improvement on IEMOCAP and CMU-MOSEI benchmark datasets.
• The methodology promises enhanced robustness in emotion recognition, with significant implications for applications in human-computer interaction, robotics, and psychological assessments.

Multiplicative Multimodal Emotion Recognition with M3ER

The paper "M3ER: Multiplicative Multimodal Emotion Recognition using Facial, Textual, and Speech Cues" presents an advanced approach to emotion recognition by leveraging multiple modalities: facial expressions, speech, and textual input. This method, termed M3ER, enhances the robustness and accuracy of emotion recognition models by introducing a data-driven multiplicative fusion mechanism coupled with a resilience to sensor noise, making it well-fitted for real-world applications.

Key Contributions

M3ER employs a novel fusion technique that distinguishes itself from traditional additive methods by focusing on multiplicative fusion. The key innovation is in how M3ER dynamically assesses the reliability of each modality per sample, thereby adjusting its reliance on each accordingly. This multiplicative approach overcomes the limitations of prior additive methods, which often assume uniform reliability across modalities. The paper also integrates a preprocessing step utilizing Canonical Correlational Analysis (CCA) to evaluate the effectiveness of input modalities, allowing M3ER to disregard or adapt to corrupted inputs by generating proxy features.

The paper presents strong empirical evidence of M3ER's efficacy through evaluations on two benchmark datasets, IEMOCAP and CMU-MOSEI, where it achieved mean accuracies of 82.7% and 89.0%, respectively. These results denote a significant improvement of approximately 5% over previous state-of-the-art methods, underscoring the effectiveness of the proposed multiplicative approach.

Methodological Insights

M3ER's architecture incorporates a modality check phase that proactively identifies and suppresses ineffectual modalities. This is particularly relevant in "in-the-wild" datasets, which are often plagued by sensor noise and occlusions. The incorporation of CCA facilitates the differentiation between effective and ineffectual modalities. Following this, M3ER's architecture is augmented by a feature transformation mechanism, which allows the generation of proxy features for corrupted modalities based on reliable modality data.

The fusion strategy of M3ER also merits attention. Instead of traditional early or late fusion, M3ER employs a multiplicative combination approach. This strategy is realized through a novel loss function that emphasizes the reliability of input features, giving higher weights to more reliable modalities, thus underscoring its adaptability.

Implications and Future Directions

The results demonstrated by M3ER have significant ramifications for multimodal emotion recognition systems, particularly in enhancing robustness against data anisotropy and noise. The proposed methodologies have potential applications in diverse fields, including human-computer interaction, robotics, and psychological assessments, where emotion recognition plays a pivotal role.

While M3ER exhibits impressive performance, the paper acknowledges limitations regarding the confusion between certain emotion classes and the binary classification scheme employed. The authors suggest that future research could benefit from exploring probability distributions over emotion classes to capture the inherent subjectivity in human emotional perception.

Additionally, the authors propose future investigations into more complex fusion techniques that could further boost predictive accuracies. Extending the model to encompass an expanded array of modalities, including contextual and physiological cues, could also enhance its applicability and precision in naturalistic settings.

Conclusion

M3ER represents a sophisticated step forward in the field of emotion recognition, offering practical solutions and methodological advancements that align well with the demands for real-time, robust emotion recognition systems. Through its integration of CCA and multiplicative fusion, M3ER distinctly balances the reliability of multiple modalities, proving to be a significant contribution to the field of multimodal machine learning. The proposed direction for subsequent research and potential improvements highlight the evolving nature of emotion recognition technologies and set a promising trajectory for further developments.