Hypergraph Multi-Modal Learning for EEG-based Emotion Recognition in Conversation

Abstract: Emotional Recognition in Conversation (ERC) is an important method for diagnosing health conditions such as autism or depression, as well as understanding emotions in individuals who struggle to express their feelings. Current ERC methods primarily rely on complete semantic textual information, including audio and visual data, but face challenges in integrating physiological signals such as electroencephalogram (EEG). This paper proposes a novel Hypergraph Multi-Modal Learning Framework (Hyper-MML), designed to effectively identify emotions in conversation by integrating EEG with audio and video information to capture complex emotional dynamics. Experimental results demonstrate that Hyper-MML significantly outperforms traditional methods in emotion recognition. This is achieved through a Multi-modal Hypergraph Fusion Module (MHFM), which actively models higher-order relationships between multi-modal signals, as validated on the EAV dataset. Our proposed Hyper-MML serves as an effective communication tool for healthcare professionals, enabling better engagement with patients who have difficulty expressing their emotions.