A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances (2203.06935v3)

Abstract: Affective computing plays a key role in human-computer interactions, entertainment, teaching, safe driving, and multimedia integration. Major breakthroughs have been made recently in the areas of affective computing (i.e., emotion recognition and sentiment analysis). Affective computing is realized based on unimodal or multimodal data, primarily consisting of physical information (e.g., textual, audio, and visual data) and physiological signals (e.g., EEG and ECG signals). Physical-based affect recognition caters to more researchers due to multiple public databases. However, it is hard to reveal one's inner emotion hidden purposely from facial expressions, audio tones, body gestures, etc. Physiological signals can generate more precise and reliable emotional results; yet, the difficulty in acquiring physiological signals also hinders their practical application. Thus, the fusion of physical information and physiological signals can provide useful features of emotional states and lead to higher accuracy. Instead of focusing on one specific field of affective analysis, we systematically review recent advances in the affective computing, and taxonomize unimodal affect recognition as well as multimodal affective analysis. Firstly, we introduce two typical emotion models followed by commonly used databases for affective computing. Next, we survey and taxonomize state-of-the-art unimodal affect recognition and multimodal affective analysis in terms of their detailed architectures and performances. Finally, we discuss some important aspects on affective computing and their applications and conclude this review with an indication of the most promising future directions, such as the establishment of baseline dataset, fusion strategies for multimodal affective analysis, and unsupervised learning models.

A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances

The paper provides a comprehensive survey on the domain of affective computing, focusing on emotion models, databases, and recent methodological advancements. Affective computing, situated at the intersection of emotion recognition and sentiment analysis, has gained momentum, particularly due to the availability of extensive public databases and advancements in ML and deep learning (DL) technologies.

Emotion Models and Databases

The authors categorize emotion models into two principal types: discrete and dimensional. Discrete models classify emotions into distinct categories, as exemplified by Ekman's six basic emotions (e.g., anger, happiness) and Plutchik's wheel model. In contrast, dimensional models, such as the Valence-Arousal-Dominance (VAD) model, represent emotions in a continuous, multi-dimensional space, which better captures the complexities of affective states.

The review also identifies five categories of databases critical for the training and evaluation of affective computing systems—textual, speech/audio, visual, physiological, and multimodal. Each category encompasses a range of datasets, including established benchmarks like IMDB, EmoDB, CK+, and DEAP, each providing unique challenges and opportunities for model development and evaluation.

Unimodal Affect Recognition

The paper explores unimodal affect recognition, bifurcating it into ML-based and DL-based approaches. Textual sentiment analysis, speech emotion recognition (SER), visual emotion recognition, and physiological emotion recognition (primarily EEG and ECG) are systematically dissected. For textual analysis, the paper emphasizes the transition from traditional ML approaches, which relied heavily on feature engineering, to DL models capable of learning discriminative features directly from data. In SER and visual emotion recognition, DL models outperform traditional techniques in learning complex representations from raw input data.

Multimodal Affective Analysis

In addressing multimodal affective analysis, the authors categorize approaches based on the type of fusion strategy employed—feature-level, decision-level, model-level, and hybrid fusion. They further differentiate approaches by modality combinations, such as multi-physical (e.g., visual-audio), multi-physiological, and physical-physiological fusions. This comprehensive treatment highlights the superior performance of multimodal systems over unimodal counterparts, attributed to their ability to capture richer affective cues.

Discussion and Implications

The review provides a detailed discussion on the implications of combining various modalities and fusion strategies. It highlights the strengths of DL models in handling complex, high-dimensional affective data, though these models are significantly reliant on large-scale, high-quality datasets. The authors also discuss the limitations of current systems in real-world scenarios, pointing to the necessity for robust databases and advanced fusion strategies that incorporate rule-based or statistical knowledge.

Future Directions

The paper identifies critical areas for future research, including the development of comprehensive databases that encompass diverse scenarios and annotations, the exploration of zero-shot and unsupervised learning for robust affective analysis, and the integration of affective computing systems in practical applications like robotics. These considerations underline the dynamic and rapidly evolving nature of affective computing research.

In summary, the paper serves as a valuable resource for understanding the landscape of affective computing, providing insights into current methodologies, highlighting practical challenges, and proposing directions for future exploration.