Towards Reading Hidden Emotions: A comparative Study of Spontaneous Micro-expression Spotting and Recognition Methods (1511.00423v2)

Abstract: Micro-expressions (MEs) are rapid, involuntary facial expressions which reveal emotions that people do not intend to show. Studying MEs is valuable as recognizing them has many important applications, particularly in forensic science and psychotherapy. However, analyzing spontaneous MEs is very challenging due to their short duration and low intensity. Automatic ME analysis includes two tasks: ME spotting and ME recognition. For ME spotting, previous studies have focused on posed rather than spontaneous videos. For ME recognition, the performance of previous studies is low. To address these challenges, we make the following contributions: (i)We propose the first method for spotting spontaneous MEs in long videos (by exploiting feature difference contrast). This method is training free and works on arbitrary unseen videos. (ii)We present an advanced ME recognition framework, which outperforms previous work by a large margin on two challenging spontaneous ME databases (SMIC and CASMEII). (iii)We propose the first automatic ME analysis system (MESR), which can spot and recognize MEs from spontaneous video data. Finally, we show our method outperforms humans in the ME recognition task by a large margin, and achieves comparable performance to humans at the very challenging task of spotting and then recognizing spontaneous MEs.

• The paper introduces a training-free spotting method using LBP that achieves a 92.98% AUC on the CASMEII database.
• It proposes a recognition framework with motion magnification where gradient-based features surpass LBP, achieving up to 81.69% accuracy on SMIC-VIS.
• The integrated MESR system demonstrates competitive performance against human observers, paving the way for advanced real-world applications in emotion analysis.

Analyzing Spontaneous Micro-expressions: Advancements and Challenges

The paper Towards Reading Hidden Emotions: A Comparative Study of Spontaneous Micro-expression Spotting and Recognition Methods, authored by Xiaobai Li et al., explores the challenges and solutions in the automatic analysis of spontaneous micro-expressions (MEs). The authors effectively address two primary tasks: micro-expression spotting and micro-expression recognition, focusing particularly on the spontaneous variants which are inherently brief and subtle, thus posing significant challenges for detection and recognition.

Micro-expression Spotting

One of the notable contributions of this paper is the development of a training-free method for spotting spontaneous MEs in long videos. The proposed method utilizes feature difference contrast to detect variations in facial expressions. This approach is significant as it moves beyond the more straightforward posed expressions previously studied, which do not emulate the complexity and subtlety of spontaneous expressions accurately.

The paper employs Local Binary Patterns (LBP) and the Histogram of Optical Flow (HOOF) as feature descriptors. While both features contribute to the task, LBP emerged as the more effective descriptor, achieving an Area Under the Curve (AUC) of 92.98% on the CASMEII database. This finding underscores LBP's robustness in identifying rapid and faint micro-expressions amidst other facial dynamics.

Micro-expression Recognition

Building further on spotting, the paper introduces a sophisticated recognition framework that leverages motion magnification and various spatial-temporal features. Recognizing spontaneous MEs still holds immense potential, especially given their applications in fields such as forensic science and psychotherapy.

Through extensive experiments on SMIC and CASMEII databases, the authors evaluate the performance of three core feature descriptors: LBP, Histograms of Oriented Gradients (HOG), and Histograms of Image Gradient Orientation (HIGO). Interestingly, the paper finds that gradient-based features (HOG and HIGO) outperform LBP on color video data, with recognition accuracies reaching 81.69% on the SMIC-VIS dataset after magnification. These findings are crucial as they highlight the importance of selecting appropriate features relative to the data type and capture conditions.

Overall System and Human Comparison

A central novel element of the paper is the introduction of a comprehensive automatic ME analysis system titled MESR that integrates both spotting and recognition. The system achieves a recognition accuracy of 42.42% on SMIC-E-VIS, which, while lower than pure recognition accuracy, still demonstrates competitive performance compared to human subjects. The paper's direct comparison to human capability in recognizing MEs further signifies the advancement and applicability of such an automatic system, noting that the system even outperforms humans in pure recognition tasks.

Implications and Future Directions

The paper's contributions now form a foundation for further exploration in the automated analysis of spontaneous micro-expressions. The advancement from posed to spontaneous micro-expressions considers more realistic and unconstrained scenarios, enhancing the potential for practical application. The paper also suggests that future research efforts could focus on improving spotting accuracy, particularly by reducing false positives due to movements like eye blinks, and enhancing recognition through deeper neural networks and learning paradigms.

In conclusion, the paper makes substantial strides in the field of micro-expression analysis, with the methodologies developed showing promising initial results and offering a pathway for improvements. This area occupies a niche with significant implications for applied emotion recognition and nonverbal communication assessment, further emphasizing the need for robust and adaptable frameworks in real-world environments.