Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms (2201.12728v2)

Abstract: Unlike the conventional facial expressions, micro-expressions are involuntary and transient facial expressions capable of revealing the genuine emotions that people attempt to hide. Therefore, they can provide important information in a broad range of applications such as lie detection, criminal detection, etc. Since micro-expressions are transient and of low intensity, however, their detection and recognition is difficult and relies heavily on expert experiences. Due to its intrinsic particularity and complexity, video-based micro-expression analysis is attractive but challenging, and has recently become an active area of research. Although there have been numerous developments in this area, thus far there has been no comprehensive survey that provides researchers with a systematic overview of these developments with a unified evaluation. Accordingly, in this survey paper, we first highlight the key differences between macro- and micro-expressions, then use these differences to guide our research survey of video-based micro-expression analysis in a cascaded structure, encompassing the neuropsychological basis, datasets, features, spotting algorithms, recognition algorithms, applications and evaluation of state-of-the-art approaches. For each aspect, the basic techniques, advanced developments and major challenges are addressed and discussed. Furthermore, after considering the limitations of existing micro-expression datasets, we present and release a new dataset - called micro-and-macro expression warehouse (MMEW) - containing more video samples and more labeled emotion types. We then perform a unified comparison of representative methods on CAS(ME)2 for spotting, and on MMEW and SAMM for recognition, respectively. Finally, some potential future research directions are explored and outlined.

• The paper systematically surveys video-based facial micro-expression analysis, addressing data scarcity and the challenges of detecting involuntary expressions.
• It introduces the micro-and-macro expression warehouse (MMEW) dataset to bridge gaps between traditional and micro-expression recognition methods.
• The study leverages neuropsychological insights and advanced techniques like optical flow and deep learning to enhance practical emotion detection.

Video-based Facial Micro-Expression Analysis: An Expert Review

Facial micro-expressions, brief and involuntary facial movements, unveil genuine emotions that individuals often attempt to hide. These micro-expressions, lasting between 0.065 and 0.5 seconds, are challenging to capture and interpret due to their subtlety and transient nature. The paper provides a comprehensive survey of the video-based analysis of these micro-expressions, pushing the boundaries of traditional facial analysis methods and focusing attention on this emerging research domain.

Key Contributions and Insights

The analysis of spontaneous micro-expressions is crucial in fields like lie detection and criminal investigations due to the information they reveal about concealed emotions. Despite its potential, video-based micro-expression analysis remains a daunting task, driven by the nuanced nature of the expressions, lack of robust datasets, and limitations in existing feature extraction and recognition techniques. This paper addresses these challenges through a systematic survey of datasets, features, and algorithms, along with the release of a new dataset, emphasizing the integrated evaluation of methodologies.

A central element of the work is the distinction between macro- and micro-expressions. Unlike macro-expressions, which are voluntary and can last several seconds, micro-expressions are involuntary, fleeting, and localized, making them difficult to discern through manual observation. The research paper delineates a systematic overview of methodologies for detecting and recognizing micro-expressions via video analysis, involving neuropsychological underpinnings and available datasets and directing further research and development in this area.

Analytical Framework and Methodologies

The survey incorporates a detailed cascade structure for micro-expression determination, including the evaluation of neuropsychological bases, the intricacies of existing datasets, advanced development in features, and algorithm refinement. Highlighted research challenges include the insufficiency of micro-expression datasets, the necessity for effective spotting and recognition algorithms, and the integration of neuropsychological elements for improved detection.

To confront the challenges identified, the paper introduces the micro-and-macro expression warehouse (MMEW), offering an enriched collection of micro-expressions with more video samples and labeled emotions compared to existing datasets. The MMEW dataset facilitates the exploration of interconnectedness between macro- and micro-expressions, leveraging their commonalities to enhance recognition systems.

The survey showcases exhaustive comparative analyses on the CAS(ME)2, SAMM, and MMEW datasets, employing diverse methodologies such as the main directional mean optical flow (MDMO) for spotting and various traditional and deep learning algorithms for recognition. The clear exposition of these techniques demonstrates the potential for enriching micro-expression datasets through pretraining with related macro-expressions data, thus opening new avenues for research focusing on cross-modal emotion detection.

Theoretical and Practical Implications

The theoretical implications of this research are significant within AI and emotion detection fields. It challenges researchers to innovate in the design of algorithms that can autonomously capture and interpret emotional subtleties from short-lived, involuntary expressions. Furthermore, by releasing a comprehensive dataset, this work invites the academic community to develop advanced models that refine the detection and interpretation of micro-expressions in real-world applications.

On a practical level, the systematic evaluation and dataset provision enable the design of more reliable and robust systems for micro-expression analysis. Such research finds relevance in high-stakes environments where the detection of concealed emotions can have critical implications, such as in security and psychological assessments.

Future Directions

Future directions in this field are vast and promising, encompassing areas such as privacy protection, generalization in cross-dataset evaluations, and improvements in preprocessing methods. Bridging the gap between macro and micro-expressions and utilizing explainable AI models are also essential for advancing the understanding and practicality of these analyses. Harnessing advancements in deep learning for more authentic generation of synthetic datasets could further upscale analytical capabilities.

In conclusion, this paper serves as a comprehensive guide and benchmark in the domain of video-based facial micro-expression analysis. By addressing current limitations and projecting forward-thinking solutions, it paves the way for innovations that can eventually make micro-expression analysis a staple in diverse practical applications.