Geometric Feature-Based Facial Expression Recognition in Image Sequences Using Multi-Class AdaBoost and Support Vector Machines (1604.03225v1)

Abstract: Facial expressions are widely used in the behavioral interpretation of emotions, cognitive science, and social interactions. In this paper, we present a novel method for fully automatic facial expression recognition in facial image sequences. As the facial expression evolves over time facial landmarks are automatically tracked in consecutive video frames, using displacements based on elastic bunch graph matching displacement estimation. Feature vectors from individual landmarks, as well as pairs of landmarks tracking results are extracted, and normalized, with respect to the first frame in the sequence. The prototypical expression sequence for each class of facial expression is formed, by taking the median of the landmark tracking results from the training facial expression sequences. Multi-class AdaBoost with dynamic time warping similarity distance between the feature vector of input facial expression and prototypical facial expression, is used as a weak classifier to select the subset of discriminative feature vectors. Finally, two methods for facial expression recognition are presented, either by using multi-class AdaBoost with dynamic time warping, or by using support vector machine on the boosted feature vectors. The results on the Cohn-Kanade (CK+) facial expression database show a recognition accuracy of 95.17% and 97.35% using multi-class AdaBoost and support vector machines, respectively.

• The paper introduces a novel method that leverages elastic bunch graph matching for precise tracking of facial landmarks in dynamic image sequences.
• The paper utilizes dynamic time warping and Multi-Class AdaBoost for effective feature selection, with SVM achieving up to 97.35% recognition accuracy.
• The paper’s approach offers promising applications in emotion-sensitive technologies by addressing temporal variations in facial expressions.

Geometric Feature-Based Facial Expression Recognition in Image Sequences Using Multi-Class AdaBoost and Support Vector Machines

The paper by Ghimire and Lee explores automatic facial expression recognition through a geometric feature-based approach leveraging both Multi-Class AdaBoost and Support Vector Machines (SVM). The authors offer a method centered on dynamic analysis of facial image sequences, a domain of considerable interest given the expressive power of human facial movements, often comprising over half of the communicative effectiveness in social interactions.

Methodological Overview

The methodology focuses on tracking facial expressions over time via landmark-based feature extraction without utilizing facial texture information. A key aspect is the application of elastic bunch graph matching to initialize and track facial landmarks across frames, acquiring a sequence of geometric features as the expression transitions from a neutral to an intense state.

1. Landmark Initialization and Tracking: Facial landmarks are tracked using Gabor wavelet-based jets captured in a bunch graph structure. The initial coordinates, extracted through elastic bunch graph matching, are continuously updated across image frames, potentiated to maintain accuracy in aligning with the neutral face startup.
2. Feature Extraction and Normalization: Two types of features, single landmark changes and paired landmark movements, are employed to capture the geometric dynamics of facial transformations. These features are normalized using dynamic time warping (DTW), allowing for the temporal alignment critical in capturing nuanced expression changes.
3. Dimensionality Reduction and Classification: AdaBoost is innovatively applied for feature selection, identifying discriminative elements within the vast feature pool. The weak classifiers in AdaBoost rely on DTW similarity measures to facilitate this feature selection process. Subsequently, SVM leverages these selected features to serve as a robust facial expression classifier.

Experimental Results and Performance

The authors validate their methodology using the Extended Cohn-Kanade (CK+) facial expression dataset, a benchmark in the field for evaluating expression recognition systems. Recognition accuracies of 95.17% via Multi-Class AdaBoost and 97.35% via SVM on AdaBoost-selected features underscore the potential efficacy of their approach. The performance constraints are comparable to, and sometimes exceed, those documented in existing literature, such as the manually intensive 99.7% recognition rate described by Kotsia and Pitas, with the proposed method shining due to its fully automated process and comparable robustness.

Implications and Future Directions

Ghimire and Lee’s research delineates a comprehensive geometric-based facial expression recognition methodology that eschews reliance on appearance-based feature data, focusing instead on the geometric shifts in landmark positions. This approach not only reinforces the feasibility of achieving high accuracy in automated systems but also aids in understanding the crucial time-dependent dynamics pertinent to different emotional expressions.

The integration of DTW in weak classifiers suggests broader applications in fields requiring dynamic sequence alignment and may foster advancements within other temporal analysis domains. Future research initiatives could extend these methods to deal with diverse datasets beyond prototypical expressions or explore augmenting the geometric framework with more holistic facial motion analysis techniques.

Such investigations are likely to push towards systems that cater to real-world variabilities, including those posed by occlusion, head-pose variations, and non-prototypical expression manifestations, thereby enhancing both theoretical underpinnings and practical deployments in emotion-sensitive technologies.