Towards Semi-Supervised Deep Facial Expression Recognition with An Adaptive Confidence Margin (2203.12341v2)

Abstract: Only parts of unlabeled data are selected to train models for most semi-supervised learning methods, whose confidence scores are usually higher than the pre-defined threshold (i.e., the confidence margin). We argue that the recognition performance should be further improved by making full use of all unlabeled data. In this paper, we learn an Adaptive Confidence Margin (Ada-CM) to fully leverage all unlabeled data for semi-supervised deep facial expression recognition. All unlabeled samples are partitioned into two subsets by comparing their confidence scores with the adaptively learned confidence margin at each training epoch: (1) subset I including samples whose confidence scores are no lower than the margin; (2) subset II including samples whose confidence scores are lower than the margin. For samples in subset I, we constrain their predictions to match pseudo labels. Meanwhile, samples in subset II participate in the feature-level contrastive objective to learn effective facial expression features. We extensively evaluate Ada-CM on four challenging datasets, showing that our method achieves state-of-the-art performance, especially surpassing fully-supervised baselines in a semi-supervised manner. Ablation study further proves the effectiveness of our method. The source code is available at https://github.com/hangyu94/Ada-CM.

• The paper introduces Adaptive Confidence Margin (Ada-CM), a novel semi-supervised method that dynamically adjusts confidence thresholds and uses contrastive learning to improve facial expression recognition by better utilizing unlabeled data.
• Ada-CM demonstrates superior performance compared to existing pseudo-labeling methods like FixMatch and enhances robustness across various challenging facial expression datasets.
• This approach significantly reduces the need for large labeled datasets and enables improved model generalization for real-world facial expression recognition applications.

Towards Semi-Supervised Deep Facial Expression Recognition with An Adaptive Confidence Margin

Facial expression recognition (FER) remains a challenging domain despite significant advancements in computer vision models due to the intricacy and often subtlety involved in facial expressions. The paper, "Towards Semi-Supervised Deep Facial Expression Recognition with An Adaptive Confidence Margin," proposes a novel approach called Adaptive Confidence Margin (Ada-CM) for enhancing semi-supervised learning in this domain. The method’s primary objective is to fully exploit unlabeled data through an innovative mechanism that dynamically adjusts the confidence threshold based on learning difficulty, thus transcending the limitations of fixed-threshold methods commonly employed in semi-supervised learning algorithms.

In existing semi-supervised learning frameworks, only the subset of data with high-confidence predictions is used for model training, which is a notable drawback given the varying degrees of prediction certainty among different classes of facial expressions. The Ada-CM approach addresses this shortcoming by introducing an adaptive threshold system that accounts for the nuances in facial expression prediction difficulties across different categories. By learning and applying an Adaptive Confidence Margin, the proposed method allows for leveraging all unlabeled data, dividing it into two subsets based on comparison with the adaptive confidence margin. Data with predictable pseudo-labels and high confidence scores are used to match predictions, while low-confidence samples contribute through a feature-level contrastive learning objective.

The implications of this research are twofold: First, by dynamically adjusting confidence margins according to the learning difficulty of expressions, Ada-CM ensures that learning is not biased towards expressions inherently easier to predict, such as happiness. Second, by employing a contrastive learning approach for low-confidence data, the method enriches feature similarity, thereby increasing the model’s robustness in effectively capturing diverse facial expressions.

Ada-CM demonstrates significant efficacy against existing pseudo-labeling methods such as FixMatch and achieves superior performance over fully-supervised baselines, albeit leveraging a semi-supervised learning approach. The method's robustness is validated across challenging datasets, namely RAF-DB, SFEW, AffectNet, and CK+, where Ada-CM excels particularly in scenarios with limited labeled data availability.

For experts in the field, the introduction of dynamic thresholds—distinguished by their alignment with specific expression categories—offers a promising avenue for establishing more nuanced training regimes. This method’s adaptability not only reduces reliance on extensive labeled datasets but also enables improved generalization, crucial for real-world applications where obtaining large-scale labeled datasets is a logistical challenge.

Further research could explore the implications of Ada-CM in related fields, such as real-time emotion detection or domain adaptation in cross-cultural datasets, where the representation of expressions can manifest differently. Additionally, extending Ada-CM to multimodal recognition, integrating speech and gesture cues alongside visual data, could establish more comprehensive emotion recognition systems, enabling more sophisticated human-computer interactions.

Overall, Ada-CM exemplifies a significant advancement in semi-supervised learning for FER by offering improved data utilization and model adaptability, potentially influencing future developments in AI for emotion recognition and beyond.