TransFER: Learning Relation-aware Facial Expression Representations with Transformers (2108.11116v1)

Abstract: Facial expression recognition (FER) has received increasing interest in computer vision. We propose the TransFER model which can learn rich relation-aware local representations. It mainly consists of three components: Multi-Attention Dropping (MAD), ViT-FER, and Multi-head Self-Attention Dropping (MSAD). First, local patches play an important role in distinguishing various expressions, however, few existing works can locate discriminative and diverse local patches. This can cause serious problems when some patches are invisible due to pose variations or viewpoint changes. To address this issue, the MAD is proposed to randomly drop an attention map. Consequently, models are pushed to explore diverse local patches adaptively. Second, to build rich relations between different local patches, the Vision Transformers (ViT) are used in FER, called ViT-FER. Since the global scope is used to reinforce each local patch, a better representation is obtained to boost the FER performance. Thirdly, the multi-head self-attention allows ViT to jointly attend to features from different information subspaces at different positions. Given no explicit guidance, however, multiple self-attentions may extract similar relations. To address this, the MSAD is proposed to randomly drop one self-attention module. As a result, models are forced to learn rich relations among diverse local patches. Our proposed TransFER model outperforms the state-of-the-art methods on several FER benchmarks, showing its effectiveness and usefulness.

• The paper introduces TransFER, a novel facial expression recognition method using Vision Transformers to learn relation-aware representations from local facial patches.
• TransFER employs Multi-Attention Dropping, ViT-FER, and Multi-head Self-Attention Dropping components to enhance robustness and capture diverse relational insights among patches.
• TransFER achieved state-of-the-art accuracy on benchmark FER datasets, demonstrating the effectiveness of its relation-aware approach for practical applications like human-computer interaction.

Analysis of "TransFER: Learning Relation-aware Facial Expression Representations with Transformers"

This paper introduces TransFER, a novel approach to Facial Expression Recognition (FER) that leverages the capabilities of Vision Transformers (ViT). The research presents a comprehensive method focused on enhancing FER by developing relation-aware facial expression representations, particularly through the identification and utilization of diverse local facial patches. The authors address the perennial challenges in FER, such as distinguishing expressions with large inter-class similarities and coping with variations in expressions within the same class due to demographic factors.

Facial Expression Recognition has traditionally been a challenging domain in computer vision due to the subtle differences between expression categories and substantial intra-class variation. Previous approaches have largely focused on either global facial representations or localistic techniques that might overlook significant patch diversity. TransFER departs from these models by integrating transformers and a novel series of components to handle the representation of expressions more robustly and contextually.

The architecture of the TransFER model is underpinned by three primary components: Multi-Attention Dropping (MAD), ViT-FER, and Multi-head Self-Attention Dropping (MSAD).

1. Multi-Attention Dropping (MAD): This component addresses the inconsistency of local patch visibility caused by variations in pose and occlusion. By randomly dropping certain attention sources during training, it forces the model to explore additional local patches, thus ensuring robustness and adaptability in FER tasks.
2. ViT-FER: Leveraging Vision Transformers, this component enriches the relationships between different local patches by utilizing a global attention view that reinforces local patch characteristics, working towards representing the holistic expression more competently.
3. Multi-head Self-Attention Dropping (MSAD): Employing a method of selecting and dropping one self-attention module, this component mitigates the risk of redundant similarity in multi-head self-attention outputs. The intervention compels the model to develop a broader spectrum of relational insights among patches.

Experimental evaluations reflect the efficacy of the TransFER model. Tested against prevalent FER datasets like RAF-DB, FERPlus, and AffectNet, TransFER outperformed existing approaches, achieving a record accuracy of 90.91% on RAF-DB. This performance exemplifies the benefit of the relation-aware hybrid approach proposed by the authors, setting a new benchmark in FER tasks.

The paper’s approach effectively amalgamates the strong points of the ViT architecture with the specificity demand in FER—ensuring that attention mechanisms target a dynamic spectrum of expressions by considering the entirety of possible complementary patches. On a practical level, the adaptability and robustness introduced by TransFER can significantly enhance human-computer interaction systems, enabling more nuanced and contextually adaptative emotional intelligence in AI frameworks.

In terms of theoretical implications, this work underscores the opportunity and benefits of combining attention mechanisms with traditional convolutional networks for visual tasks that require granular attention to object parts. The willingness to learn diverse local representations and implicitly encourage wider relational mapping through novel MA(SD) mechanisms can positively influence other computer vision applications requiring detailed object analysis.

Future research avenues could explore the integration of TransFER in multi-modal emotional recognition environments, assessing its adaptability to incorporate auditory or textual emotional cues. Additionally, understanding the potential of transformers in unsupervised learning contexts within the field of FER may offer new directions to enhance dataset efficiency and model generalizability. The continued development and optimization of such hybrid models could further amplify both the scope and depth of transformer-based applications in the burgeoning field of affective computing.