J$\hat{\text{A}}$A-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attention (2003.08834v3)

Abstract: Facial action unit (AU) detection and face alignment are two highly correlated tasks, since facial landmarks can provide precise AU locations to facilitate the extraction of meaningful local features for AU detection. However, most existing AU detection works handle the two tasks independently by treating face alignment as a preprocessing, and often use landmarks to predefine a fixed region or attention for each AU. In this paper, we propose a novel end-to-end deep learning framework for joint AU detection and face alignment, which has not been explored before. In particular, multi-scale shared feature is learned firstly, and high-level feature of face alignment is fed into AU detection. Moreover, to extract precise local features, we propose an adaptive attention learning module to refine the attention map of each AU adaptively. Finally, the assembled local features are integrated with face alignment feature and global feature for AU detection. Extensive experiments demonstrate that our framework (i) significantly outperforms the state-of-the-art AU detection methods on the challenging BP4D, DISFA, GFT and BP4D+ benchmarks, (ii) can adaptively capture the irregular region of each AU, (iii) achieves competitive performance for face alignment, and (iv) also works well under partial occlusions and non-frontal poses. The code for our method is available at https://github.com/ZhiwenShao/PyTorch-JAANet.

• The paper introduces a novel framework that jointly optimizes AU detection and face alignment via an adaptive attention module to enhance accuracy.
• It employs hierarchical multi-scale convolutional layers that refine attention maps using dynamically predicted facial landmarks.
• Experiments on multiple datasets demonstrate the method's robustness against occlusions, varied head poses, and data imbalance in facial analysis.

An Analysis of Joint Facial Action Unit Detection and Face Alignment via Adaptive Attention

The paper "J$\hat{\text{A}$A-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attention" introduces a novel framework for integrating facial action unit (AU) detection and face alignment in an end-to-end manner using deep learning. Historically, these two tasks have been primarily treated as separate, with facial landmarks often used merely as a preprocessing step for AU detection to delineate regions of interest (ROIs). This paper proposes a unified approach leveraging the intrinsic correlations between AU detection and face alignment, suggesting that improvements in one could inherently benefit the other.

Proposed Method: J$\hat{\text{A}$A-Net

The essence of the J$\hat{\text{A}$A-Net approach is its adaptive attention learning module, which refines the attention map of each AU by utilizing both global and local facial features learned through shared multi-scale convolutional layers. This methodology contrasts with previous fixed-attention or Gaussian-based approaches by allowing the network to dynamically adapt attention maps based on the predicted facial landmarks, thus capturing irregular AU regions more effectively.

Key Components:

1. Hierarchical and Multi-Scale Region Learning: This foundational module extracts features over a range of scales using a custom convolutional filter design. This hierarchy supports AUs of varying sizes but requires fewer parameters than traditional convolutional layers.
2. Face Alignment Integration: Unlike other models that treat face alignment as only a means to preprocess or normalize the input, in J$\hat{\text{A}$A-Net, the face alignment task directly influences AU detection. This is achieved by feeding face alignment features into the AU detection pathway and using predicted landmarks to initialize AU attention maps.
3. Adaptive Attention Learning: This is the core innovation of the paper. Each AU has its attention map, which is refined via a branch-wise network, supervised by a locally focused AU detection loss. This enables the model to adaptively adjust attention across the spatial domain influenced by each AU's characteristics.

Experimental Results

The experimental evaluation across several datasets (BP4D, DISFA, GFT, and BP4D+) demonstrated the efficacy of the proposed framework. J$\hat{\text{A}$A-Net consistently outperformed state-of-the-art methods in AU detection benchmarks. On notably challenging datasets like DISFA, characterized by significant data imbalance, the model maintained robust performance levels, indicating its strong generalization capabilities. Additionally, the ability to handle partial occlusion and variations in head poses was highlighted, demonstrating robustness across various real-world conditions.

Implications and Future Directions

The success of J$\hat{\text{A}$A-Net underscores the advantages of joint learning systems in facial analysis tasks. By exploiting the synergies between AU detection and face alignment, the framework sets a precedent for multi-task learning paradigms in computer vision. The insights gained could be extended to other problems where multiple related tasks are traditionally treated in isolation.

Future research might explore incorporating temporal information to extend this approach to video-based AU detection, potentially utilizing recurrent neural networks (RNNs) for dynamic attention refinement. Additionally, expanding the framework's capacity to handle even more severe occlusions or larger variations in lighting and facial expressions could make the methodology more robust for diverse applications, including real-time emotion recognition and human-computer interaction.

Conclusion

The J$\hat{\text{A}$A-Net framework presents a significant methodological advancement in the joint learning of AU detection and face alignment. With its adaptive attention module and integrated learning approach, it has set a new benchmark for accuracy and robustness in facial expression analysis, paving the way for future innovations in related fields.