Hypergraph-Guided Disentangled Spectrum Transformer Networks for Near-Infrared Facial Expression Recognition (2312.05907v1)
Abstract: With the strong robusticity on illumination variations, near-infrared (NIR) can be an effective and essential complement to visible (VIS) facial expression recognition in low lighting or complete darkness conditions. However, facial expression recognition (FER) from NIR images presents more challenging problem than traditional FER due to the limitations imposed by the data scale and the difficulty of extracting discriminative features from incomplete visible lighting contents. In this paper, we give the first attempt to deep NIR facial expression recognition and proposed a novel method called near-infrared facial expression transformer (NFER-Former). Specifically, to make full use of the abundant label information in the field of VIS, we introduce a Self-Attention Orthogonal Decomposition mechanism that disentangles the expression information and spectrum information from the input image, so that the expression features can be extracted without the interference of spectrum variation. We also propose a Hypergraph-Guided Feature Embedding method that models some key facial behaviors and learns the structure of the complex correlations between them, thereby alleviating the interference of inter-class similarity. Additionally, we have constructed a large NIR-VIS Facial Expression dataset that includes 360 subjects to better validate the efficiency of NFER-Former. Extensive experiments and ablation studies show that NFER-Former significantly improves the performance of NIR FER and achieves state-of-the-art results on the only two available NIR FER datasets, Oulu-CASIA and Large-HFE.
- Island loss for learning discriminative features in facial expression recognition. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), 302–309. IEEE.
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations.
- Compound facial expressions of emotion. Proceedings of the national academy of sciences, 111(15): E1454–E1462.
- Facial action coding system. Environmental Psychology & Nonverbal Behavior.
- Hypergraph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, 3558–3565.
- Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778.
- Learning invariant deep representation for nir-vis face recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
- Wasserstein CNN: Learning invariant features for NIR-VIS face recognition. IEEE transactions on pattern analysis and machine intelligence, 41(7): 1761–1773.
- Disentangled spectrum variations networks for NIR–VIS face recognition. IEEE Transactions on Multimedia, 22(5): 1234–1248.
- Domain-Private Factor Detachment Network for NIR-VIS Face Recognition. IEEE Transactions on Information Forensics and Security, 17: 1435–1449.
- Orthogonal transformer: An efficient vision transformer backbone with token orthogonalization. Advances in Neural Information Processing Systems, 35: 14596–14607.
- Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform.
- A deeper look at facial expression dataset bias. IEEE Transactions on Affective Computing, 13(2): 881–893.
- Automated facial expression recognition based on FACS action units. In Proceedings third IEEE international conference on automatic face and gesture recognition, 390–395. IEEE.
- Karolinska directed emotional faces. PsycTESTS Dataset, 91: 630.
- T2V-DDPM: Thermal to Visible Face Translation using Denoising Diffusion Probabilistic Models. In 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG), 1–7. IEEE.
- Feature decomposition and reconstruction learning for effective facial expression recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 7660–7669.
- Recognizing action units for facial expression analysis. IEEE Transactions on pattern analysis and machine intelligence, 23(2): 97–115.
- Visualizing data using t-SNE. Journal of machine learning research, 9(11).
- Attention is all you need. Advances in neural information processing systems, 30.
- Suppressing uncertainties for large-scale facial expression recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 6897–6906.
- EASE: Robust Facial Expression Recognition via Emotion Ambiguity-SEnsitive Cooperative Networks. In Proceedings of the 30th ACM International Conference on Multimedia, 218–227.
- Co-Completion for Occluded Facial Expression Recognition. In Proceedings of the 30th ACM International Conference on Multimedia, 130–140.
- Transfer: Learning relation-aware facial expression representations with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 3601–3610.
- Hifacegan: Face renovation via collaborative suppression and replenishment. In Proceedings of the 28th ACM international conference on multimedia, 1551–1560.
- CMOS-GAN: Semi-Supervised Generative Adversarial Model for Cross-Modality Face Image Synthesis. IEEE Transactions on Image Processing, 32: 144–158.
- Facial expression recognition from near-infrared videos. Image and vision computing, 29(9): 607–619.
- Former-dfer: Dynamic facial expression recognition transformer. In Proceedings of the 29th ACM International Conference on Multimedia, 1553–1561.
- Learning deep global multi-scale and local attention features for facial expression recognition in the wild. IEEE Transactions on Image Processing, 30: 6544–6556.