Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Bridging the Gap: Protocol Towards Fair and Consistent Affect Analysis (2405.06841v2)

Published 10 May 2024 in cs.CV and cs.LG

Abstract: The increasing integration of machine learning algorithms in daily life underscores the critical need for fairness and equity in their deployment. As these technologies play a pivotal role in decision-making, addressing biases across diverse subpopulation groups, including age, gender, and race, becomes paramount. Automatic affect analysis, at the intersection of physiology, psychology, and machine learning, has seen significant development. However, existing databases and methodologies lack uniformity, leading to biased evaluations. This work addresses these issues by analyzing six affective databases, annotating demographic attributes, and proposing a common protocol for database partitioning. Emphasis is placed on fairness in evaluations. Extensive experiments with baseline and state-of-the-art methods demonstrate the impact of these changes, revealing the inadequacy of prior assessments. The findings underscore the importance of considering demographic attributes in affect analysis research and provide a foundation for more equitable methodologies. Our annotations, code and pre-trained models are available at: https://github.com/dkollias/Fair-Consistent-Affect-Analysis

Definition Search Book Streamline Icon: https://streamlinehq.com
References (72)
  1. Learning facial expression-aware global-to-local representation for robust action unit detection. Applied Intelligence, pages 1–21, 2024.
  2. Exploiting emotional dependencies with graph convolutional networks for facial expression recognition. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pages 1–8, 2021.
  3. Data-driven covid-19 detection through medical imaging. In 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), pages 1–5. IEEE, 2023.
  4. A large imaging database and novel deep neural architecture for covid-19 diagnosis. In 2022 IEEE 14th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), page 1–5. IEEE, 2022.
  5. Uncertainty-guided contrastive learning for single source domain generalisation. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6935–6939. IEEE, 2024.
  6. Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In Proceedings of IEEE International Conference on Computer Vision & Pattern Recognition (CVPR’16), Las Vegas, NV, USA, June 2016.
  7. Vitfer: facial emotion recognition with vision transformers. Applied System Innovation, 5(4):80, 2022.
  8. Biomechanics-guided facial action unit detection through force modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8694–8703, 2023.
  9. P. Ekman. Facial action coding system (facs). A human face, 2002.
  10. A. H. Farzaneh and X. Qi. Facial expression recognition in the wild via deep attentive center loss. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2402–2411, 2021.
  11. Covid-19 computer-aided diagnosis through ai-assisted ct imaging analysis: Deploying a medical ai system. arXiv preprint arXiv:2403.06242, 2024.
  12. Sayette group formation task (gft) spontaneous facial expression database. In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pages 581–588. IEEE, 2017.
  13. Simultaneous prediction of valence/arousal and emotions on affectnet, aff-wild and afew-va. Procedia Computer Science, 170:634–641, 2020.
  14. K. Karkkainen and J. Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1548–1558, 2021.
  15. D. Kollias. Abaw: Valence-arousal estimation, expression recognition, action unit detection & multi-task learning challenges. arXiv preprint arXiv:2202.10659, 2022.
  16. D. Kollias. Abaw: Learning from synthetic data & multi-task learning challenges. In European Conference on Computer Vision, pages 157–172. Springer, 2023.
  17. D. Kollias. Multi-label compound expression recognition: C-expr database & network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5589–5598, 2023.
  18. Ai-enabled analysis of 3-d ct scans for diagnosis of covid-19 & its severity. In 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), page 1–5. IEEE, 2023.
  19. A deep neural architecture for harmonizing 3-d input data analysis and decision making in medical imaging. Neurocomputing, 542:126244, 2023.
  20. Domain adaptation, explainability & fairness in ai for medical image analysis: Diagnosis of covid-19 based on 3-d chest ct-scans. arXiv preprint arXiv:2403.02192, 2024.
  21. Deep transparent prediction through latent representation analysis. arXiv preprint arXiv:2009.07044, 2020.
  22. Photorealistic facial synthesis in the dimensional affect space. In European Conference on Computer Vision, pages 475–491. Springer, 2018.
  23. Deep neural network augmentation: Generating faces for affect analysis. International Journal of Computer Vision, pages 1–30, 2020.
  24. Interweaving deep learning and semantic techniques for emotion analysis in human-machine interaction. In 2015 10th International Workshop on Semantic and Social Media Adaptation and Personalization (SMAP), pages 1–6. IEEE, 2015.
  25. Recognition of affect in the wild using deep neural networks. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on, pages 1972–1979. IEEE, 2017.
  26. Facernet: a facial expression intensity estimation network. arXiv preprint arXiv:2303.00180, 2023.
  27. Analysing affective behavior in the first abaw 2020 competition. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)(FG), pages 794–800. IEEE Computer Society, 2020.
  28. Face behavior a la carte: Expressions, affect and action units in a single network. arXiv preprint arXiv:1910.11111, 2019.
  29. Distribution matching for heterogeneous multi-task learning: a large-scale face study. arXiv preprint arXiv:2105.03790, 2021.
  30. Distribution matching for multi-task learning of classification tasks: a large-scale study on faces & beyond. arXiv preprint arXiv:2401.01219, 2024.
  31. Deep neural architectures for prediction in healthcare. Complex & Intelligent Systems, 4(2):119–131, 2018.
  32. Abaw: Valence-arousal estimation, expression recognition, action unit detection & emotional reaction intensity estimation challenges. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5888–5897, 2023.
  33. The 6th affective behavior analysis in-the-wild (abaw) competition. arXiv preprint arXiv:2402.19344, 2024.
  34. Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond. International Journal of Computer Vision, pages 1–23, 2019.
  35. Btdnet: A multi-modal approach for brain tumor radiogenomic classification. Applied Sciences, 13(21):11984, 2023.
  36. Transparent adaptation in deep medical image diagnosis. In TAILOR, page 251–267, 2020.
  37. D. Kollias and S. Zafeiriou. Training deep neural networks with different datasets in-the-wild: The emotion recognition paradigm. In 2018 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2018.
  38. D. Kollias and S. Zafeiriou. Expression, affect, action unit recognition: Aff-wild2, multi-task learning and arcface. arXiv preprint arXiv:1910.04855, 2019.
  39. D. Kollias and S. Zafeiriou. Affect analysis in-the-wild: Valence-arousal, expressions, action units and a unified framework. arXiv preprint arXiv:2103.15792, 2021.
  40. D. Kollias and S. Zafeiriou. Analysing affective behavior in the second abaw2 competition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3652–3660, 2021.
  41. D. Kollias and S. P. Zafeiriou. Exploiting multi-cnn features in cnn-rnn based dimensional emotion recognition on the omg in-the-wild dataset. IEEE Transactions on Affective Computing, 2020.
  42. Factorized higher-order cnns with an application to spatio-temporal emotion estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6060–6069, 2020.
  43. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pages 2584–2593. IEEE, 2017.
  44. Self-supervised representation learning from videos for facial action unit detection. In Proceedings of the IEEE/CVF Conference on Computer vision and pattern recognition, pages 10924–10933, 2019.
  45. Facial expression recognition based on multi-modal features for videos in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5871–5878, 2023.
  46. Pose-disentangled contrastive learning for self-supervised facial representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9717–9728, 2023.
  47. Learning multi-dimensional edge feature-based au relation graph for facial action unit recognition. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, pages 1239–1246, 2022.
  48. Poster v2: A simpler and stronger facial expression recognition network. arXiv preprint arXiv:2301.12149, 2023.
  49. Disfa: A spontaneous facial action intensity database. Affective Computing, IEEE Transactions on, 4(2):151–160, 2013.
  50. Valence and arousal estimation in-the-wild with tensor methods. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pages 1–7. IEEE, 2019.
  51. Affectnet: A database for facial expression, valence, and arousal computing in the wild. arXiv preprint arXiv:1708.03985, 2017.
  52. Efficient labelling of affective video datasets via few-shot & multi-task contrastive learning. In Proceedings of the 31st ACM International Conference on Multimedia, pages 6161–6170, 2023.
  53. Examining subject-dependent and subject-independent human affect inference from limited video data. In 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG), pages 1–6. IEEE, 2023.
  54. A. Psaroudakis and D. Kollias. Mixaugment & mixup: Augmentation methods for facial expression recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2367–2375, 2022.
  55. Multi-view dynamic facial action unit detection. Image and Vision Computing, 2018.
  56. J. A. Russell. Evidence of convergent validity on the dimensions of affect. Journal of personality and social psychology, 36(10):1152, 1978.
  57. Medical image segmentation: A review of modern architectures. In Computer Vision–ECCV 2022 Workshops: Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII, pages 691–708. Springer, 2023.
  58. Classifying emotions and engagement in online learning based on a single facial expression recognition neural network. IEEE Transactions on Affective Computing, 13(4):2132–2143, 2022.
  59. R. T. Schaefer. Encyclopedia of race, ethnicity, and society, volume 1. Sage, 2008.
  60. Joint facial action unit recognition and self-supervised optical flow estimation. Pattern Recognition Letters, 2024.
  61. Critically examining the domain generalizability of facial expression recognition models. arXiv preprint arXiv:2106.15453, 2023.
  62. Tal emotionet challenge 2020 rethinking the model chosen problem in multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 412–413, 2020.
  63. Distract your attention: Multi-head cross attention network for facial expression recognition. Biomimetics, 8(2):199, 2023.
  64. Facial action unit recognition in the wild with multi-task cnn self-training for the emotionet challenge. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 410–411, 2020.
  65. Raf-au database: in-the-wild facial expressions with subjective emotion judgement and objective au annotations. In Proceedings of the Asian conference on computer vision, 2020.
  66. Local region perception and relationship learning combined with feature fusion for facial action unit detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5784–5791, 2023.
  67. Aff-wild: Valence and arousal ‘in-the-wild’challenge. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on, pages 1980–1987. IEEE, 2017.
  68. Multi-modal facial affective analysis based on masked autoencoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5792–5801, 2023.
  69. Multimodal channel-mixing: Channel and spatial masked autoencoder on facial action unit detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6077–6086, 2024.
  70. Learn from all: Erasing attention consistency for noisy label facial expression recognition. In European Conference on Computer Vision, pages 418–434. Springer, 2022.
  71. Learning deep global multi-scale and local attention features for facial expression recognition in the wild. IEEE Transactions on Image Processing, 30:6544–6556, 2021.
  72. Poster: A pyramid cross-fusion transformer network for facial expression recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3146–3155, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Guanyu Hu (46 papers)
  2. Eleni Papadopoulou (2 papers)
  3. Dimitrios Kollias (48 papers)
  4. Paraskevi Tzouveli (5 papers)
  5. Jie Wei (14 papers)
  6. Xinyu Yang (109 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now