Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Toward Fair Facial Expression Recognition with Improved Distribution Alignment (2306.06696v1)

Published 11 Jun 2023 in cs.CV

Abstract: We present a novel approach to mitigate bias in facial expression recognition (FER) models. Our method aims to reduce sensitive attribute information such as gender, age, or race, in the embeddings produced by FER models. We employ a kernel mean shrinkage estimator to estimate the kernel mean of the distributions of the embeddings associated with different sensitive attribute groups, such as young and old, in the Hilbert space. Using this estimation, we calculate the maximum mean discrepancy (MMD) distance between the distributions and incorporate it in the classifier loss along with an adversarial loss, which is then minimized through the learning process to improve the distribution alignment. Our method makes sensitive attributes less recognizable for the model, which in turn promotes fairness. Additionally, for the first time, we analyze the notion of attractiveness as an important sensitive attribute in FER models and demonstrate that FER models can indeed exhibit biases towards more attractive faces. To prove the efficacy of our model in reducing bias regarding different sensitive attributes (including the newly proposed attractiveness attribute), we perform several experiments on two widely used datasets, CelebA and RAF-DB. The results in terms of both accuracy and fairness measures outperform the state-of-the-art in most cases, demonstrating the effectiveness of the proposed method.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Analysis of semi-supervised methods for facial expression recognition. In 2022 10th International Conference on Affective Computing and Intelligent Interaction (ACII), pages 1–8, 2022.
  2. Understanding and predicting bonding in conversations using thin slices of facial expressions and body language. In International Conference on Intelligent Virtual Agents, pages 64–74. Springer, 2016.
  3. Facetoponet: Facial expression recognition using face topology learning. IEEE Transactions on Artificial Intelligence, pages 1–14, 2022.
  4. Face trees for expression recognition. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pages 1–5, 2021.
  5. Face image quality vector assessment for biometrics applications. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, pages 511–520, January 2023.
  6. Investigating bias and fairness in facial expression recognition. In Adrien Bartoli and Andrea Fusiello, editors, Computer Vision – ECCV 2020 Workshops, pages 506–523, Cham, 2020. Springer International Publishing.
  7. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2584–2593. IEEE, 2017.
  8. The effect of model compression on fairness in facial expression recognition. ArXiv, abs/2201.01709, 2022.
  9. Face2exp: Combating data biases for facial expression recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20291–20300, June 2022.
  10. Wasserstein fair classification. In Ryan P. Adams and Vibhav Gogate, editors, Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pages 862–872. PMLR, 22–25 Jul 2020.
  11. Exploring disentangled feature representation beyond face identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  12. Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579–2605, 2008.
  13. Gender stereotyping impact in facial expression recognition. In Machine Learning and Principles and Practice of Knowledge Discovery in Databases: International Workshops of ECML PKDD 2022, Grenoble, France, September 19–23, 2022, Proceedings, Part I, pages 9–22. Springer, 2023.
  14. Turning a blind eye: Explicit removal of biases and variation from deep neural network embeddings. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, September 2018.
  15. Towards fairness in visual recognition: Effective strategies for bias mitigation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 8916–8925, 2020.
  16. Sensitivenets: Learning agnostic representations with application to face images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(6):2158–2164, 2021.
  17. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’18, page 335–340, New York, NY, USA, 2018. Association for Computing Machinery.
  18. Optimized pre-processing for discrimination prevention. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  19. Bias remediation in driver drowsiness detection systems using generative adversarial networks. IEEE Access, 8:55592–55601, 2020.
  20. Blinded by beauty: Attractiveness bias and accurate perceptions of academic performance. PloS one, 11(2):e0148284, 2016.
  21. Kernel mean shrinkage estimators. Journal of Machine Learning Research, 17(48):1–41, 2016.
  22. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  23. A survey on bias and fairness in machine learning. ACM Comput. Surv., 54(6), jul 2021.
  24. Bias mitigation post-processing for individual and group fairness. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2847–2851, 2019.
  25. Data augmentation for discrimination prevention and bias disambiguation. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, AIES ’20, page 358–364, New York, NY, USA, 2020. Association for Computing Machinery.
  26. The variational fair autoencoder. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
  27. Facial expression recognition: Impact of gender on fairness and expressions. In Proceedings of the XXII International Conference on Human Computer Interaction, Interacción ’22, New York, NY, USA, 2022. Association for Computing Machinery.
  28. Domain-incremental continual learning for mitigating bias in facial expression and action unit recognition. IEEE Transactions on Affective Computing, pages 1–15, 2022.
  29. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017.
  30. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 3987–3995. JMLR.org, 2017.
  31. Memory aware synapses: Learning what (not) to forget. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision – ECCV 2018, pages 144–161, Cham, 2018. Springer International Publishing.
  32. Re-evaluating continual learning scenarios: A categorization and case for strong baselines. arXiv preprint arXiv:1810.12488, 2018.
  33. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS ’12, page 214–226, New York, NY, USA, 2012. Association for Computing Machinery.
  34. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  35. Counterfactual fairness for facial expression recognition. In Leonid Karlinsky, Tomer Michaeli, and Ko Nishino, editors, Computer Vision – ECCV 2022 Workshops, pages 245–261, Cham, 2023. Springer Nature Switzerland.
  36. Understanding and mitigating annotation bias in facial expression recognition. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14960–14971, 2021.
  37. Comparison of linear shrinkage estimators of a large covariance matrix in normal and non-normal distributions. Computational Statistics & Data Analysis, 95:95–108, 2016.
  38. BEN ADLER. Hilbert spaces and the riesz representation theorem. The University of Chicago Mathematics REU, 2021.
  39. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  40. Tensorflow: A system for large-scale machine learning. In 12th {normal-{\{{USENIX}normal-}\}} Symposium on Operating Systems Design and Implementation ({normal-{\{{OSDI}normal-}\}} 16), pages 265–283, 2016.
  41. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2015.
  42. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 2980–2988, 2017.
  43. Progress & compress: A scalable framework for continual learning. In International conference on machine learning, pages 4528–4537. PMLR, 2018.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Mojtaba Kolahdouzi (5 papers)
  2. Ali Etemad (118 papers)