Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Gaussian Harmony: Attaining Fairness in Diffusion-based Face Generation Models (2312.14976v1)

Published 21 Dec 2023 in cs.CV and cs.CY

Abstract: Diffusion models have achieved great progress in face generation. However, these models amplify the bias in the generation process, leading to an imbalance in distribution of sensitive attributes such as age, gender and race. This paper proposes a novel solution to this problem by balancing the facial attributes of the generated images. We mitigate the bias by localizing the means of the facial attributes in the latent space of the diffusion model using Gaussian mixture models (GMM). Our motivation for choosing GMMs over other clustering frameworks comes from the flexible latent structure of diffusion model. Since each sampling step in diffusion models follows a Gaussian distribution, we show that fitting a GMM model helps us to localize the subspace responsible for generating a specific attribute. Furthermore, our method does not require retraining, we instead localize the subspace on-the-fly and mitigate the bias for generating a fair dataset. We evaluate our approach on multiple face attribute datasets to demonstrate the effectiveness of our approach. Our results demonstrate that our approach leads to a more fair data generation in terms of representational fairness while preserving the quality of generated samples.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. How does gender balance in training data affect face recognition accuracy? In 2020 ieee international joint conference on biometrics (ijcb), pages 1–10. IEEE, 2020.
  2. Gendered differences in face recognition accuracy explained by hairstyles, makeup, and facial morphology. IEEE Transactions on Information Forensics and Security, 17:127–137, 2021.
  3. Face regions impact recognition accuracy differently across demographics. In 2022 IEEE International Joint Conference on Biometrics (IJCB), pages 1–9. IEEE, 2022.
  4. Data augmentation generative adversarial networks. arXiv preprint arXiv:1711.04340, 2017.
  5. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018.
  6. Fair generative modeling via weak supervision. In International Conference on Machine Learning, pages 1887–1898. PMLR, 2020a.
  7. Fair generative modeling via weak supervision. In International Conference on Machine Learning, pages 1887–1898. PMLR, 2020b.
  8. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  9. Gustavo H de Rosa and Joao P Papa. A survey on text generation using generative adversarial networks. Pattern Recognition, 119:108098, 2021.
  10. How are attributes expressed in face dcnns? In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pages 85–92. IEEE, 2020.
  11. Pass: protected attribute suppression system for mitigating bias in face recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15087–15096, 2021a.
  12. Distill and de-bias: Mitigating bias in face verification using knowledge distillation. arXiv preprint arXiv:2112.09786, 2021b.
  13. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021.
  14. Wav2pix: Speech-conditioned face generation using generative adversarial networks. In ICASSP, pages 8633–8637, 2019.
  15. Fair generation through prior modification. In 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), 2020.
  16. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020a.
  17. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020b.
  18. Bias correction of learned generative models using likelihood-free importance weighting. Advances in neural information processing systems, 32, 2019a.
  19. Bias correction of learned generative models using likelihood-free importance weighting. Advances in neural information processing systems, 32, 2019b.
  20. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  21. Collaborative diffusion for multi-modal face generation and editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6080–6090, 2023.
  22. Magnet: Uniform sampling from deep generative network manifolds without retraining. In International Conference on Learning Representations, 2021.
  23. 50 years of test (un) fairness: Lessons for machine learning. In Proceedings of the conference on fairness, accountability, and transparency, pages 49–58, 2019.
  24. Imperfect imaganation: Implications of gans exacerbating biases on facial data augmentation and snapchat face lenses. Artificial Intelligence, 304:103652, 2022.
  25. Suspect face generation. In 2020 3rd International Conference on Communication System, Computing and IT Applications (CSCITA), pages 73–78. IEEE, 2020.
  26. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1548–1558, 2021.
  27. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
  28. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  29. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020.
  30. Issues related to face recognition accuracy varying based on race and skin tone. IEEE Transactions on Technology and Society, 1(1):8–20, 2020.
  31. Generative adversarial networks for image and video synthesis: Algorithms and applications. Proceedings of the IEEE, 109(5):839–862, 2021.
  32. Stable bias: Analyzing societal representations in diffusion models. arXiv preprint arXiv:2303.11408, 2023.
  33. Studying bias in gans through the lens of race. In European Conference on Computer Vision, pages 344–360. Springer, 2022.
  34. A multimodal comparison of latent denoising diffusion probabilistic models and generative adversarial networks for medical image synthesis. Scientific Reports, 13(1):12098, 2023.
  35. Social biases through the text-to-image generation lens. arXiv preprint arXiv:2304.06034, 2023.
  36. Few-shot image generation via cross-domain correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10743–10752, 2021.
  37. Analyzing bias in diffusion-based face generation models. arXiv preprint arXiv:2305.06402, 2023.
  38. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pages 1–10, 2022.
  39. High-fidelity guided image synthesis with latent diffusion models. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5997–6006. IEEE, 2023.
  40. Diffused heads: Diffusion models beat gans on talking-face generation. arXiv preprint arXiv:2301.03396, 2023.
  41. Improving the fairness of deep generative models without retraining. arXiv preprint arXiv:2012.04842, 2020.
  42. Fair generative models via transfer learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2429–2437, 2023.
  43. Face recognition accuracy across demographics: Shining a light into the problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1041–1050, 2023.
  44. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 2022.
  45. Generalizable feature learning in the presence of data bias and domain class imbalance with application to skin lesion classification. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part IV 22, pages 365–373. Springer, 2019.
  46. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Basudha Pal (4 papers)
  2. Arunkumar Kannan (1 paper)
  3. Ram Prabhakar Kathirvel (2 papers)
  4. Alice J. O'Toole (13 papers)
  5. Rama Chellappa (190 papers)
Citations (1)
View on Semantic Scholar