Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Balancing Act: Distribution-Guided Debiasing in Diffusion Models (2402.18206v3)

Published 28 Feb 2024 in cs.CV

Abstract: Diffusion Models (DMs) have emerged as powerful generative models with unprecedented image generation capability. These models are widely used for data augmentation and creative applications. However, DMs reflect the biases present in the training datasets. This is especially concerning in the context of faces, where the DM prefers one demographic subgroup vs others (eg. female vs male). In this work, we present a method for debiasing DMs without relying on additional data or model retraining. Specifically, we propose Distribution Guidance, which enforces the generated images to follow the prescribed attribute distribution. To realize this, we build on the key insight that the latent features of denoising UNet hold rich demographic semantics, and the same can be leveraged to guide debiased generation. We train Attribute Distribution Predictor (ADP) - a small mlp that maps the latent features to the distribution of attributes. ADP is trained with pseudo labels generated from existing attribute classifiers. The proposed Distribution Guidance with ADP enables us to do fair generation. Our method reduces bias across single/multiple attributes and outperforms the baseline by a significant margin for unconditional and text-conditional diffusion models. Further, we present a downstream task of training a fair attribute classifier by rebalancing the training set with our generated data.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 843–852, 2023.
  2. Label-efficient semantic segmentation with diffusion models. arXiv preprint arXiv:2112.03126, 2021.
  3. A prompt array keeps the bias away: Debiasing vision-language models with adversarial learning. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 806–822, Online only, 2022. Association for Computational Linguistics.
  4. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11472–11481, 2022.
  5. Fair generative modeling via weak supervision. In International Conference on Machine Learning, pages 1887–1898. PMLR, 2020.
  6. Debiasing vision-language models via biased prompts. arXiv preprint arXiv:2302.00070, 2023.
  7. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  8. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021.
  9. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  10. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
  11. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  12. Magnet: Uniform sampling from deep generative network manifolds without retraining. In International Conference on Learning Representations, 2021.
  13. Imperfect imaganation: Implications of gans exacerbating biases on facial data augmentation and snapchat selfie lenses. arXiv preprint arXiv:2001.09528, 2020.
  14. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
  15. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
  16. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  17. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020.
  18. Diffusion models already have a semantic latent space. arXiv preprint arXiv:2210.10960, 2022.
  19. Lucien Le Cam. Asymptotic methods in statistical decision theory. Springer Science & Business Media, 2012.
  20. Implicit maximum likelihood estimation. arXiv preprint arXiv:1809.09087, 2018.
  21. Stable bias: Evaluating societal representations in diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
  22. Studying bias in gans through the lens of race. In European Conference on Computer Vision, pages 344–360. Springer, 2022.
  23. Measuring bias in multimodal models: Multimodal composite association score. In International Workshop on Algorithmic Bias in Search and Recommendation, pages 17–30. Springer, 2023.
  24. Gandiffface: Controllable generation of synthetic datasets for face recognition with realistic variations. arXiv preprint arXiv:2305.19962, 2023.
  25. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
  26. Diffusion models beat gans on image classification. arXiv preprint arXiv:2307.08702, 2023.
  27. Breaking the spurious causality of conditional generation via fairness intervention with corrective sampling. Transactions on Machine Learning Research, 2023.
  28. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
  29. Analyzing bias in diffusion-based face generation models. arXiv preprint arXiv:2305.06402, 2023.
  30. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  31. Fair attribute classification through latent space de-biasing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9301–9310, 2021.
  32. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
  33. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  34. Unbiased face synthesis with diffusion models: Are we there yet? arXiv preprint arXiv:2309.07277, 2023.
  35. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500–22510, 2023.
  36. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  37. Diffusion causal models for counterfactual estimation. In Conference on Causal Learning and Reasoning, pages 647–668. PMLR, 2022.
  38. Fairness gan: Generating datasets with fairness properties using a generative adversarial network. IBM Journal of Research and Development, 63(4/5):3–1, 2019.
  39. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22522–22531, 2023.
  40. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
  41. Dear: Debiasing vision-language models with additive residuals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6820–6829, 2023.
  42. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020.
  43. Improving the fairness of deep generative models without retraining. arXiv preprint arXiv:2012.04842, 2020.
  44. Fair generative models via transfer learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2429–2437, 2023.
  45. Effective data augmentation with diffusion models. arXiv preprint arXiv:2302.07944, 2023.
  46. A fair generative model using lecam divergence. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10034–10042, 2023.
  47. Decaf: Generating fair synthetic data using causally-aware generative networks. Advances in Neural Information Processing Systems, 34:22221–22233, 2021.
  48. Sketch-guided text-to-image diffusion models. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1–11, 2023.
  49. Diffusion models for medical anomaly detection. In International Conference on Medical image computing and computer-assisted intervention, pages 35–45. Springer, 2022.
  50. Fairgan: Fairness-aware generative adversarial networks. In 2018 IEEE International Conference on Big Data (Big Data), pages 570–575. IEEE, 2018.
  51. Inclusive gan: Improving data and minority coverage in generative models. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16, pages 377–393. Springer, 2020.
  52. Iti-gen: Inclusive text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3969–3980, 2023.
  53. Debiased fine-tuning for vision-language models by prompt regularization. arXiv preprint arXiv:2301.12429, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Rishubh Parihar (12 papers)
  2. Abhijnya Bhat (2 papers)
  3. Saswat Mallick (1 paper)
  4. Abhipsa Basu (3 papers)
  5. Jogendra Nath Kundu (26 papers)
  6. R. Venkatesh Babu (108 papers)
Citations (7)
View on Semantic Scholar