Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Counterfactual Image Editing (2403.09683v1)

Published 7 Feb 2024 in cs.CV and cs.LG

Abstract: Counterfactual image editing is an important task in generative AI, which asks how an image would look if certain features were different. The current literature on the topic focuses primarily on changing individual features while remaining silent about the causal relationships between these features, as present in the real world. In this paper, we formalize the counterfactual image editing task using formal language, modeling the causal relationships between latent generative factors and images through a special type of model called augmented structural causal models (ASCMs). Second, we show two fundamental impossibility results: (1) counterfactual editing is impossible from i.i.d. image samples and their corresponding labels alone; (2) even when the causal relationships between the latent generative factors and images are available, no guarantees regarding the output of the model can be provided. Third, we propose a relaxation for this challenging problem by approximating non-identifiable counterfactual distributions with a new family of counterfactual-consistent estimators. This family exhibits the desirable property of preserving features that the user cares about across both factual and counterfactual worlds. Finally, we develop an efficient algorithm to generate counterfactual images by leveraging neural causal models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (105)
  1. Diffusion visual counterfactual explanations. Advances in Neural Information Processing Systems, 35:364–377, 2022.
  2. Identifiability of Path-Specific Effects. In Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence, pp.  357–363. Morgan-Kaufmann Publishers, 2005.
  3. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  18208–18218, 2022.
  4. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  5. Counterfactual Probabilities: Computational Methods, Bounds, and Applications. In de Mantaras, R. L. and D.~Poole (eds.), Uncertainty in Artificial Intelligence 10, pp.  46–54. Morgan Kaufmann, San Mateo, CA, 1994.
  6. Bandits with unobserved confounders: A causal approach. In Advances in Neural Information Processing Systems, pp.  1342–1350, 2015.
  7. On pearl’s hierarchy and the foundations of causal inference. In Probabilistic and Causal Inference: The Works of Judea Pearl, pp.  507–556. Association for Computing Machinery, New York, NY, USA, 2022.
  8. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
  9. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems, 35:38319–38331, 2022.
  10. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=B1xsqj09Fm.
  11. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  18392–18402, 2023.
  12. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pp.  77–91. PMLR, 2018.
  13. Using latent space regression to analyze and leverage compositionality in {gan}s. In International Conference on Learning Representations, 2021.
  14. Isolating sources of disentanglement in variational autoencoders. In NeurIPS, 2018.
  15. Child, R. Very deep {vae}s generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations, 2021.
  16. Nested counterfactual identification from arbitrary surrogate experiments. Advances in Neural Information Processing Systems, 34:6856–6867, 2021a.
  17. Nested counterfactual identification from arbitrary surrogate experiments. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.  6856–6867. Curran Associates, Inc., 2021b.
  18. Vqgan-clip: Open domain image generation and editing with natural language guidance. In European Conference on Computer Vision, pp.  88–105. Springer, 2022.
  19. Deng, L. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6):141–142, 2012.
  20. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. Advances in neural information processing systems, 31, 2018.
  21. Adversarial feature learning. In International Conference on Learning Representations, 2017.
  22. Generating images with perceptual similarity metrics based on deep networks. Advances in neural information processing systems, 29, 2016.
  23. Adversarially learned inference. In International Conference on Learning Representations, 2017.
  24. A framework for contrastive self-supervised learning and designing a new approach. arXiv preprint arXiv:2009.00104, 2020.
  25. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG), 41(4):1–13, 2022.
  26. Ganalyze: Toward visual definitions of cognitive image properties. In Proceedings of the ieee/cvf international conference on computer vision, pp.  5744–5753, 2019.
  27. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
  28. Explaining classifiers with causal concept effect (cace). arXiv preprint arXiv:1907.07165, 2019a.
  29. Counterfactual visual explanations. In International Conference on Machine Learning, pp.  2376–2384. PMLR, 2019b.
  30. Ganspace: Discovering interpretable gan controls. Advances in Neural Information Processing Systems, 33:9841–9850, 2020.
  31. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  32. Heckman, J. J. Randomization and Social Policy Evaluation. In Manski, C. and Garfinkle, I. (eds.), Evaluations: Welfare and Training Programs, pp.  201–230. Harvard University Press, Cambridge, MA, 1992.
  33. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
  34. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  35. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1):2249–2281, 2022.
  36. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429–439, 1999.
  37. Causal identification under Markov equivalence. In Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, pp.  978–987. AUAI Press, Aug 2018.
  38. Identification of conditional causal effects under Markov equivalence. In Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32, pp.  11512–11520. Curran Associates, Inc., 2019.
  39. Causal discovery from soft interventions with unknown targets: Characterization and learning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.  9551–9561. Curran Associates, Inc., 2020.
  40. Causal identification under markov equivalence: Calculus, algorithm, and completeness. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.  3679–3690. Curran Associates, Inc., 2022.
  41. On the "steerability" of generative adversarial networks. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=HylsTT4FvB.
  42. Towards realistic individual recourse and actionable explanations in black-box decision making systems. arXiv preprint arXiv:1907.09615, 2019.
  43. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
  44. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  4401–4410, 2019.
  45. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pp.  2207–2217. PMLR, 2020.
  46. Cycle-consistent counterfactuals by latent transformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10203–10212, 2022.
  47. Disentangling by factorising. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.  2649–2658. PMLR, 10–15 Jul 2018. URL https://proceedings.mlr.press/v80/kim18b.html.
  48. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  49. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  50. Experimental design for learning causal graphs with latent variables. In Advances in Neural Information Processing Systems 30, pp.  7018–7028. Curran Associates, Inc., 2017a.
  51. Causalgan: Learning causal implicit generative models with adversarial training. arXiv preprint arXiv:1709.02023, 2017b.
  52. CausalGAN: Learning causal implicit generative models with adversarial training. In International Conference on Learning Representations, 2018.
  53. Characterization and learning of causal graphs with latent variables from soft interventions. In Advances in Neural Information Processing Systems 32, pp.  14346–14356, Vancouver, Canada, 2019. Curran Associates, Inc.
  54. Clipstyler: Image style transfer with a single text condition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  18062–18071, 2022.
  55. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. In First Conference on Causal Learning and Reasoning, 2021.
  56. Generalized transportability: Synthesis of experiments from heterogeneous domains. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, New York, NY, 2020. AAAI Press.
  57. Causal discovery from observational and interventional data across multiple environments. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  58. Challenging common assumptions in the unsupervised learning of disentangled representations. ArXiv, abs/1811.12359, 2019a.
  59. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp.  4114–4124. PMLR, 2019b.
  60. Weakly-supervised disentanglement without compromises. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.  6348–6359. PMLR, 13–18 Jul 2020a. URL https://proceedings.mlr.press/v119/locatello20a.html.
  61. Disentangling factors of variations using few labels. In International Conference on Learning Representations, 2020b. URL https://openreview.net/forum?id=SygagpEKwB.
  62. Manski, C. F. Nonparametric bounds on treatment effects. American Economic Review, Papers and Proceedings, 80:319–323, 1990.
  63. Causal transportability for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  7521–7531, 2022.
  64. Joint causal inference from multiple contexts. The Journal of Machine Learning Research, 21(1):3919–4026, 2020.
  65. Causal interpretability for machine learning-problems, methods and evaluation. ACM SIGKDD Explorations Newsletter, 22(1):18–33, 2020.
  66. Counterfactual (non-) identifiability of learned structural causal models. arXiv preprint arXiv:2301.09031, 2023.
  67. DiffuseVAE: Efficient, controllable and high-fidelity generation from low-dimensional latents. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
  68. Automatic differentiation in pytorch. 2017.
  69. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  2085–2094, 2021.
  70. Deep structural causal models for tractable counterfactual inference. Advances in Neural Information Processing Systems, 33:857–869, 2020.
  71. Pearl, J. Causality: Models, Reasoning, and Inference. Cambridge University Press, New York, NY, USA, 2nd edition, 2000.
  72. Pearl, J. Direct and indirect effects. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, UAI’01, pp.  411–420, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608001.
  73. Pearl, J. Causality: Models, Reasoning, and Inference. Cambridge University Press, New York, 2nd edition, 2009.
  74. The Book of Why. Basic Books, New York, 2018.
  75. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, 2017.
  76. Causal fairness analysis. Technical Report R-90, Causal Artificial Intelligence Lab, Columbia University, Jul 2022.
  77. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.  8748–8763. PMLR, 2021.
  78. Beyond trivial counterfactual explanations with diverse valuable explanations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  1056–1065, 2021.
  79. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4713–4726, 2022.
  80. Explaingan: Model explanation via decision boundary crossing transformations. In Proceedings of the European Conference on Computer Vision (ECCV), pp.  666–681, 2018.
  81. Diffusion causal models for counterfactual estimation. In Schölkopf, B., Uhler, C., and Zhang, K. (eds.), Proceedings of the First Conference on Causal Learning and Reasoning, volume 177 of Proceedings of Machine Learning Research, pp.  647–668. PMLR, 11–13 Apr 2022.
  82. Counterfactual generative networks. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=BXewfAYMmJw.
  83. Toward causal representation learning. Proceedings of the IEEE, 109(5):612–634, 2021.
  84. Weakly supervised disentangled generative causal representation learning. Journal of Machine Learning Research, 23:1–55, 2022.
  85. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  9243–9252, 2020.
  86. Effects of Treatment on the Treated: Identification and Generalization. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, Quebec, 2009. AUAI Press.
  87. Identification of Personalized Effects Associated With Causal Pathways. In Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, pp.  530–539, 2018.
  88. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
  89. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
  90. Causation, Prediction, and Search. MIT Press, Cambridge, MA, 2nd edition, 2000.
  91. Permutation-based causal structure learning with unknown intervention targets. In Conference on Uncertainty in Artificial Intelligence, pp.  1039–1048. PMLR, 2020.
  92. Nvae: A deep hierarchical variational autoencoder. Advances in neural information processing systems, 33:19667–19679, 2020.
  93. Interpretable counterfactual explanations guided by prototypes. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13–17, 2021, Proceedings, Part II 21, pp.  650–665. Springer, 2021.
  94. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  95. Counterfactual explanations and algorithmic recourses for machine learning: a review. arXiv preprint arXiv:2010.10596, 2020.
  96. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:16451–16467, 2021.
  97. Nonparametric identifiability of causal representations from unknown interventions. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  98. Scout: Self-aware discriminant counterfactual explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  8981–8990, 2020.
  99. The causal-neural connection: Expressiveness, learnability, and inference. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.  10823–10836. Curran Associates, Inc., 2021. URL https://causalai.net/r80.pdf.
  100. Neural causal models for counterfactual identification and estimation. In International Conference on Learning Representations, 2022.
  101. Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  9593–9602, 2021.
  102. Self-attention generative adversarial networks. In International conference on machine learning, pp.  7354–7363. PMLR, 2019.
  103. Fairness in decision-making–the causal explanation formula. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pp.  2037–2045, 2018.
  104. Partial counterfactual identification from observational and experimental data. In Proceedings of the 39th International Conference on Machine Learning (ICML-22), 2022.
  105. Identifiability guarantees for causal disentanglement from soft interventions. arXiv preprint arXiv:2307.06250, 2023.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets