Benchmarking Counterfactual Image Generation (2403.20287v4)

Abstract: Generative AI has revolutionised visual content editing, empowering users to effortlessly modify images and videos. However, not all edits are equal. To perform realistic edits in domains such as natural image or medical imaging, modifications must respect causal relationships inherent to the data generation process. Such image editing falls into the counterfactual image generation regime. Evaluating counterfactual image generation is substantially complex: not only it lacks observable ground truths, but also requires adherence to causal constraints. Although several counterfactual image generation methods and evaluation metrics exist, a comprehensive comparison within a unified setting is lacking. We present a comparison framework to thoroughly benchmark counterfactual image generation methods. We integrate all models that have been used for the task at hand and expand them to novel datasets and causal graphs, demonstrating the superiority of Hierarchical VAEs across most datasets and metrics. Our framework is implemented in a user-friendly Python package that can be extended to incorporate additional SCMs, causal methods, generative models, and datasets for the community to build on. Code: https://github.com/gulnazaki/counterfactual-benchmark.

• The paper presents a novel benchmark framework that systematically evaluates counterfactual image generation using SCM-conditioned models.
• It assesses models with clear metrics including composition, effectiveness, minimality, and realism—using FID and latent divergence for accurate measurements.
• Experiments show the HVAE model outperforms others on datasets like MorphoMNIST and CelebA, underscoring its practical strengths.

Benchmarking Counterfactual Image Generation Methods

Introduction to Counterfactual Evaluation

Counterfactual image generation is an area of significant interest due to its potential implications in various fields including medical imaging, data augmentation, and interpretability in machine learning models. The capability to generate images under hypothetical scenarios ("what if" questions) lays the groundwork for advances in AI applications that require an understanding of causal relationships. Despite its importance, the criteria for evaluating counterfactual image generation continue to evolve, reflecting the novelty and complexity of the task. This paper introduces a novel framework for benchmarking counterfactual image generation models, particularly focusing on models conditioned on Structural Causal Models (SCM). The presented framework encapsulates metrics for assessing composition, effectiveness, intervention minimality, and image realism, offering a comprehensive evaluation landscape.

Benchmark Framework Overview

The benchmarking framework aims at providing an extensive evaluation across several key metrics:

• Composition and Effectiveness: Leveraging axiomatic definitions of counterfactuals, these metrics assess how changes in intervened variables are reflected in the generated images, and verify the null effect when no interventions are applied.
• Minimality: This metric evaluates the extent of change in the counterfactual image, advocating for minimal alteration from the original image thereby adhering to the sparse mechanism shift hypothesis.
• Realism: Measured by the Fréchet Inception Distance (FID), this metric quantifies the visual quality and authenticity of the generated images relative to a dataset of real images.

Three model families conditioned on SCM, namely Variational Autoencoders (VAE), Conditional Hierarchical Variational Autoencoders (HVAE), and Generative Adversarial Networks (GAN) are evaluated using this framework. An operational Python package encapsulating the benchmark framework is also introduced, providing future researchers with a tool to evaluate new counterfactual image generation methods.

Implementational Details and Metrics

The metrics incorporated into the framework are informed by both the axiomatic properties of counterfactuals and perceptual quality assessments. Specifically, the Composition metric is refined through embedding spaces and perceptual similarity assessments to better capture visual fidelity. Effectiveness is quantified via anti-causal prediction accuracy, providing a direct measure of the intervention's outcome on the target variables. Realism and minimality are reported using FID and a novel Counterfactual Latent Divergence metric, respectively, addressing the dual objectives of generating plausible and minimally altered images.

Experimental Results

The framework was applied to compare the performances of VAE, HVAE, and GAN models across two datasets: MorphoMNIST and CelebA. The HVAE demonstrated superior performance in generating high-quality, realistic images with minimal interventions across both datasets, showcasing the benefits of a hierarchical latent space in capturing complex image attributes. Additionally, this model exhibited strong adherence to the expected theoretical properties of counterfactuals, as evidenced by high scores in composition and effectiveness metrics.

Theoretical and Practical Implications

The proposed benchmark consolidates a methodological basis for evaluating counterfactual image generation. By offering a holistic set of metrics, this work navigates the trade-offs between image realism, faithfulness to interventions, and minimality of changes - critical considerations in applications ranging from synthetic data generation to explainable AI. Moreover, the focus on SCM-conditioned generative models underlines the significance of causality in the future development of AI systems.

Future Perspectives

This paper lays groundwork for future research in counterfactual image generation, opening avenues for the evaluation of diffusion models and other generative frameworks within a causal context. Expanding the benchmark to include more diverse causal mechanisms and datasets remains an important future endeavor. Furthermore, adapting the framework to evaluate emerging generative models underscores the dynamic nature of this research area, promising continuous advancements in understanding and manipulating complex causal relationships in image data.

Conclusive Remarks

The development of a comprehensive benchmark for counterfactual image generation marks a significant step towards establishing standardized evaluation criteria in this field. By systematically assessing multiple aspects of generated images, this framework paves the way for advancements in causal inference and its applications in artificial intelligence. The Python package accompanying this paper further democratizes access to these tools, encouraging rigorous and transparent evaluation of future models.