Efficient Shapley Values for Attributing Global Properties of Diffusion Models to Data Group (2407.03153v1)

Abstract: As diffusion models are deployed in real-world settings, data attribution is needed to ensure fair acknowledgment for contributors of high-quality training data and to identify sources of harmful content. Previous work focuses on identifying individual training samples important for the generation of a given image. However, instead of focusing on a given generated image, some use cases require understanding global properties of the distribution learned by a diffusion model (e.g., demographic diversity). Furthermore, training data for diffusion models are often contributed in groups rather than separately (e.g., multiple artworks from the same artist). Hence, here we tackle the problem of attributing global properties of diffusion models to groups of training data. Specifically, we develop a method to efficiently estimate Shapley values by leveraging model pruning and fine-tuning. We empirically demonstrate the utility of our method with three use cases: (i) global image quality for a DDPM trained on a CIFAR dataset, (ii) demographic diversity for an LDM trained on CelebA-HQ, and (iii) overall aesthetic quality for a Stable Diffusion model LoRA-finetuned on Post-Impressionist artworks.

• The paper introduces an efficient method that leverages model pruning and fine-tuning to approximate Shapley values for attributing global properties in diffusion models.
• The study validates its approach across use cases in image quality, demographic diversity, and aesthetic assessment to confirm its effectiveness.
• The findings enhance fairness and transparency in generative models while significantly reducing the computational demands of traditional attribution techniques.

Efficient Shapley Values for Attributing Global Properties of Diffusion Models to Data Groups

This research paper addresses a crucial challenge in the domain of generative models, specifically diffusion models, which have shown remarkable success in tasks such as image generation. As these models become integral in various real-world applications, understanding the contribution of training data becomes critical, particularly for maintaining fairness and identifying biases.

The paper introduces a method to attribute global properties of diffusion models to grouped training data. Traditional methods often focus on individual data attribution concerning the generation of a specific image. However, this research shifts the perspective to global properties, which characterize the overall distribution a diffusion model learns. This approach is essential when training datasets are contributed in groups, such as artworks from the same artist or various images from a particular demographic.

By leveraging model pruning and fine-tuning, the authors propose an efficient technique for estimating Shapley values. Shapley values, rooted in cooperative game theory, provide a robust framework for fair data attribution, ensuring that the contributions of different data groups toward global model properties are fairly acknowledged. Estimating Shapley values typically demands extensive computational resources, including retraining models on multiple data subsets. The proposed method circumvents this by employing efficient model pruning and fine-tuning strategies to approximate the original retraining process, thus reducing the computational load.

Empirical validations across three main use cases demonstrate the efficacy of the proposed method:

1. Global Image Quality: A denoising diffusion probabilistic model (DDPM) trained on the CIFAR dataset was used to analyze how group-wise data contributions affect the model's overall image quality.
2. Demographic Diversity: For a latent diffusion model (LDM) trained on the CelebA-HQ dataset, the method was used to measure how different groups contribute to demographic diversity, a global property reflecting potential biases in generated content.
3. Aesthetic Quality: Testing on a Stable Diffusion model fine-tuned on Post-Impressionist artworks highlighted the method's effectiveness in attributing aesthetic contributions from different artistic groups.

The research also contrasts this approach with existing attribution methods that are more localized and do not focus on global model properties. Unlike methods such as TRAK or D-TRAK, which often emphasize pixel-level or feature-level amalgamation, this method affords an aggregated understanding of group-level contribution, marking a distinct advancement in the analysis of generative models.

From a theoretical standpoint, this work contributes substantially to the field by presenting an attribution system that can be generalized to various model types and data groupings, addressing the fairness and equity concerns prevalent in machine learning. Practically, deploying such methods can enhance the accountability of diffusion models in deployment settings, fostering greater trust and transparency.

Looking forward, this research opens pathways to further explore and refine attribution methods, especially in the context of large-scale models and datasets. The integration of unlearning techniques and more intricate pruning strategies could yield even more potent computational savings and attribution accuracy. Additionally, expanding these methodologies to handle proprietary models or datasets where direct model access may be restricted poses a promising research direction, potentially employing in-context learning methods or other inference-driven strategies to approximate similar Shapley values.

In conclusion, the authors present a compelling framework that not only aids researchers and practitioners in understanding the intricate dependencies of data-driven generative models but also steers the community towards more equitable and bias-aware model development practices.