- The paper demonstrates that incorporating as little as 3% AI-generated images in retraining leads to significant visual distortions and reduced diversity in Stable Diffusion outputs.
- Quantitative evaluations using FID and CLIP scores reveal a progressive decline in image quality correlating with higher ratios of self-generated data.
- Attempts to recover performance by retraining on real images show only partial improvement, highlighting persistent artifacts and vulnerabilities in generative models.
Analysis of Nepotistically Trained Generative-AI Models Collapse
This paper presents an investigation into the vulnerabilities of generative AI models, specifically within the context of image synthesis. The paper examines the phenomenon of "model-poisoning," where generative-AI systems are retrained on their own outputs. Through iterative retraining, the authors demonstrate that this practice leads to a significant degradation in image quality and diversity, a process referred to as model collapse.
Key Findings
The research focuses on Stable Diffusion, a prominent open-source text-to-image model. The authors reveal several critical insights:
- Model Vulnerability: It is shown that even a minimal inclusion of AI-generated content (as little as 3%) in the retraining dataset results in pronounced visual distortion and reduced diversity in the generated images. This degradation persists across various demographic and non-demographic prompts.
- Quantitative Evaluation: The findings are supported by quantitative assessments using the Fréchet Inception Distance (FID) and Contrastive Language-Image Pre-training (CLIP) scores. The scores illustrate a progressive decline in image quality with increased ratios of self-generated content in the retraining dataset.
- Healing Attempts: Attempts to "heal" the poisoned models by retraining them solely on real images show only partial recovery. Although there is some improvement in FID and CLIP scores, visual artifacts and lack of diversity remain in certain instances.
Methodology
The authors utilize a robust methodological approach:
- Dataset Configuration: Using the FFHQ facial dataset, the authors generate various sets of images by incorporating different ratios of real and AI-generated images. This methodical variation allows for a thorough examination of model behavior across different retraining scenarios.
- Iterative Retraining: The paper adopts an iterative retraining process spanning multiple generations of image synthesis, providing comprehensive insights into the model's performance over time.
- Control Experiments: Control experiments are conducted to account for potential biases such as color histogram discrepancies and low-quality image artifacts, ensuring the validity of the results.
Implications and Future Work
The paper underscores significant concerns about the recursive training of generative models. Practical implications include the risk of unintended data poisoning from online scraping, and the potential for adversarial attacks by embedding AI-generated data within public datasets.
To mitigate these vulnerabilities, the paper suggests several countermeasures, such as enhanced detection of AI-generated content and robust watermarking techniques. However, the authors acknowledge the limitations of these solutions, including the possibility of removing watermarks and the challenges in ensuring image provenance.
Future research directions suggested include:
- Investigating the underlying causes of model poisoning within training data and architecture.
- Exploring resilience-building measures for generative models to withstand self-poisoning.
- Evaluating cross-model poisoning effects, such as the impact on Stable Diffusion models when retrained with images from other generators like DALL-E or Midjourney.
Overall, this paper contributes valuable insights into the fragility of generative AI systems when retrained on self-generated content, reinforcing the need for careful dataset management and the development of more resilient generative techniques.