Understanding Hallucinations in Diffusion Models through Mode Interpolation (2406.09358v2)

Abstract: Colloquially speaking, image generation models based upon diffusion processes are frequently said to exhibit "hallucinations," samples that could never occur in the training data. But where do such hallucinations come from? In this paper, we study a particular failure mode in diffusion models, which we term mode interpolation. Specifically, we find that diffusion models smoothly "interpolate" between nearby data modes in the training set, to generate samples that are completely outside the support of the original training distribution; this phenomenon leads diffusion models to generate artifacts that never existed in real data (i.e., hallucinations). We systematically study the reasons for, and the manifestation of this phenomenon. Through experiments on 1D and 2D Gaussians, we show how a discontinuous loss landscape in the diffusion model's decoder leads to a region where any smooth approximation will cause such hallucinations. Through experiments on artificial datasets with various shapes, we show how hallucination leads to the generation of combinations of shapes that never existed. Finally, we show that diffusion models in fact know when they go out of support and hallucinate. This is captured by the high variance in the trajectory of the generated sample towards the final few backward sampling process. Using a simple metric to capture this variance, we can remove over 95% of hallucinations at generation time while retaining 96% of in-support samples. We conclude our exploration by showing the implications of such hallucination (and its removal) on the collapse (and stabilization) of recursive training on synthetic data with experiments on MNIST and 2D Gaussians dataset. We release our code at https://github.com/locuslab/diffusion-model-hallucination.

• The paper identifies mode interpolation as a primary cause of hallucinations in diffusion models, demonstrated through controlled 1D and 2D Gaussian experiments.
• The paper introduces a trajectory variance metric that reliably detects out-of-support samples with high sensitivity, mitigating performance issues in recursive training.
• The paper highlights that integrating robust detection mechanisms can enhance model stability by filtering hallucinated samples during the reverse diffusion process.

Understanding Hallucinations in Diffusion Models through Mode Interpolation

The paper "Understanding Hallucinations in Diffusion Models through Mode Interpolation" by Aithal et al., addresses the phenomenon of hallucination in diffusion models, characterized by the generation of samples lying completely outside the support of the training distribution. This research paper rigorously explores mode interpolation, a previously under-examined cause of such hallucinations.

Key Findings

Diffusion models have become prevalent in various generative tasks due to their ability to produce high-quality, diverse images. However, diffusion models also have failure modes, including hallucinations. The paper's primary contributions can be summarized as follows:

1. Mode Interpolation Phenomenon: The paper identifies mode interpolation as a key driver behind hallucinations. This occurs when diffusion models interpolate between nearby data modes, resulting in samples that are not part of the training data.
2. 1D and 2D Gaussian Experiments: Through controlled experiments with synthetic 1D and 2D Gaussian mixtures, the authors show how mode interpolation leads to hallucinations, providing clear visual evidence and numerical analysis of this phenomenon.
3. Trajectory Variance as a Metric: The authors propose a novel detection mechanism based on the variance in the trajectory of generated samples during the reverse diffusion process. This metric effectively distinguishes between hallucinated and non-hallucinated samples.
4. Implications for Recursive Training: The paper analyzes the impact of hallucinations in recursive model training, demonstrating how they can lead to model collapse over successive generations of training.

Detailed Contributions

Mode Interpolation and Hallucinations

The authors provide a comprehensive analysis of mode interpolation in diffusion models. They trained diffusion models on a mixture of 1D and 2D Gaussians and observed that the models produced samples interpolating between distinct modes, resulting in novel, out-of-support samples, or hallucinations. This effect was shown to decrease with an increasing number of training samples and smaller mode variances.

Score Function Analysis

To further understand the underlying cause of hallucinations, the authors examined the learned score function of the diffusion models. They found that the smooth approximation of the score function by the neural network leads to mode interpolation. Sharp discontinuities between data modes in the true distribution are smoothed out, causing interpolations during the reverse diffusion process.

Trajectory Variance Metric

A significant contribution of this paper is the introduction of a trajectory variance metric. By analyzing the variance in predicted sample trajectories during reverse diffusion, the authors provide a method to detect out-of-support samples. This metric demonstrated high sensitivity and specificity (>0.92) in detecting hallucinations in synthetic datasets.

Recursive Training Implications

Recursive generative model training, where models are trained on their own outputs, poses risks of model collapse. The authors conducted experiments with recursive training on 2D Gaussians and the MNIST dataset, showing that undetected hallucinations exacerbate model collapse. By applying their variance-based filtering mechanism, the authors could mitigate hallucinations and maintain model performance over successive generations.

Implications and Future Directions

This research provides a crucial understanding of the hallucination phenomenon in diffusion models, revealing a significant failure mode related to mode interpolation. By showing how diffusion models interpolate between data modes and proposing a detection metric, the authors offer a practical solution to mitigate hallucinations.

Practical Implications

• Training on Synthetic Data: The findings are particularly relevant as synthetic data generation becomes widespread. Future models exposed to large volumes of machine-generated data need robust detection mechanisms to avoid learning from hallucinations.
• Improving Model Stability: The proposed variance-based detection metric can be integrated into training pipelines to filter out hallucinated samples, thereby enhancing model stability and quality in recursive training setups.

Theoretical Implications

• Score Function Approximation: The paper highlights the limitations of neural networks in approximating discontinuous score functions. Future research could explore advanced neural architectures or regularization techniques to better capture such discontinuities.
• Mode Interaction: The observation that mode interpolation happens primarily between nearby modes opens new directions for studying the interactions between different data modes in high-dimensional spaces.

Conclusion

Aithal et al.'s paper makes a significant contribution to understanding and mitigating hallucinations in diffusion models. By identifying mode interpolation as a primary cause and proposing an effective detection mechanism based on trajectory variance, the authors provide valuable insights and practical tools for improving generative model performance. This research lays the groundwork for future studies to further explore hallucinations and other failure modes in diffusion models, ultimately contributing to the development of more reliable and robust generative models.