Reliable Fidelity and Diversity Metrics for Generative Models (2002.09797v2)

Abstract: Devising indicative evaluation metrics for the image generation task remains an open problem. The most widely used metric for measuring the similarity between real and generated images has been the Fr\'echet Inception Distance (FID) score. Because it does not differentiate the fidelity and diversity aspects of the generated images, papers have introduced variants of precision and recall metrics to diagnose those properties separately. In this paper, we show that even the latest version of the precision and recall metrics are not reliable yet. For example, they fail to detect the match between two identical distributions, they are not robust against outliers, and the evaluation hyperparameters are selected arbitrarily. We propose density and coverage metrics that solve the above issues. We analytically and experimentally show that density and coverage provide more interpretable and reliable signals for practitioners than the existing metrics. Code: https://github.com/clovaai/generative-evaluation-prdc.

Reliable Fidelity and Diversity Metrics for Generative Models

The paper "Reliable Fidelity and Diversity Metrics for Generative Models" addresses a critical aspect of image generation tasks involving the evaluation metrics for generative models. Traditional metrics, such as the Fréchet Inception Distance (FID), have provided a single score assessment of the distance between real and generated images, which fails to differentiate between fidelity and diversity—the two essential qualities that characterize the efficacy of generative models.

Key Contributions

1. Critique of Existing Metrics: The paper critiques existing metrics like precision and recall, which, despite their capabilities to measure fidelity and diversity separately, exhibit several shortcomings. These include an inability to detect a match between identical distributions, lack of robustness to outliers, insensitivity to mode dropping, and arbitrary hyperparameter selection. The paper finds that even the latest improvements in these metrics remain inadequate for precisely evaluating generative models.
2. Proposal of Density and Coverage Metrics: To address the issues with existing metrics, the authors introduce density and coverage metrics. These metrics are designed to be both empirically reliable and theoretically analyzable. They base their approach on manipulating manifold estimations to enhance robustness against the aforementioned drawbacks.
3. Analysis and Comparison: The paper provides comprehensive analytical and empirical comparisons between the proposed metrics and existing methods. The authors show that density and coverage provide more interpretable and reliable signals by addressing the pitfalls of existing metrics like overestimation of manifolds and susceptibility to outliers.
4. Focus on Embedding Techniques: An important aspect of the work is its focus on the role of embeddings in generative model evaluation. While traditional evaluations use embeddings derived from pre-trained ImageNet models, the authors argue that such embeddings can lead to biased assessments. Particularly when data distributions deviate significantly from ImageNet-like distributions, they observe that embeddings from randomly initialized models can offer a more unbiased and accurate evaluation.

Practical and Theoretical Implications

From a practical standpoint, the introduction of density and coverage metrics could significantly enhance model diagnostics, leading to better understanding and tuning of generative models. The authors show that density better captures how well-generated samples populate the regions where real samples are dense, and coverage ensures that the generated samples span the full diversity of real samples.

Theoretically, these new metrics also facilitate systematic hyperparameter tuning by deriving expected values when real and generated distributions match. This systematic approach significantly reduces the pitfalls associated with arbitrary selections in previous metrics.

Future Directions

Beyond the immediate impact on generative model assessment, this paper opens several avenues for future research:

• Application to Other Domains: While primarily focused on image generation, these metrics could be adapted for other data types, such as text or audio, where similar fidelity and diversity concerns are present.
• Integration with Unsupervised Learning: These metrics could be integrated into training processes, potentially enabling models that self-correct training paths skewing fidelity or diversity.
• Expanding Embedding Strategies: Further exploration of embedding strategies could lead to enhancements in model evaluation, particularly in domains far removed from pre-training data distributions.

In conclusion, the paper advances the field's understanding of evaluation metrics for generative models by highlighting existing deficiencies and proposing more stable and interpretable alternatives. The density and coverage metrics provide a robust framework for evaluating the fundamental aspects of generative models, contributing to more effective and refined models in practice.