How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models (2102.08921v2)

Abstract: Devising domain- and model-agnostic evaluation metrics for generative models is an important and as yet unresolved problem. Most existing metrics, which were tailored solely to the image synthesis setup, exhibit a limited capacity for diagnosing the different modes of failure of generative models across broader application domains. In this paper, we introduce a 3-dimensional evaluation metric, ($\alpha$-Precision, $\beta$-Recall, Authenticity), that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample- and distribution-level diagnoses of model fidelity and diversity. We introduce generalization as an additional, independent dimension (to the fidelity-diversity trade-off) that quantifies the extent to which a model copies training data -- a crucial performance indicator when modeling sensitive data with requirements on privacy. The three metric components correspond to (interpretable) probabilistic quantities, and are estimated via sample-level binary classification. The sample-level nature of our metric inspires a novel use case which we call model auditing, wherein we judge the quality of individual samples generated by a (black-box) model, discarding low-quality samples and hence improving the overall model performance in a post-hoc manner.

• The paper proposes a novel three-dimensional metric system that assesses synthetic data fidelity, diversity, and generalization.
• It employs hyperspherical data embeddings with one-class neural networks to compute α-Precision and β-Recall, enhancing diagnostic accuracy.
• Experimental results demonstrate effective detection of mode collapse and overfitting, offering actionable insights for generative model auditing.

Evaluation and Auditing of Generative Models: A Quantitative Approach

The paper "How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models" presents a novel metric system designed to evaluate generative models. This approach effectively addresses the limitations seen in previous domain-specific measures. The proposed metric framework enables model-agnostic evaluation and emphasizes three critical aspects: fidelity, diversity, and generalization of synthetic data. These components promise enhanced diagnosis of generative model performance across various applications.

Core Contributions

The authors introduce a three-dimensional metric system composed of $\alpha$-Precision, $\beta$-Recall, and Authenticity. This framework enables a detailed assessment of generative models:

1. Fidelity ($\alpha$-Precision): This metric quantifies how closely the synthetic data aligns with real samples, emphasizing the generation of realistic samples.
2. Diversity ($\beta$-Recall): It measures whether the synthetic data captures the full variability of the real dataset, ensuring that models do not miss out on diverse data points.
3. Generalization (Authenticity): This metric evaluates whether generative models are merely memorizing the training data or generating new, unseen samples.

The paper extends standard precision-recall analysis by incorporating minimum volume sets, offering a more granular evaluation by focusing on probability density measures rather than solely on data supports.

Methodological Innovation

A significant methodological innovation is the use of hyperspherical data embeddings for computing the $\alpha$-Precision and $\beta$-Recall metrics. The authors employ one-class neural networks to project real data into hyperspheres, streamlining the estimation of $\alpha$- and $\beta$-supports and thus, improving diagnostics for generative models' failure modes.

Additionally, the implementation of the authenticity measure as a hypothesis test highlights the potential of integrating statistical measures for assessing overfitting in generative models, especially important in privacy-sensitive applications.

Experimental Insights and Use Cases

Empirical evidence is provided through various experiments, demonstrating the applicability and effectiveness of the proposed metrics:

• COVID-19 Synthetic Data: Evaluations demonstrate the metric's capability to rank generative models by their predictive utility on synthetic data. Notably, the ADS-GAN model exhibited superior performance, aligning with ground-truth rankings based on predictive accuracy.
• Synthetic Data Auditing: The authors showcase how sample-level auditing can enhance synthetic dataset quality without modifying the generative model, a particularly useful feature for ensuring reliable model outputs.
• Mode Dropping in MNIST: The metrics exposed common failure modes such as mode collapse by revealing diversity issues through the $\beta$-Recall score, underscoring the system's diagnostic power.
• Re-evaluation of Time-Series Generative Models: The practical value of these metrics was further proven in a real-world competition context, where they provided nuanced insights into model authenticity.

Future Implications and Research Directions

The introduction of this metric system suggests potential future developments in the evaluation of generative models. The ability to disentangle generative performance into fidelity, diversity, and generalization provides comprehensive insights beneficial for model selection and improvement. This approach could extend into evaluating fairness in synthetic datasets and further refine privacy-preserving generative modeling techniques.

Furthermore, by allowing for the adjustment of model outputs post hoc, the metrics could facilitate advances in applications like data augmentation, domain adaptation, and beyond. The consideration of density-matching beyond mere support overlap could lead to better integration of synthetic data into various AI systems, enhancing their reliability and applicability across domains.

In conclusion, the paper offers a substantial contribution to generative model evaluation. By presenting a robust framework that aligns closely with practical requirements, it facilitates improved performance diagnostics and model auditing, thus pushing the frontier of synthetic data evaluation.