2000 character limit reached
Revisiting Precision and Recall Definition for Generative Model Evaluation (1905.05441v1)
Published 14 May 2019 in cs.LG, cs.CV, and stat.ML
Abstract: In this article we revisit the definition of Precision-Recall (PR) curves for generative models proposed by Sajjadi et al. (arXiv:1806.00035). Rather than providing a scalar for generative quality, PR curves distinguish mode-collapse (poor recall) and bad quality (poor precision). We first generalize their formulation to arbitrary measures, hence removing any restriction to finite support. We also expose a bridge between PR curves and type I and type II error rates of likelihood ratio classifiers on the task of discriminating between samples of the two distributions. Building upon this new perspective, we propose a novel algorithm to approximate precision-recall curves, that shares some interesting methodological properties with the hypothesis testing technique from Lopez-Paz et al (arXiv:1610.06545). We demonstrate the interest of the proposed formulation over the original approach on controlled multi-modal datasets.
- Wasserstein generative adversarial networks. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 214–223, International Convention Centre, Sydney, Australia, 06–11 Aug 2017a. PMLR. URL http://proceedings.mlr.press/v70/arjovsky17a.html.
- Wasserstein gan. arXiv preprint arXiv:1701.07875, 2017b.
- Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=B1xsqj09Fm.
- Generative adversarial nets. In Advances in neural information processing systems, pp. 2672–2680, 2014.
- Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626–6637, 2017.
- Globally and locally consistent image completion. ACM Transactions on Graphics (TOG), 36(4):107, 2017.
- Quantitatively evaluating gans with divergences proposed for training. arXiv preprint arXiv:1803.01045, 2018.
- Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=Hk99zCeAb.
- Auto-encoding variational bayes. Proceedings of the International Conference on Learning Representations (ICLR), 2014.
- Word translation without parallel data. In International Conference on Learning Representations (ICLR), 2018. URL https://openreview.net/forum?id=H196sainb.
- Pacgan: The power of two samples in generative adversarial networks. In Advances in Neural Information Processing Systems, pp. 1498–1507, 2018.
- Revisiting classifier two-sample tests. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
- Are gans created equal? a large-scale study. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31, pp. 698–707. Curran Associates, Inc., 2018.
- Linking losses for density ratio and class-probability estimation. In International Conference on Machine Learning, pp. 304–313, 2016.
- Which training methods for gans do actually converge? In International Conference on Machine Learning (ICML), pp. 3478–3487, 2018.
- Unrolled generative adversarial networks. 2017.
- Deep face recognition. In British Machine Vision Conference, 2015.
- Segan: Speech enhancement generative adversarial network. InterSpeech, 2017.
- A norm for signed measures. application to non local transport equation with source term. 2017.
- Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR, abs/1511.06434, 2015. URL http://arxiv.org/abs/1511.06434.
- Generative adversarial text to image synthesis. Proceedings of the International Conference on Machine Learning (ICML), 2016.
- Assessing generative models via precision and recall. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31, pp. 5234–5243. Curran Associates, Inc., 2018.
- Improved techniques for training gans. In Lee, D. D., Sugiyama, M., Luxburg, U. V., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 29, pp. 2234–2242. Curran Associates, Inc., 2016.
- Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818–2826, 2016.
- Detecting overfitting of deep generative networks via latent recovery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.