The Vendi Score: A Diversity Evaluation Metric for Machine Learning (2210.02410v2)

Abstract: Diversity is an important criterion for many areas of ML, including generative modeling and dataset curation. However, existing metrics for measuring diversity are often domain-specific and limited in flexibility. In this paper, we address the diversity evaluation problem by proposing the Vendi Score, which connects and extends ideas from ecology and quantum statistical mechanics to ML. The Vendi Score is defined as the exponential of the Shannon entropy of the eigenvalues of a similarity matrix. This matrix is induced by a user-defined similarity function applied to the sample to be evaluated for diversity. In taking a similarity function as input, the Vendi Score enables its user to specify any desired form of diversity. Importantly, unlike many existing metrics in ML, the Vendi Score does not require a reference dataset or distribution over samples or labels, it is therefore general and applicable to any generative model, decoding algorithm, and dataset from any domain where similarity can be defined. We showcase the Vendi Score on molecular generative modeling where we found it addresses shortcomings of the current diversity metric of choice in that domain. We also applied the Vendi Score to generative models of images and decoding algorithms of text where we found it confirms known results about diversity in those domains. Furthermore, we used the Vendi Score to measure mode collapse, a known shortcoming of generative adversarial networks (GANs). In particular, the Vendi Score revealed that even GANs that capture all the modes of a labeled dataset can be less diverse than the original dataset. Finally, the interpretability of the Vendi Score allowed us to diagnose several benchmark ML datasets for diversity, opening the door for diversity-informed data augmentation.

• The paper introduces the Vendi Score as a metric that measures diversity using the exponential of the Shannon entropy of eigenvalues from a similarity matrix.
• The authors demonstrate its broad applicability across domains including molecular and image generative modeling, GAN evaluation, and NLP decoding.
• The study highlights the metric’s enhanced sensitivity to compound feature interactions, providing refined diversity assessments over conventional methods.

The Vendi Score: A Novel Diversity Evaluation Metric in Machine Learning

The paper "The Vendi Score: A Diversity Evaluation Metric for Machine Learning" introduces an innovative approach to assessing diversity within ML models and datasets. Traditional diversity metrics in ML often rely on specific domain constraints or reference datasets, thereby restricting their universal applicability. This paper addresses these limitations by introducing the Vendi Score (VS)—a generalized and adaptable metric derived from concepts in ecology and quantum statistical mechanics.

The Vendi Score is defined as the exponential of the Shannon entropy of the eigenvalues of a similarity matrix derived from a user-specified similarity function. This approach allows for a versatile means of evaluating diversity that can be tailored to various domains and datasets without dependence on external reference datasets or distributions. By employing a user-defined similarity function, the Vendi Score can accommodate different interpretations of diversity that may be most relevant to a given application, whether that is generative modeling, decoding algorithms, or dataset evaluation.

The authors demonstrated the utility of the Vendi Score across several domains: molecular generative modeling, image generative modeling, GAN evaluation, and NLP decoding algorithms. Each application showcased how the Vendi Score could identify nuanced diversity characteristics that are not as readily captured by existing metrics. For example, the paper presents results indicating discrepancies between Vendi Score evaluations and traditional metrics such as IntDiv, particularly highlighting cases where the existing metrics either fail to distinguish or misrepresent diversity.

The theoretical properties of the Vendi Score are rigorously analyzed, showcasing its ability to represent diversity as an "effective number of dissimilar elements" in a sample. It possesses commendable properties such as symmetry, partitioning ability, and sensitivity to sample correlations, which further substantiate its utility as a metric over other traditional methods like IntDiv that often fail to capture diversity arising from compound feature interactions.

Calculating the Vendi Score requires finding eigenvalues of a similarity matrix, a process that typically demands O(n^3) computational complexity. However, the presence of low-dimensional embeddings can significantly optimize this computation to O(d^2n), where d is manageable when embeddings are available. The paper further alludes to the empirical estimator of the kernel entropy, noting its convergence rate, which is favorable and proportional to 1/√n.

In practice, the Vendi Score has broad implications for enhancing diversity evaluation, potentially informing diversity-informed data augmentation strategies, which is crucial when handling limited datasets. Additionally, this facilitates a more accurate diagnosis of potential biases or deficiencies within datasets or models, thus guiding improvements in Bayesian modeling processes and robust ML model training.

In conclusion, this paper presents the Vendi Score as an effective, versatile, and theoretically grounded metric for evaluating diversity across diverse ML applications. Its transparent dependence on user-defined similarity functions positions it as a highly adaptable tool, paving the way for refined diversity assessments across various ML landscapes. This new perspective invites deeper exploration into its application, performance, and further refinements that could evolve its scope and impact within the domain of machine learning research.