---
title: 'Vendi Score: Kernel-Based Diversity Metric'
url: https://www.emergentmind.com/topics/vendi-score
type: topic
---

# Vendi Score: Kernel-Based Diversity Metric

The Vendi Score is a general, kernel-based diversity metric designed for quantifying the effective number of distinct elements in a finite sample, with deep connections to ecological diversity indices and quantum statistics. Its foundation is the entropy of the spectrum of a similarity matrix constructed via a user-specified positive semi-definite kernel. Unlike domain-specific or label-dependent metrics, the Vendi Score allows for tunable sensitivity to rare versus abundant types, admits theoretical generalizations, and has become a standard tool for measuring and optimizing diversity across machine learning, computational biology, ecology, and experimental design.

## 1. Formal Definition and Mathematical Foundations

Let $\mathcal{X} = \{x_1, \dots, x_n\}$ be a collection of $n$ items (e.g., images, sequences, trajectories) and $k:\mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}_+$ a user-chosen positive semi-definite kernel with the normalization $k(x_i,x_i) = 1$. Construct the $n \times n$ similarity matrix $K$, where $K_{ij} = k(x_i, x_j)$. For the trace-normalized matrix $\tilde K = K / \mathrm{tr}(K)$, let $\lambda_1,\ldots,\lambda_n$ denote its eigenvalues (so $\sum_{i=1}^n \lambda_i = 1$).

The **Vendi Score of order $q$** (with $q \ge 0$) is defined as:

\[
\mathrm{VS}_q(\mathcal{X};k) = 
\begin{cases}
\left(\sum_{i=1}^n \lambda_i^q \right)^{\frac{1}{1-q}} & \text{if } q\ne 1 \\
\exp\left(- \sum_{i=1}^n \lambda_i \log \lambda_i \right) & \text{if } q=1
\end{cases}
\]

The $q=1$ case recovers the exponential of the Shannon/von Neumann entropy of the spectrum (also equivalent to the Hill number of order 1 in ecology):

\[
\mathrm{VS}_1(\mathcal{X};k) = \exp\left( -\sum_{i=1}^n \lambda_i \log \lambda_i \right)
\]

This effective-number interpretation means that $\mathrm{VS}_q(\mathcal{X};k)$ lies in $[1, n]$, achieving $n$ if all items are mutually orthogonal and $1$ if all are identical [2310.12952][2210.02410].

The kernel function $k$ is central—by selection of $k$ the user defines which differences are considered meaningful.

## 2. Algorithmic Computation and Scalability

The computation of the Vendi Score proceeds as follows:

1. **Compute the $n \times n$ kernel matrix:** $K_{ij} = k(x_i, x_j)$.
2. **Normalize:** $\tilde K = K / \mathrm{tr}(K)$.
3. **Spectral decomposition:** Obtain eigenvalues $\lambda_1,\dots,\lambda_n$ of $\tilde K$.
4. **Aggregate:** Evaluate the relevant $q$-order function as above to compute $\mathrm{VS}_q$.

Pseudocode:
```python
def vendi_score(x, k, q=1):
    n = len(x)
    K = np.array([[k(xi, xj) for xj in x] for xi in x])
    Kbar = K / np.trace(K)
    lambdas = eigvals(Kbar)
    if q == 1:
        entropy = -np.sum(lambdas * np.log(lambdas + 1e-12))
        return np.exp(entropy)
    else:
        return (np.sum(lambdas ** q)) ** (1.0 / (1 - q))
```

**Complexity:** Kernel matrix computation is $O(n^2)$; eigen-decomposition is $O(n^3)$ in general. If high-dimensional embeddings are available (dimension $d \ll n$), one can leverage the low-rank structure for $O(nd^2)$ complexity [2210.02410][2509.02930]. For large-scale problems, efficient approximations via the Nyström method or random projections enable sub-cubic scaling [2410.21719].

## 3. Theoretical Properties and Parameter Interpretation

### Bounds, Invariance, and Interpretability

- **Range:** $1 \le \mathrm{VS}_q(\mathcal{X};k) \le n$.
- **Duplication invariance:** Duplicating an item does not increase VS; redundancy is not counted as diversity.
- **Similarity sensitivity:** The score interpolates between "species richness" ($q \to 0$, counts modes) and "dominant-mode" ($q \to \infty$, counts major clusters).
- **Label-free:** No need for class labels or type frequencies; purely uses sample similarities [2310.12952][2509.22520].

### Tuning via $q$

- $q < 1$: Sensitive to rare clusters or outliers; emphasizes counting "distinct modes."
- $q = 1$: Balances rare and common types (Shannon entropy analog).
- $q > 1$: Emphasizes dominant groups; insensitive to rare types.
- $q \to 0$: Counts the number of nonzero modes (matrix rank).
- $q \to \infty$: Returns $1/\max_i \lambda_i$, i.e., size of largest cluster.

This allows targeted sensitivity in applications—rare variant detection in genomics ($q \ll 1$) or memorization in deep generative modeling ($q \to \infty$) [2310.12952][2509.22520].

## 4. Practical Application Domains

### Machine Learning

- **Generative modeling:** VS distinguishes between generative models that look similar under conventional metrics but differ in sample redundancy and true diversity. It is used to diagnose mode collapse, memorization, and duplication [2210.02410][2502.10828].
- **Active learning:** Vendi Information Gain policies combine informativeness with sample diversity for acquisition, outperforming entropy- or uncertainty-based selection [2505.09007][2509.10390].
- **Self-supervised RL:** VS serves as an intrinsic reward, encouraging agents to discover maximally diverse policies under arbitrary similarity functions [2509.02930].

### Experimental Design and Discovery

- **Quality-weighted VS:** In scientific discovery and experimental design (e.g., active search, BO), VS is extended with a quality multiplier, yielding $q\mathrm{VS} = \mathrm{mean\ quality} \times \mathrm{VS}$. Such criteria flexibly balance exploitation (high score) and exploration (diversity), resulting in 70–170% increases in effective discoveries [2405.02449].

### Computational Biology and Genomics

- **Epidemiology:** VS quantifies the diversity of viral populations in time-resolved sequence data and detects emerging low-diversity clusters indicative of new variants. It is particularly effective for unsupervised, reference-free tracking in large-scale surveillance [2509.22520].
- **Protein/materials universe analysis:** The Vendiscope applies VS with learned weighting to entire scientific datasets, quantifying rarity and identifying near-duplicate and high-diversity instances at scale [2502.10828].

### Generative Model Evaluation

- **Conditional and Information-Vendi:** For generative models conditioned on prompts, VS has been extended to decompose observed diversity into model-induced vs. prompt-induced components, enabling precise analysis of text-to-image, image-to-text, and video generators [2411.02817].

### OOD Detection

- **Vendi Novelty Score (VNS):** Measures the increase in VS when a test sample is added to the in-distribution set. VNS achieves state-of-the-art OOD detection using only samples and similarities, avoiding density estimation [2602.10062].

## 5. Approximation Methods and Convergence

### Truncated and Approximated Versions

- For large $n$, the full spectrum is expensive and may not converge quickly (especially under infinite-dimensional kernels such as RBF/Gaussian).
- The $t$-truncated Vendi Score uses just the top $t$ eigenvalues, requiring only $O(t)$ samples for convergence. Efficient approximations via Nyström and FKEA random-feature methods concentrate tightly around the truncated statistic, with precise finite-sample error bounds [2410.21719].

### Empirical findings

- On finite-dimensional kernels, VS converges rapidly with $n \sim d$, where $d$ is feature dimension.
- On infinite-dimensional kernels, convergence requires truncation and approximation. Nyström and random features provide accurate, scalable solutions.

## 6. Pitfalls, Limitations, and Implementation Guidance

- **Kernel dependence:** The selected similarity function fundamentally determines what diversity is measured.
- **Computational scaling:** Exact VS is $O(n^3)$; scalable SVD/approximation methods are advised for $n>10^3$.
- **Reference-freeness:** VS measures internal diversity; it must be paired with a quality or precision metric to avoid high-diversity but low-quality (e.g., random noise) artifacts [2210.02410].
- **Sensitivity parameter tuning:** The $q$ parameter must be selected according to application needs; $q=1$ is generally robust, but $q<1$ for rare species and $q\gg1$ for memorization/duplication detection [2310.12952][2405.02449].
- **Sparse or imbalanced data:** Imbalanced prevalence affects sensitivity; in extreme cases, the probability-weighted form of VS is recommended [2502.10828].

## 7. Extensions and Theoretical Innovations

- **Conditional, Information, and Entropic Decompositions:** Matrix-based analogs of conditional entropy and mutual information using the Vendi entropy underpin recent advances in prompt disentanglement for generative models, label-free information gain estimation, and active learning [2411.02817][2505.09007].
- **Generalized Information Metrics:** The Vendi Information Gain (VIG) provides an asymmetric, similarity-aware extension of mutual information, reducing to MI when samples are maximally distinct and outperforming MI in sample-based, high-dimensional, and geometric settings [2505.09007].
- **Gradient and Differentiability:** VS-based objectives are differentiable, facilitating their use in gradient-based optimization for experimental design and generative modeling [2502.10828][2405.02449][2406.04551].

---

**Key References**  
- Fundamental metric, theoretical properties, and ML applications: [2210.02410], [2310.12952]  
- RL/skill learning: [2509.02930]  
- Scalability and convergence: [2410.21719]  
- Quality-weighted experimental design: [2405.02449]  
- Genomics/epidemiology: [2509.22520]  
- Prompt-based generation and conditional diversity: [2411.02817]  
- OOD detection: [2602.10062]  
- Vendiscope and large-data applications: [2502.10828]  
- Generalized information theory and VIG: [2505.09007]

Source: https://www.emergentmind.com/topics/vendi-score