Vendi Information Gain (VIG) Overview

• VIG is an information-theoretic measure that computes entropy from kernel eigenvalues to quantify how observing a variable reduces uncertainty.
• It overcomes mutual information limitations by integrating sample similarity, enabling robust, density-free analysis in high-dimensional domains.
• Its asymmetric design supports applications in causal inference, active learning, and generative modeling in fields like cognitive science and epidemiology.

Vendi Information Gain (VIG) is an information-theoretic measure designed to overcome limitations of traditional mutual information by incorporating sample similarity directly into entropy computations. VIG quantifies how much uncertainty about a variable is reduced upon observing another, but—crucially—operates on sample sets via kernel-based similarity metrics. This enables robust, flexible use in high-dimensional scientific and machine learning domains where density-based mutual information estimation is challenging or impractical.

1. Formal Definition and Entropic Foundations

VIG is a sample-based information measure that utilizes the Vendi entropy—a generalization of classical entropy constructed from the eigenvalues of a kernel similarity matrix. For a dataset $D = \{\theta_1, \theta_2, \ldots, \theta_n\}$ and a positive semi-definite kernel $k$ (with $k(\theta, \theta) = 1$), one computes the kernel matrix $K_{ij} = k(\theta_i, \theta_j)$. The Vendi score for order $q$ is then

$$\operatorname{VS}_q(D; k) = \exp\left(\frac{1}{1-q} \log \sum_{i} (\bar{\lambda}_i)^q \right)$$

where $\bar{\lambda}_i$ are normalized nonzero eigenvalues of $K$.

The Vendi entropy, the logarithm of the Vendi score, is given by: $$H_V(D; q) = \frac{1}{1-q} \log\left( \sum_{i} (\bar{\lambda}_i)^q \right)$$ VIG quantifies information gain as the reduction in Vendi entropy when conditioning on another variable $y$: $$\operatorname{VIG}(\theta, y; q) = H_V(D; q) - \mathbb{E}_y[H_V(D_y; q)]$$ where $D_y$ denotes the conditional sample set given $y$.

For the case where samples are maximally dissimilar and $q=1$, Vendi entropy reduces to the Shannon entropy, ensuring that VIG generalizes and recovers mutual information (MI): $$\operatorname{VIG}(\theta, y; q=1) = H(\theta) - H(\theta|y) = I(\theta; y)$$ If the kernel encodes similarity (not the identity), VIG properly downgrades information from similar samples.

2. Limitations of Mutual Information and Motivation for VIG

MI is widely used but encounters several critical shortcomings in modern machine learning and scientific analysis:

• Dependency on tractable density estimation: MI requires explicit probabilistic models and tractable entropy, which become infeasible in high-dimensional or nonparametric settings.
• Insensitivity to sample similarity: MI treats all outcome classes as equally distinct; realistic data often exhibits continuous similarity that MI ignores.
• Symmetry: By nature, MI is symmetric ($I(X; Y) = I(Y; X)$), which can obscure directional information flow in applications such as causal inference or response modeling.

VIG directly rectifies these deficits:

• Sample-based and kernel-driven: Requires only samples and a similarity function, not densities.
• Naturally incorporates similarity: The kernel structure embeds pairwise similarities, ensuring less “gain” for confusable or closely related outcomes.
• Asymmetric: VIG quantifies the reduction in entropy for one variable upon observing another, aligning with the semantics of information gain in causal and predictive directions.

3. Methodological Implications and Computation

VIG computation proceeds by:

• Selecting a kernel for the data (e.g., Gaussian, cosine, or custom, subject to $k(\theta, \theta) = 1$).
• Constructing the empirical kernel matrix and obtaining its spectrum.
• Evaluating Vendi entropy on both marginal (unconditional sample set) and conditional (given each observation) sample sets.
• Taking differences to yield VIG.

This method can be performed efficiently on finite sample sets, and naturally supports extensions such as quality weights, truncated eigenspectra (for stable convergence in infinite-dimensional kernels (Ospanov et al., 29 Oct 2024)), and matrix-based conditioning (e.g., for prompt-conditioned generative modeling (Jalali et al., 5 Nov 2024)).

4. Applications in Science and Machine Learning

VIG features prominently in multiple domains:

Domain	VIG Role	Key Advantage
Cognitive Science	Models response times to stimuli	Captures confusability, matches observed behavior
Epidemiology	Level-set estimation, hotspot detection	Robust active acquisition, especially for high-dim.
Active Learning	Batch selection for labeling	Improves accuracy, maximizes dataset-wide diversity
Generative Models	Evaluates diversity, information alignment	Decomposes prompt-induced vs model-induced diversity

In cognitively motivated settings, VIG’s sensitivity to similarity enables direct mapping between distinguishability and information gain, outperforming MI-based alternatives for modeling human responses. In data acquisition (e.g., level-set estimation, disease spread), VIG provides principled sample selection when MI is intractable due to dimensionality.

Active learning with VIG selects batches not solely on local uncertainty but on their global impact on model prediction entropy, yielding more informative and diverse labeled sets. In generative modeling, conditional and information Vendi scores allow for precise evaluation of internal model diversity and alignment with prompts.

5. Theoretical Properties and Generalizations

VIG exhibits several desirable theoretical properties:

• Generalizes MI: Reduces to MI for orthogonal kernels or maximally dissimilar samples.
• Similarity-aware: Kernel structure ensures that VIG is a measure of both informativeness and variety.
• Boundedness and additivity: VIG is bounded between zero (no gain) and the initial entropy; additive over independent variables.
• Practical estimation: Only requires a sample set, not explicit distributions, making it robust to data limitations and high-dimensional feature spaces.

Extensions include:

• Quality-weighted diversity (qVS): Combines average performance and diversity for experimental design (Nguyen et al., 3 May 2024).
• Conditional and information Vendi scores: Enables decomposing total diversity into internal (model) and external (prompt/text) sources (Jalali et al., 5 Nov 2024).
• Truncated spectrum for convergence: Ensures reliable finite-sample estimation in settings with high feature dimension (Ospanov et al., 29 Oct 2024).

6. Impact, Limitations, and Future Perspectives

The introduction of VIG as an alternative to MI marks a significant extension of traditional information theory, particularly for applications where MI's mathematical definition or practical estimation falters. VIG’s flexibility allows it to inform data-driven discovery in fields ranging from cognitive modeling and epidemiology to ecology—where analyses depend critically on mixture, redundancy, and diversity among samples.

A notable limitation is the necessity of kernel selection; the informativeness of VIG depends on the appropriateness of the similarity function for the domain problem. Further, numerical stability in very high-dimensional data may require spectrum truncation or approximation schemes.

Future research directions include:

• Dynamic tuning of Vendi score hyperparameters (order $q$, kernel bandwidth).
• Integration with Bayesian and frequentist frameworks for more general uncertainty quantification.
• Exploration of VIG-based policies for experimental design and scientific search, especially in “active” learning or discovery contexts.

7. Summary Table: VIG versus Mutual Information

Criterion	Mutual Information (MI)	Vendi Information Gain (VIG)
Computation	Density-based, Shannon entropy	Sample-based, kernel Vendi entropy
Similarity	Ignores sample similarity	Directly incorporates similarity
Symmetry	Symmetric ($I(X; Y) = I(Y; X)$)	Generally asymmetric
Dimensionality	Intractable in high dimensions	Robust via kernel sampling
Applications	Limited in some ML, science	Active learning, data acquisition, cognitive science

Vendi Information Gain provides a mathematically general, sample-based alternative to mutual information, yielding both practical and theoretical advances for quantifying information gain in contemporary science and machine learning (Nguyen et al., 13 May 2025, Ospanov et al., 29 Oct 2024, Jalali et al., 5 Nov 2024, Nguyen et al., 3 May 2024, Nguyen et al., 12 Sep 2025).