---
title: Diversity (β-Recall) in Generative Models
url: https://www.emergentmind.com/topics/diversity-beta-recall
type: topic
---

# Diversity (β-Recall) in Generative Models

Diversity ($\beta$-Recall) quantifies the extent to which a generative model covers the modes or the support of the target data distribution. In the context of generative modeling, $\beta$-Recall is a principled metric that measures the fraction of the real data manifold captured by the generated samples, and is thus a canonical evaluation for diversity or mode coverage. It is widely employed in the assessment of both image and text generators, serves as the recall axis in two-dimensional precision–recall frontiers, and is essential to diagnosing mode dropping or coverage defects even when global metrics such as FID are favorable [1904.06991][2405.01611][2309.01590].

## 1. Formal Definitions and Theoretical Foundations

### Standard Definition

Given $N$ real samples $X = \{x_i\}_{i=1}^N$ from the reference distribution and $M$ generated samples $Y = \{y_j\}_{j=1}^M$, and a fixed embedding $f(\cdot)$, the $\varepsilon$-Recall is defined as
\[
\mathrm{Recall}(\varepsilon) = \frac{1}{N} \sum_{i=1}^N I_{\varepsilon}(x_i; Y),
\]
where
\[
I_\varepsilon(x_i; Y) = 
\begin{cases}
1, & \min_{y_j \in Y} \|f(x_i) - f(y_j)\| \leq \varepsilon, \\
0, & \text{otherwise}.
\end{cases}
\]
Sweeping $\varepsilon$ yields the recall curve $R(\varepsilon)$. The $\beta$-Recall is operationalized in two forms:
- **Fixed-coverage $\beta$-Recall:** For fixed $\beta \in (0,1)$, find the smallest $\varepsilon_\beta$ such that $R(\varepsilon_\beta) \geq \beta$; report either $\varepsilon_\beta$ or simply note $\beta$ is achieved at this scale.
- **Area under curve (AUC) $\beta$-Recall:** Aggregate recall over all scales, e.g.
\[
\mathrm{AUC} = \int_0^{\varepsilon_{\max}} R(\varepsilon) w(\varepsilon) d\varepsilon,
\]
where $w(\varepsilon)$ can be uniform [1904.06991].

### Precision–Recall Curve Theory

The unifying formalism [Simon et al. 2019][2405.01611][2305.18910] parameterizes the precision–recall (PR) frontier between distributions $P$ (real) and $Q$ (model) by a scalar $\lambda > 0$,
\[
\alpha_\lambda(P, Q) = \int \min(\lambda p(x), q(x)) dx, \qquad \beta_\lambda(P, Q) = \frac{\alpha_\lambda(P, Q)}{\lambda},
\]
with $(\beta_\lambda, \alpha_\lambda)$ tracing out the Pareto-optimal fidelity–diversity trade-off. Here, $\beta_\lambda$ is the $\beta$-Recall at trade-off parameter $\lambda$.

## 2. Practical Estimation and Computational Methodology

There are two dominant empirical paradigms for estimating $\beta$-Recall:

### Nonparametric kNN-based Estimation

- **Feature Construction:** Embed both real and generated data in a semantic feature space (e.g., Inception-V3, VGG-16 for vision; GPT2/PCA for text).
- **kNN Support Estimation:** For each $x_i$, find its minimum distance to the generated set in feature space. $\varepsilon$ is swept or set to the $k$th-nearest neighbor's distance. For the recall curve, either sweep $k$ or $\varepsilon$ [1904.06991][2402.10693][2306.09618].
- **Computation:** 
  - For $\beta$-Recall at fixed $\beta$, compute $R(\varepsilon_k)$ for a grid of thresholds.
  - For PR curve estimation, split data into train/validation sets, fit classifiers, and compute $(\beta_\lambda, \alpha_\lambda)$ consistent with the PRD-curve theory [2405.01611].
- **Hyperparameters:** Number of samples ($N, M$), $k$, embedding $f$, grid over $\varepsilon$ or $\lambda$ [1904.06991][2402.10693].

### Probabilistic/Ball-based and Kernel Estimation

- **P-recall:** Rather than hard thresholding, $P$-recall (or "Probabilistic Recall") assigns a soft kernel $p_{ij} = \max\{0,1 - \|x_i - y_j\|/R\}$ between real and generated pairs, compositing all contributions for each $x_i$
\[
\mathrm{P\mbox{-}recall} = \frac{1}{N} \sum_{i=1}^N \Bigl[ 1 - \prod_{j=1}^M (1 - p_{ij}) \Bigr]
\]
and $R$ is a global scale set by average kNN distance among fakes [2309.01590]. This method is more robust to outliers and is sensitive to the extent and density of the generated distribution.


## 3. Interpretations, Trade-offs, and Diverse Contexts

$\beta$-Recall cleanly operationalizes **diversity** as the fraction of reference instances that lie inside the estimated support of the model distribution. High $\beta$-Recall across scales implies generative coverage: broad mode coverage and insensitivity to mode dropping. This is in contrast to precision, which corresponds to sample fidelity or quality [1904.06991][2405.01611][2402.10693].

By varying the parameter $\varepsilon$ (or $\lambda$ as the PR curve parameter), one can dial trade-offs: small $\varepsilon$ (or small $\lambda$) yields stricter matches, favoring high precision and selectivity, while large values relax coverage and favor recall.

**Fixed $\beta$-Recall** is interpretable as the minimal scale required to cover a desired fraction of real data modes. **AUC $\beta$-Recall** balances recall across all scales and can serve as a summary score.

In language modeling, analogous kNN and $\beta$-scaled metrics map to the distinctiveness or paraphrase diversity of generations, extending recall-style evaluation to open-ended text [2303.10527][2402.10693].

## 4. Failure Modes, High-Dimensional Effects, and Remedies

### High-Dimensional Asymmetry

In high-dimensional regimes, standard kNN-based $\beta$-Recall degenerates due to the curse of dimensionality. It may saturate at 1 when the model support contains the real data manifold, and 0 just outside it, regardless of actual overlap, thereby failing to capture meaningful gradations in diversity [2306.09618]. This emergent asymmetry leads to misinterpretations: e.g., small shifts of the generative support past the real data manifold’s boundary can cause $\beta$-Recall to precipitously drop or rise.

The **symmetric Recall** $\mathrm{symRecall}(P, Q) = \min\{\mathrm{Recall}(P, Q), \mathrm{cRecall}(P, Q)\}$, where $\mathrm{cRecall}$ uses real data to define the support and checks for covered generated points, restores symmetry and validity in high-dimensional tests [2306.09618].

### Outlier Sensitivity and Robustness

kNN-based $I$-Recall is susceptible to sample outliers: a single outlier can expand coverage radii, falsely inflating recall. Probabilistic or kernel-based P-recall mitigates this by using soft membership and global radii, so outliers receive minimal weight [2309.01590].

### Embedding Dependence

$\beta$-Recall is sensitive to the choice of feature embedding. Changing $f(\cdot)$ can rescale distance thresholds and thus alter the absolute $\varepsilon$ values, though relative comparisons between models remain robust if a consistent embedding is used [1904.06991][2402.10693].

## 5. Applications and Extensions

### Generative Model Evaluation

$\beta$-Recall is integral to the evaluation of GANs, flows, and diffusion models. Comparing the recall and precision axes exposes the full quality–diversity spectrum. For example, mode dropping manifests as high precision but low recall; overdispersed or low-quality outputs yield the opposite. Complete PR curves reveal more nuanced trade-offs than scalar FID scores [2405.01611][2305.18910][1904.06991].

### Direct Optimization in Model Training

Recent work operationalizes $\beta$-Recall as a direct optimization target. Precision–Recall Divergence ($D^{PR}_\lambda$), a one-parameter $f$-divergence family, admits minimization via adversarial training or variational estimation to explicitly steer generators toward desired regions on the PR frontier [2305.18910]. Algorithms can target enhanced diversity (recall, small $\lambda$) or fidelity (precision, large $\lambda$), with explicit and tunable trade-off.

### Domain-Specific Instantiations

- **Language modeling:** $\beta$-Recall quantifies distinct paraphrastic or pattern coverage ("d-recall") as in [2303.10527]. Here, the setwise recall is the ratio of distinct pattern types generated to the total in the gold corpus.
- **Conformal selection:** In conformal selection and candidate diversity (as in DACS), $\beta$-Recall appears in the F$_\beta$-Recall score: $\mathrm{F}_\beta=(1+\beta^2) |S\cap H| / (\beta^2 |H| + |S|)$, trading off diversity against set size under FDR constraints [2506.16229].
- **Large language models:** Adapted to text generation, recall quantifies how much of the reference embedding support is covered, with $\beta$-scaling applied to radii for trade-offs [2402.10693].

## 6. Limitations, Recommendations, and Best Practices

- Consistent embeddings are mandatory across model comparisons.
- Report both precision and $\beta$-Recall curves (or AUCs); scalar summaries (e.g., minimum radius $\varepsilon_\beta$, F$_{\beta}$-score, or PR frontier area) can collapse information but should not replace full curves [2405.01611][1904.06991].
- Use large $N, M$ for stable estimation; $k$ values around 4 or $k = \sqrt{N}$ balance local and global sensitivity.
- Outlier robustness: prefer probabilistic recall or symmetric Recall in high-dimensional spaces [2309.01590][2306.09618].
- In language applications, augment evaluations with both pattern diversity (d-recall/$\beta$-Recall) and exhaustiveness (e-recall) [2303.10527].

## 7. Comparative Table of $\beta$-Recall Formulations

| Reference        | Definition / Key Formulation                                      | Notable Context      |
|------------------|------------------------------------------------------------------|----------------------|
| [1904.06991]     | Fraction of real samples within $\varepsilon$-ball of a generated sample; AUC or fixed-$\beta$ | Image GANs, StyleGAN, BigGAN |
| [2405.01611]     | $\beta_\lambda = \alpha_\lambda/\lambda$ from PRD curve          | Universal PR analysis |
| [2309.01590]     | P-recall, probabilistic kernel over all model samples             | Outlier-robust diversity |
| [2306.09618]     | Symmetric recall: $\min\{\mathrm{Recall}, \mathrm{cRecall}\}$    | High-dimensional regime |
| [2303.10527]     | d-recall: fraction of distinct covered template types             | Information extraction |
| [2506.16229]     | F$_\beta$-Recall: $(1+\beta^2)|S\cap H| / (\beta^2 |H|+|S|)$     | Conformal selection   |
| [2402.10693]     | Fraction of reference support inside generated kNN balls, with optional $\beta$-scaling | LLMs, text diversity |

## References

- Improved Precision and Recall Metric for Assessing Generative Models [1904.06991]
- Unifying and extending Precision Recall metrics for assessing generative models [2405.01611]
- Precision-Recall Divergence Optimization for Generative Modeling with GANs and Normalizing Flows [2305.18910]
- Probabilistic Precision and Recall Towards Reliable Evaluation of Generative Models [2309.01590]
- Emergent Asymmetry of Precision and Recall for Measuring Fidelity and Diversity of Generative Models in High Dimensions [2306.09618]
- Two Kinds of Recall [2303.10527]
- Diversifying Conformal Selections [2506.16229]
- Exploring Precision and Recall to assess the quality and diversity of LLMs [2402.10693]

Source: https://www.emergentmind.com/topics/diversity-beta-recall