Diversity-Aware Scoring in System Evaluation

• Diversity-aware scoring is a method that integrates diversity into metrics to counteract redundancy and promote varied outputs.
• It combines relevance and diversity by balancing traditional scoring with measures that enhance coverage and reduce duplication.
• The approach is applied across language generation, retrieval, active learning, and ensemble selection to improve overall system performance.

Diversity-aware scoring denotes the incorporation of diversity into a metric, discriminant function, acquisition rule, or constrained selector so that systems are not rewarded solely for relevance, accuracy, or average quality. Across language generation, retrieval, active learning, ensemble learning, data curation, diffusion guidance, and committee selection, the common objective is to reduce redundancy, improve coverage, and preserve meaningful variation that would otherwise be suppressed by brevity, mode-seeking, correlated score dimensions, or independent top-$k$ truncation (Liu et al., 2022, Zhu et al., 2015, Friedman et al., 2022).

1. Conceptual basis and recurrent failure modes

A recurring motivation for diversity-aware scoring is that naïve scoring schemes systematically prefer concentrated or redundant solutions. In language generation, the original corpus-level Distinct metric uses

$Distinct=\frac{N}{C},$

where $N$ is the number of distinct tokens or $n$-grams and $C$ is the total count. Because $C$ grows immediately with length while $N$ grows more slowly, longer outputs receive lower scores even when two systems generate from the same underlying distribution. The same summary notes that Distinct can be “gamed” by producing shorter responses, so a model may appear more diverse because it is shorter rather than because it is more varied (Liu et al., 2022).

In retrieval, plain top-$k$ ranking has an analogous weakness. A ranker scores candidates independently and then truncates, so it has no explicit mechanism to stop near-duplicate spans from filling the context window. The “context bubble” formulation for enterprise retrieval-augmented generation reframes context construction as a constrained selection problem over structured evidence, precisely because top-$k$ retrieval causes fragmentation in information graphs, over-retrieval, duplication of content, and insufficient query context including 2nd and 3rd order facets (Khurshid et al., 15 Jan 2026).

In data selection, the same pathology appears as correlation collapse. ODiS reports that directly selecting top-scored data often degrades downstream performance and that sampling from a broader score range is required to recover results. The paper attributes this non-monotonicity to correlated evaluation dimensions: top-scored data cluster in a narrow region of data space, so diversity is overlooked even when multiple metrics are used (He et al., 21 Oct 2025).

Conditional natural language generation introduces another failure mode: the assumption that diversity in references is noise. The distribution-aware evaluation work argues that metrics based on one generated sentence against one or a few references, with max or average aggregation, are inappropriate when reference diversity is semantic signal rather than annotation noise. In image and video captioning, such metrics reward a central caption near the middle of the reference distribution and do not reward matching the broader support of the human caption set (Chan et al., 2022).

These examples suggest a shared diagnosis: relevance-, loss-, or score-only optimization tends to reward central modes, short outputs, or repeated evidence unless diversity is represented explicitly in the scoring object itself.

2. Diversity as an evaluation metric

One major branch of diversity-aware scoring concerns intrinsic evaluation metrics. Expectation-Adjusted Distinct (EAD) replaces Distinct’s length normalization with an expectation-based denominator. Under a uniform upper-bound distribution over a vocabulary of size $V$, the proposed score is

$Distinct=\frac{N}{C},$0

The appendix comparison emphasizes that $Distinct=\frac{N}{C},$1 shrinks with length, while EAD’s incremental scaling stabilizes, and the paper reports better correlation with human diversity judgments on both DailyDialog and OpenSubtitles. On DailyDialog, EAD improves Pearson/Spearman/Kendall’s Tau from $Distinct=\frac{N}{C},$2 to $Distinct=\frac{N}{C},$3; on OpenSubtitles, from $Distinct=\frac{N}{C},$4 to $Distinct=\frac{N}{C},$5 (Liu et al., 2022).

The Vendi Score generalizes diversity evaluation beyond discrete counts by treating diversity as the effective rank of a similarity matrix. Given a positive semidefinite similarity function $Distinct=\frac{N}{C},$6 with $Distinct=\frac{N}{C},$7, a kernel matrix $Distinct=\frac{N}{C},$8, and eigenvalues $Distinct=\frac{N}{C},$9 of $N$0, the score is

$N$1

Its stated advantages are that it is reference-free, label-free, domain-general, and controlled by a user-defined notion of similarity. The paper interprets it as an effective number of distinct elements and shows that it detects duplicates, mode imbalance, and correlated lack of variation more reliably than average pairwise similarity measures in several settings (Friedman et al., 2022).

A later extension introduces a family of Vendi scores indexed by order $N$2,

$N$3

with $N$4 recovering the original Vendi Score. The order parameter controls sensitivity to rare versus common structures: $N$5 emphasizes rare items or rare modes, $N$6 emphasizes common items, and $N$7 depends only on the largest eigenvalue. The paper recommends small $N$8 when diversity or rare-mode sensitivity is the goal, and $N$9 when duplication or memorization is the target of diagnosis (Pasarkar et al., 2023).

A distinct line of work measures diversity through conditional predictability. The Decan metric defines

$n$0

where $n$1 is the residual surprise of the last response conditioned on the prompt and all earlier responses, and $n$2 is a plausibility term derived from the reciprocal geometric-mean perplexity of the responses under the prompt alone. The central claim is that diversity is a property of $n$3, not just of a response set in isolation. On McDiv prompt_gen, the reported result is OCA $n$4, below SentBERT’s $n$5 but above surface statistics such as distinct-$n$6; on OLMo-2-7B, the score drops monotonically across base $n$7 SFT $n$8 DPO $n$9 RLVR/Instruct, which the paper interprets as post-training diversity loss (Khoriaty et al., 1 Jun 2026).

Distribution-aware evaluation for conditional generation takes yet another route: compare small sets of model candidates and references as distributions rather than score one candidate against one reference. The paper introduces Triangle-Rank Metrics and kernel-based metrics such as Frechet BERT Distance and MMD-BERT, arguing that these are more appropriate when reference diversity is signal and when the evaluation target is distribution matching rather than best-reference overlap (Chan et al., 2022).

For text corpora, a standardization study reaches a more operational conclusion. It finds that length is a major confounder, that compression ratio captures much of the same information as slow $C$0-gram overlap homogeneity scores, and that a compact reporting suite should include compression ratio, POS-tag compression ratio, self-repetition of long $C$1-grams, and Self-BLEU, while BERTScore homogenization offers little practical value in the reported summarization experiments (Shaib et al., 2024).

3. Scoring and reranking in retrieval and generation systems

In retrieval and generation pipelines, diversity-aware scoring usually appears not as a standalone metric but as part of a reranking or constrained selection stage. The enterprise “context bubble” framework first scores chunks by lexical relevance, structural priors, and a length penalty,

$C$2

and then accepts a chunk only if it fits the global token budget, respects a per-section budget, and satisfies a hard lexical-overlap gate $C$3. The paper explicitly distinguishes this from MMR-like soft reranking: diversity is implemented as a hard overlap gate plus budget constraints, which the authors describe as more interpretable, auditable, deterministic, and easier to tune in enterprise settings (Khurshid et al., 15 Jan 2026).

Loom incorporates diversity in a lighter-weight manner. Diversity enters retrieval through multiplicative noise $C$4 “to promote diversity across repeated calls,” and it enters final outfit scoring through a statement-item penalty: if an outfit contains more than one “statement” item, the score is penalized by $C$5 for each additional statement piece. The paper is explicit that this is not a formal diversity metric based on embedding spread or attribute entropy; diversity is mostly operationalized through direction-level stylistic differentiation, color variation, and a statement-item penalty, and the authors describe it as practical but shallow (Berlia, 11 May 2026).

KeyScore for video-language understanding also combines relevance with non-redundancy, but through frame-level multimodal signals. It uses semantic similarity $C$6, temporal representativeness $C$7, and contextual drop impact

$C$8

then aggregates normalized components to rank frames. The drop term is explicitly a marginal-contribution measure: a frame is valuable when removing it causes a large decline in video-caption similarity. STACFP complements this by clustering spatio-temporal frame proposals, thereby encouraging visually diverse and temporally distributed candidates before scoring begins (Lin et al., 7 Oct 2025).

SPARKE extends diversity-aware scoring into diffusion sampling. Its prompt-aware mechanism conditions latent-sample diversity on prompt similarity by combining a prompt kernel matrix and a latent kernel matrix through a Hadamard product. The conditional inverse score yields a gradient in which the contribution of a previous sample is weighted by the square of prompt similarity. The paper’s central engineering claim is computational: by using order-2 Renyi Kernel Entropy in latent space, entropy estimation drops from $C$9 to $C$0, and the gradient for the newest sample reduces to $C$1, making prompt-aware diversity guidance feasible over long generation runs (Jalali et al., 11 Jun 2025).

A plausible implication of these systems is that diversity-aware scoring in deployment settings often takes the form of a constrained second-stage selector: retrieval or generation produces plausible candidates, and a downstream scorer enforces coverage, non-redundancy, or stylistic separation under operational budgets.

4. Data selection, active learning, and coverage-oriented subset construction

A second major branch treats diversity-aware scoring as a criterion for choosing training data or unlabeled examples. D$C$2 formalizes instruction-tuning data selection as a two-step process: a scoring step assigns diversity $C$3, difficulty $C$4, and dependability $C$5, and a selection step greedily maximizes their product. Diversity is the distance from a sample to its nearest selected point,

$C$6

with $C$7 defined as cosine distance between model-produced sample embeddings. The next point is chosen by

$C$8

The paper emphasizes that diversity is not used alone: it is multiplicatively gated by a difficulty term designed to discount context-oriented generation diversity and by a dependability score from an external LLM (Zhang et al., 14 Mar 2025).

In active learning, BEMPS replaces error-based acquisition with expected improvement in a strictly proper scoring rule and then adds a batch diversification wrapper. Each unlabeled example is represented not by raw features but by its vector of expected score changes over an estimation pool,

$C$9

The batch algorithm keeps the top $N$0 by scalar acquisition score, runs $N$1-Means on these vectors, and selects one representative per cluster. The paper argues that this yields diversity in expected influence on the scoring objective rather than generic geometric diversity in embedding space (Tan et al., 2021).

ODiS addresses a different coverage problem: correlated rubric dimensions in pre-training data selection. It labels each document with an 11-dimensional score vector, centers the score matrix, applies PCA, retains the top $N$2 principal components subject to an explained-variance threshold, and then trains a RoBERTa-based regressor for each component. Final selection is not based on a single scalar score; instead, the method selects top-scored data within each orthogonal dimension and unions the subsets. The reported inter-dimension overlap is less than $N$3, which the paper presents as evidence that orthogonal decomposition preserves diversity while still selecting high-scoring data (He et al., 21 Oct 2025).

These methods share a coverage-based intuition. Diversity is modeled as nearest-neighbor distance, orthogonal score coverage, or expected-impact separation, and the selected subset is intended to capture the overall distribution rather than a narrow set of hard, high-loss, or highly scored examples.

5. Learning objectives that jointly optimize quality and diversity

Diversity-aware scoring also appears inside learning objectives. In imbalanced binary classification, DAMVI learns a weighted majority vote

$N$4

and optimizes the PAC-Bayesian C-Bound, which depends jointly on Gibbs risk and disagreement

$N$5

The scoring is diversity-aware because classifier weights are learned by maximizing an empirical objective whose numerator reflects weighted accuracy and whose denominator reflects disagreement, after hard positive examples have been upweighted (Goyal et al., 2020).

DivBO makes the same idea explicit in AutoML ensemble search. It introduces a diversity surrogate $N$6 that predicts pairwise diversity between unseen configurations from their validation-set class-probability vectors, builds a temporary pool approximating the current post-hoc ensemble, and combines a performance acquisition rank with a diversity acquisition rank using a time-varying weight. The reported effect is that suggested learners have much higher minimum diversity than BO-ES or RB-ES, and final ensembles have higher pairwise disagreement while achieving the best average ranks on both validation and test error (Shen et al., 2023).

In ranking, structural learning of diverse search results defines a single discriminant function whose score combines relevance features and pairwise diversity features. The model is trained with a structural SVM using a loss based on diversity-correlated evaluation measures such as ERR-IA, $N$7-NDCG, and NRBP. The paper’s central claim is that directly optimizing DCEM and representing diversity as a feature-based scoring function is more flexible than explicit-subtopic methods such as xQuAD or PM-2 (Zhu et al., 2015).

Preference learning provides a different notion of diversity-aware scoring. SCoRa does not add a diversity penalty to items; instead, it treats diversity in the elicited signal itself as useful for score recovery. Ratings provide broad coverage and comparisons provide fine-grained discrimination among top entities. The model learns latent scores from both modalities in a unified generalized Bradley–Terry framework, and the paper proves monotonicity and robustness properties for the MAP estimator while reporting that mixed feedback can outperform ratings alone or comparisons alone when accurate ordering of top entities is critical (Fageot et al., 8 Feb 2026).

These formulations indicate that diversity-aware scoring is not restricted to output reranking. It can be embedded directly into the learning target, the search acquisition, or even the type of supervision deemed informative.

6. Constrained selection, trade-offs, and scope conditions

Several works pair diversity optimization with explicit guarantees or external constraints. DACS constructs a diverse selection set while controlling false discovery rate in a model-free fashion. Diversity is a user-chosen functional $N$8, with examples including a Sharpe ratio, a Markowitz objective, and an underrepresentation index. The selector maximizes diversity subject to a self-consistency constraint involving valid e-values,

$N$9

and chooses the stopping time by an optimal-stopping argument over expected diversity rewards. The paper emphasizes that this is not post hoc pruning of a conformal set; diversity is optimized inside the certified selection rule itself (Nair et al., 19 Jun 2025).

Committee-election models make the trade-off even more explicit. One model, MAX-$k$0-DSCR, maximizes a diversity index $k$1 subject to a score lower bound $k$2; the other, MAX-$k$3-DSAT, maximizes diversity subject to a minimum satisfaction constraint for every voter. The paper studies Richness, Simpson, Shannon entropy, and a new Lexicographic Counting Index, and reports that relaxing the score by $k$4 can improve average achieved diversity by about $k$5–$k$6 percentage points of the optimal diversity, while a $k$7 score reduction yields optimal diversity in at least $k$8 of instances (Böhm et al., 11 Feb 2026).

The surveyed literature also stresses scope conditions and limitations. EAD is presented as especially suitable for dialogue generation when systems produce different lengths, but the paper warns that on domains with strong length constraints or unusual length-generation behavior, such as Twitter, EAD may still decline with length and may be less appropriate for comparing longer outputs (Liu et al., 2022). The Vendi line emphasizes that diversity scores depend critically on the similarity function and are not quality metrics: random-noise samples can obtain high diversity under an inappropriate kernel (Friedman et al., 2022). Loom states directly that its diversity control is not learned and is not grounded in a principled geometric diversity metric (Berlia, 11 May 2026). SPARKE reports the usual diversity–fidelity tradeoff as the diversity guidance scale increases (Jalali et al., 11 Jun 2025). DACS notes a diversity–power tradeoff: more balanced or dissimilar selections may require fewer accepted items (Nair et al., 19 Jun 2025).

Taken together, these results suggest that diversity-aware scoring is most stable when diversity is not optimized in isolation. The dominant pattern is to couple diversity with a second criterion—human correlation, relevance, fidelity, dependability, score floors, FDR control, or voter satisfaction—so that variety is rewarded without collapsing quality, validity, or task utility.