Expectation-Adjusted Distinct (EAD) Metric

• Expectation-Adjusted Distinct (EAD) is a refined metric for evaluating diversity in language generation outputs, correcting the length bias of traditional Distinct-n.
• It replaces the raw token count with an expected value based on probability, effectively scaling unique token counts to reflect true diversity.
• Empirical results on datasets like DailyDialog and OpenSubtitles show that EAD correlates better with human evaluations, enabling fairer cross-model comparisons.

Expectation-Adjusted Distinct (EAD) is a refinement of the Distinct-$n$ metric for evaluating diversity in language generation. It was introduced to address the observation that the original distinct score has evident biases that tend to assign higher penalties to longer sequences, even when the underlying generative process is similarly diverse. EAD refines the calculation of distinct scores by scaling the number of distinct tokens based on their expectations, and the reported empirical and theoretical evidence indicates that this adjustment effectively removes the biases existing in the original distinct score while correlating better with human judgment in evaluating response diversity (Liu et al., 2022).

1. Origin and problem setting

Distinct-$n$ is widely used as an automatic metric for evaluating diversity in language generation tasks. In its original form, it computes diversity as the number of unique $n$-grams divided by the total number of $n$-grams in the evaluated sample. The motivation for EAD is the claim that this simple normalization introduces significant scaling issues as the total number of tokens or $n$-grams increases (Liu et al., 2022).

The central problem is length bias. As sequence length increases, the denominator grows, but the numerator grows sub-linearly due to repetition. The result is that Distinct-$n$ can unfairly penalize longer outputs, making them appear less diverse even when their underlying distribution is not less diverse. The same dependence on output length also creates dataset-insensitivity and unstable comparison: the metric can be manipulated by controlling average output length via decoding tricks, and its dependence on output length undermines its validity for comparisons across models or methods (Liu et al., 2022).

The paper situates this issue in both empirical observation and prior psycholinguistics research. It reports that even with consistent word distributions, increasing text length leads to decreasing Distinct-$n$ scores for both natural and artificial datasets, and it describes a systematic linear decline of Distinct-$n$ with sample length across several datasets (Liu et al., 2022).

2. Biases of the original Distinct-$n$

For unigrams, the original metric is written as

$\text{Distinct} = \frac{N}{C},$

where $n$0 is the number of distinct tokens and $n$1 is the total number of tokens (Liu et al., 2022).

The mathematical issue identified for this formulation is that the probability of acquiring new unique $n$2-grams in a larger sample decreases. Consequently, the growth in $n$3 is sub-linear as $n$4 increases. This violates the intuition that diversity should reflect the underlying distribution rather than length. In the summary provided for the paper, this is characterized as a failure to adjust for the “diminishing returns” of discovering new unique items with more sampling from a fixed vocabulary (Liu et al., 2022).

The empirical consequences are direct. Experiments on natural corpora such as OpenSubtitles and on artificial uniform samplings show a steep, systematic drop in Distinct-$n$5 as $n$6 grows. Even when word distributions are fixed, longer samples get lower Distinct-$n$7 scores. Decoding tricks that limit output length can therefore wrongly inflate a system’s perceived diversity. On this account, the original metric quantifies lexical richness, but it is described as unfit for cross-model or cross-sample-length comparisons because it does not neutralize the dependence on sample size (Liu et al., 2022).

A common misconception is that a higher Distinct-$n$8 score always indicates genuinely higher diversity. The argument behind EAD rejects that interpretation when average output lengths differ substantially: in such cases, a shorter output can obtain a higher score without reflecting a more diverse response distribution (Liu et al., 2022).

3. Expectation-based reformulation

EAD replaces the raw token-count denominator with an expectation. The proposal is that the denominator should not be the raw token count, but the expected number of unique tokens or $n$9-grams for a set of that size according to the assumed distribution. This is intended to account for the saturation effect: with increasing sample size, it is harder to observe new unique tokens, so the denominator should reflect the expected distinct count rather than the absolute count (Liu et al., 2022).

Let $n$0 denote vocabulary size, $n$1 the number of unique tokens observed, and $n$2 the total $n$3-gram count. For uniformly sampled tokens, the expected number of unique tokens is given as

$n$4

The Expectation-Adjusted Distinct score is then

$n$5

This formulation scales the observed number of distinct items by the expected number of distinct items under the sampling regime (Liu et al., 2022).

Several properties are highlighted. As $n$6, the denominator approaches $n$7, and thus $n$8. The incremental rate of EAD with respect to finding an additional unique token converges to $n$9, whereas in Distinct-$n$0 it becomes arbitrarily small as $n$1 increases, namely $n$2. This is presented as the formal reason EAD is length-invariant under a given distribution and therefore better reflects true diversity (Liu et al., 2022).

4. Empirical and theoretical support

The empirical case for EAD is based primarily on stronger agreement with human judgments and on stability across length variation. On both DailyDialog and OpenSubtitles, EAD is reported to show higher Pearson, Spearman, and Kendall’s Tau correlations with human-rated diversity scores than the original Distinct-$n$3 (Liu et al., 2022).

For DailyDialog, the reported correlations are Distinct: Pearson $n$4, Spearman $n$5, Kendall’s Tau $n$6; EAD: Pearson $n$7, Spearman $n$8, Kendall’s Tau $n$9. For OpenSubtitles, the reported correlations are Distinct: Pearson $n$0, Spearman $n$1, Kendall’s Tau $n$2; EAD: Pearson $n$3, Spearman $n$4, Kendall’s Tau $n$5 (Liu et al., 2022).

The paper also reports ablation and case analysis across dialogue generation models. In those comparisons, EAD is described as more accurately separating the diversity of response distributions according to human perception. In some cases, models with longer average outputs were wrongly penalized by Distinct, whereas EAD correctly reflects their relative diversity. An additional ablation states that as sample length increases, original Distinct declines linearly, while EAD remains flat (Liu et al., 2022).

The theoretical support is framed in derivative analysis and “Property Discussion” in the appendix. The key claim is that EAD’s scaling reflects the underlying probability space rather than the surface token count. As sample size grows, the change in the original Distinct score per new distinct $n$6-gram shrinks as $n$7, while for EAD it is lower-bounded by $n$8. This is the basis for the claim that EAD removes the length bias in the original metric (Liu et al., 2022).

5. Relation to similarity-aware and adjusted measures

A later paper on similarity-adjusted surprisal places EAD in a broader formal context by contrasting “distinctness” with graded similarity. In that account, EAD, like standard surprisal, treats each entity as fully distinct, with no similarities taken into account. Under identity similarity, similarity-adjusted surprisal reduces to standard surprisal, and the paper describes this as the EAD case in which all alternatives are treated as maximally distinct (Meister et al., 2024).

This connection is conceptually important because it locates EAD at one end of a family of measures. Standard Distinct and EAD assume that alternative tokens or $n$9-grams are either the same or different; similarity-adjusted formulations instead allow graded semantic, syntactic, or orthographic relatedness. This suggests that EAD corrects a specific normalization problem in diversity measurement without introducing graded similarity structure into the metric itself (Meister et al., 2024).

A broader paper on adjusted similarity measures analyzes expectation adjustment in general and emphasizes that adjusted measures are intended to have the property of $n$0 expectation under a null distribution and maximum value $n$1 under maximal similarity. It also shows that when observed data are incorporated into null distributions in inappropriate ways, adjustment can break down, yielding non-positive measures or deterministic zero under statistical standardization (Lippitt et al., 15 Jan 2026). A plausible implication is that EAD belongs to a wider family of expectation-based corrections whose interpretability depends on how the expectation term is defined, even though the EAD paper itself focuses specifically on correcting the length bias of Distinct-$n$2.

6. Usage, estimation choices, and limitations

EAD is presented as practical as well as principled. The paper states that it is efficient to compute and simple to implement, and it provides an example implementation to foster future research (Liu et al., 2022).

The metric is described as broadly applicable not only to dialogue, but also to other text generation tasks such as story generation and summarization where diversity is valued. Its principal practical use is fairer cross-model comparison: by neutralizing length-induced bias, it aims to ensure that apparent progress in diversity reflects genuine gains rather than manipulation of decoding parameters (Liu et al., 2022).

The treatment of vocabulary size is also addressed. The summary notes that the method for vocabulary size estimation is robust to different settings, as long as a consistent value is used across methods; BERT’s vocabulary is given as an example of such a consistent value (Liu et al., 2022).

The limitations stated in the summary are modest but important. For datasets with severe inherent length constraints or atypical distributions, such as Twitter, even EAD may reflect these underlying realities. The practical recommendation is therefore to check length-sensitivity on the data before application. In that sense, EAD is not presented as a universal solution to every diversity-evaluation problem, but as a targeted correction to the specific length bias of the original Distinct-$n$3 metric (Liu et al., 2022).