Penalty-Adjusted Type-Token Ratio (PATTR)

• PATTR is a lexical diversity metric that adjusts the traditional type-token ratio by penalizing deviations from a task-specific target length.
• It enables fair comparison of synthetic texts with varying lengths by reducing the bias toward shorter outputs.
• Experimental results demonstrate that PATTR outperforms baseline metrics like TTR, MATTR, and CR in filtering diverse, context-aware video scripts.

Penalty-Adjusted Type-Token Ratio (PATTR) is a lexical-diversity metric introduced to measure diversity in synthetic text under prompt-influenced length variations. It was proposed for a setting in which prompt engineering changes not only lexical choice but also response length, thereby distorting standard diversity metrics that are strongly length-dependent. PATTR modifies the basic type-token-ratio paradigm by introducing a task-specific target response length, $L_T$, so that responses are evaluated relative to an expected output size rather than rewarded for brevity alone. In the reported experiments, PATTR is positioned against Type-Token Ratio (TTR), Moving-Average TTR (MATTR), and Compression Ratio (CR), and is evaluated as a filtering and ranking criterion for synthetic text generation (Deshpande et al., 20 Jul 2025).

1. Problem setting and motivation

PATTR addresses a well-known normalization problem in lexical-diversity measurement: raw type-token ratios shrink mechanically as text length increases. In the formulation used across the relevant literature, TTR is the ratio of types to tokens, commonly written as $TTR = V/L$, where $V$ is the number of types and $L$ is the number of tokens. The 2024 study on entropy and TTR in gigaword corpora explicitly characterizes TTR as “problematic given its dependence on text length,” noting that its usefulness drops when comparing samples of different sizes (Rosillo-Rodes et al., 2024).

The 2025 PATTR paper frames this problem in synthetic text generation. In its video-script generation setting, different prompt variants induce substantial differences in output length, and conventional metrics such as TTR and CR tend to rank shorter responses as more diverse simply because lexical variety grows sublinearly with length. The paper identifies this as a classic Herdan-Heap effect: as a text gets longer, the number of unique words rises more slowly than the total number of tokens, so the ratio of unique words to total words naturally falls (Deshpande et al., 20 Jul 2025).

A closely related issue had already been identified for dialogue diversity metrics. The 2022 work on Distinct-$n$ argues that the standard distinct score, written as $Distinct = N/C$, penalizes longer sequences because the denominator is raw token count $C$, even when the underlying token-generation distribution is unchanged. That paper describes this as a “higher penalty for longer texts” and replaces the raw length normalization with an expectation-based term (Liu et al., 2022). This provides a direct antecedent for PATTR’s more task-specific length correction.

2. Formal definition and score construction

PATTR is defined as a modification of the standard type-token-ratio framework. In the paper’s per-response setup, a text string is written as $w = \{w_0, \dots, w_{L-1}\}$, where $L$ is the number of whitespace-separated words, and the baseline TTR is the ratio of unique words to total words (Deshpande et al., 20 Jul 2025).

The central modification is the introduction of a length penalty tied to a task-specific target length $L_T$. The paper defines the penalty as

$TTR = V/L$0

and states that the penalty increases linearly with the deviation from $TTR = V/L$1. The surrounding discussion further clarifies that the use of the absolute difference means the penalty applies symmetrically to both shorter and longer texts. The stated goal is not to reward longer responses as such, but to favor responses that are closer to the task-appropriate length while still rewarding lexical variety (Deshpande et al., 20 Jul 2025).

The paper also provides a corpus-level formulation under “Individual Cost Adjusted Diversity.” For each input-output pair $TTR = V/L$2 in a corpus $TTR = V/L$3,

$TTR = V/L$4

and

$TTR = V/L$5

This is contrasted with the plain TTR baseline

$TTR = V/L$6

In this formulation, PATTR retains unique-word count as the numerator while increasing the denominator as the generated response moves away from the target length. A common misconception is that any length-aware metric is simply pro-length or anti-length. The paper explicitly rejects that interpretation: by changing $TTR = V/L$7, PATTR’s correlation with length can be tuned from negative to near-zero to strongly positive, which the authors present as evidence that the metric is calibrated to task requirements rather than to length alone (Deshpande et al., 20 Jul 2025).

3. Relation to TTR, Distinct, MATTR, CR, and expectation-adjusted normalization

PATTR belongs to a broader class of diversity metrics that attempt to correct the length sensitivity of raw ratios. In the lexical-diversity literature, TTR is the canonical example of a measure that is informative but unstable across text lengths. In the generation-evaluation literature, the analogous problem appears in Distinct-$TTR = V/L$8, which is mathematically a type-token ratio over $TTR = V/L$9-grams. The 2022 refinement of Distinct replaces

$V$0

with

$V$1

where the denominator is the expected number of distinct types obtainable from $V$2 draws from a vocabulary of size $V$3. The paper’s argument is that a diversity metric should remain stable when the underlying token distribution is fixed and only length changes; raw Distinct fails this criterion, whereas Expectation-Adjusted Distinct (EAD) is designed to remove the bias toward shorter outputs (Liu et al., 2022).

PATTR differs from EAD in the object of normalization. EAD replaces naive length normalization with an expectation term derived from a reference distribution. PATTR instead introduces a task-specific penalty around a desired response length $V$4. A plausible implication is that both metrics belong to the same normalization family: each preserves a diversity numerator while replacing raw length with a correction intended to remove trivial sequence-length effects.

Relative to existing baselines in the 2025 study, TTR is described as directly biased toward short texts because the denominator is the response length itself. CR is also reported to exhibit length bias, with shorter texts often compressing differently and therefore appearing more diverse for mechanical reasons. MATTR partially alleviates length sensitivity by computing type-token ratio over a moving window, but it does not incorporate a task-specific expected length $V$5. PATTR is presented as distinct because it explicitly uses $V$6 and interprets diversity relative to the desired output length rather than to whatever length happens to be generated (Deshpande et al., 20 Jul 2025).

4. Experimental regime and observed metric behavior

The principal empirical study of PATTR is conducted on a large synthetic corpus generated for a creative writing task of video script generation. The paper reports a corpus of over 20M words generated by seven models from the LLaMA, OLMo, and Phi families: OLMo-2-13B, OLMo-2-7B, Llama-3.1-8B, Llama-3.2-3B, Llama-3.2-1B, Phi-3-med, and Phi-3-mini. Each model generates 12,000 video scripts using a structured prompting scheme with 10 unique instruction values, 10 unique style values, and 120 unique user prompt or topic values. The generation uses temperature sampling with temperature 1.0 and fixed seed 0. The appendix reports that, after cleaning, the final dataset comprises over 100,000 scripts and more than 50 million words total (Deshpande et al., 20 Jul 2025).

The paper’s first diagnostic is a “win rate” analysis for short-response preference. For each prompt configuration, a pool of 10 scripts is generated, and the metric is asked to select the most diverse one. If the selected script falls in the shortest quartile, the metric is treated as length-biased. In the reported summary for OLMo-2-13B, the short-response win rate is 67.33% for CR, 38.33% for MATTR, and 28.75% for PATTR at $V$7, dropping to 0.58% at $V$8 and 0.08% at $V$9. The paper reports similar patterns across the other models and explicitly notes that larger $L$0 values reduce the win rate for short sequences (Deshpande et al., 20 Jul 2025).

A second reported result concerns length correlation. On Llama-3.1-8B responses, the paper shows that $L$1 yields a strong negative correlation, $L$2 yields near-zero correlation, and $L$3 yields a strong positive correlation. This is used to argue that PATTR is not intrinsically aligned with either shortness or longness. Rather, the score is task-calibrated through $L$4, which functions as a control parameter for the desired response regime (Deshpande et al., 20 Jul 2025).

5. Corpus filtering and selection performance

Beyond scoring individual responses, PATTR is evaluated as a filtering criterion for synthetic corpus construction. The paper ranks generated responses by PATTR, MATTR, and CR and then retains the top-10, top-100, or top-1,000 most diverse examples. The resulting subsets are assessed with ROUGE-1/2/L, BLEU, entropy, $L$5-gram diversity, and Wasserstein distance (Deshpande et al., 20 Jul 2025).

Across these evaluations, PATTR is reported as consistently competitive and often superior to the baselines. For top-10 selections, the paper states that across 16 scenarios, PATTR beats CR in 14 and 12 cases and beats MATTR in 15 and 12 cases for top-10 and top-100 selection respectively, with similar strong performance for top-1,000 selection. In the entropy-based analysis, PATTR performs especially well when the desired length is longer; for length ranges such as 350–450 and 550–650 words, it more consistently yields the most diverse corpus. On $L$6-gram diversity, the top-10 set selected by PATTR is described as comparable to MATTR and clearly better than CR, while also achieving the highest POS-based diversity in at least one reported measure (Deshpande et al., 20 Jul 2025).

The most specific distributional comparison uses Wasserstein distance from an ideal uniform-vocabulary distribution. For the selected top-10 outputs, PATTR achieves the lowest Wasserstein distance, $L$7, versus $L$8 for MATTR and $L$9 for CR. In the paper’s interpretation, this indicates that the vocabulary distribution of PATTR-selected outputs is closest to broad lexical coverage (Deshpande et al., 20 Jul 2025).

Practically, the paper proposes PATTR for filtering synthetic generations for downstream use, including instruction tuning, dataset curation, and creative-writing corpora. The stated rationale is that a diversity filter should not inadvertently collect short, under-informative samples merely because they score well under length-biased metrics (Deshpande et al., 20 Jul 2025).

6. Theoretical scaffolding, alternative baselines, and limitations

The theoretical motivation for PATTR is supported by broader work on vocabulary growth. In large corpora, the 2024 entropy-TTR study derives $n$0 from Heaps’ law and shows that entropy and TTR obey a functional relation under Zipf and Heaps scaling. The paper’s general conclusion is that lexical diversity is not well captured by TTR alone, because TTR and entropy are mathematically linked once vocabulary-growth and frequency-distribution laws are taken seriously (Rosillo-Rodes et al., 2024). This suggests that any robust lexical-diversity metric must account for predictable length effects and not merely count observed types.

An alternative theoretical route is offered by the 2018 work on type-token curves, which argues that the usual Heaps’ law $n$1 is less accurate than a logarithmic model derived from a “perfect Zipf distribution.” Its central closed form is

$n$2

and the paper proposes this as a more accurate expected vocabulary-growth curve than a simple power law. The same work extends the model to hapaxes and higher $n$3-legomena and presents an “optimum sample” normalization via

$n$4

The paper does not define PATTR, but it explicitly describes how such equations could support a penalty-adjusted or corrected type-token metric based on expected type growth rather than raw TTR (Davis, 2018).

These theoretical studies also delimit the scope of current adjustment strategies. The entropy-TTR paper uses an orthographic notion of “word,” does not handle homographs separately, studies unigram entropy rather than dependencies, and relies on large-vocabulary asymptotics with an approximate Zipf exponent $n$5. The type-token-curve paper depends on the existence of an optimum sample size at which the corpus behaves as if it follows a perfect Zipf distribution, and it states that fit quality varies by corpus (Rosillo-Rodes et al., 2024, Davis, 2018). For PATTR specifically, the reported evidence is tied to synthetic video-script generation and to task-aligned selection under specified target lengths. A plausible implication is that PATTR should be understood as a calibrated metric for task-constrained generation, rather than as a universal replacement for all lexical-diversity measures.

In that sense, PATTR occupies a specific position in the evolution of diversity metrics. Raw TTR measures lexical variety but is strongly length-sensitive. Expectation-adjusted metrics such as EAD replace naive normalization with an expected distinct-count baseline. PATTR instead introduces an explicit penalty around a target length, making lexical-diversity assessment conditional on the generation objective itself. The unifying principle across these approaches is that diversity measurement becomes more informative when length effects are modeled rather than ignored (Liu et al., 2022, Deshpande et al., 20 Jul 2025).