Testing the Variety Hypothesis

• The paper introduces multi-label fine-tuning for language models, reducing total variation distance to better match human continuation distributions.
• It validates the variety hypothesis in computational dialectology by demonstrating stable cross-register similarity metrics across geo-referenced corpora.
• It presents a finite-sample testing framework in algebraic geometry that uses semialgebraic decision procedures to assess proximity to real algebraic varieties.

“Testing the Variety Hypothesis” names several distinct research programs in which “variety” is treated not as a single canonical target but as a structured object that must be recovered, approximated, or validated from data. In recent literature, the term appears in at least three technically different senses: as the requirement that a LLM’s next-word distribution match the full empirical distribution of human continuations for each context (Groot et al., 22 Sep 2025); as the expectation that web and tweet corpora geo-referenced to the same country exhibit stable similarity and therefore reliably represent the same language variety (Dunn, 2021); and as the problem of deciding whether a probability measure on the unit disk is supported near a real algebraic variety of fixed dimension and bounded degree (Lerario et al., 22 Jul 2025). The shared phrase therefore does not denote a single theorem or benchmark, but a family of domain-specific hypotheses about structured variability, empirical support, and testability.

1. Terminological scope

Across these three literatures, “variety” denotes different mathematical or linguistic objects, and the corresponding “test” changes accordingly.

Domain	Variety hypothesis	Primary criterion
Next-word prediction	$q_\theta(w\mid c_i)\approx p(w\mid c_i)$ for all $c_i$	Total Variation Distance
Computational dialectology	similarity between web and tweets should be stable across all languages and countries	Spearman’s $\rho$ on frequency profiles
Real algebraic geometry	decide whether $\mu$ is supported near $Z\in H(n,k,d)$	squared-distance risk and sample complexity

In “Learning to vary,” Groot et al. define the hypothesis at the level of conditional next-word distributions. If $C=\{c_1,\dots,c_N\}$ is a set of prefixes, $W_i=\{w_{i1},\dots,w_{iM_i}\}$ the human continuations observed for $c_i$, $p(w\mid c_i)$ the empirical human distribution, and $q_\theta(w\mid c_i)$ the model distribution, the requirement is

$c_i$0

In Dunn’s corpus-based work, the hypothesis is instead that “if each source (web vs. tweets) adequately represents a single underlying language variety (e.g. New Zealand English), then the similarity between these sources should be stable across all languages and countries.” In the algebraic-geometric setting, the object is a real algebraic set $c_i$1, and the task is a binary decision problem over probability measures, tolerance $c_i$2, and confidence $c_i$3 (Groot et al., 22 Sep 2025, Dunn, 2021, Lerario et al., 22 Jul 2025).

This terminological divergence is substantive rather than cosmetic. In the first case, “variety” is empirical human uncertainty over valid continuations; in the second, it is a national or geo-referenced language variety; in the third, it is an algebraic variety in the sense of real algebraic geometry. This suggests that any discussion of “testing the variety hypothesis” must specify the object class, the metric on that class, and the sampling model.

2. Distributional variety in next-word prediction

The language-model formulation begins from the claim that natural language generation tasks are often subject to inherent variability: predicting the next word given a context has multiple valid responses, evident when asking multiple humans to complete the task. The hypothesis therefore concerns the full conditional distribution rather than the modal answer. Groot et al. use the Provo Corpus, originally gathered to study eye movements during reading but repurposed for next-word prediction. Provo comprises 55 short passages that yield 2,687 contexts $c_i$4. Each context was shown to on average 40 different annotators, each asked: “What is one plausible next word?” (Groot et al., 22 Sep 2025).

The empirical human distribution is formed from the observed continuation frequencies. Contexts differ in how open-ended they are, so the study introduces an “oracle” Total Variation Distance (TVD): the $c_i$5 human continuations for a context are randomly split into two halves, two empirical distributions are estimated, and TVD is computed between them. Across contexts this oracle mean TVD is about $c_i$6, which sets a lower bound on internal human noise. The alignment metric between humans and models is

$c_i$7

A perfect match gives TVD $c_i$8; maximal disagreement approaches $c_i$9.

The fine-tuning methodology differs for the two model families studied. For pre-trained GPT-2 (124 M), the optimizer is Adam, the learning rate is $\rho$0, batch size is 16, and training runs for 3 epochs. Instead of the usual one-hot cross-entropy $\rho$1, the model minimizes the generalized multi-label cross-entropy

$\rho$2

If a word $\rho$3 tokenizes to $\rho$4, then

$\rho$5

For instruction-tuned Mistral-7B-IT, the study uses LoRA + 4-bit quantization (QLoRA; Dettmers et al., 2023), updating only low-rank adapters, with learning rate $\rho$6, batch size 32, and 4 epochs. Rather than modifying the loss, each context $\rho$7 is replicated once for every human-observed continuation $\rho$8, and each prompt–response pair is trained with the usual one-token cross-entropy $\rho$9. In both cases, the goal is to expose the model to the full set of human continuations rather than to a single “gold” label.

3. Empirical validation and trade-offs in language-model testing

Evaluation estimates $\mu$0 by sampling 40 continuations from the model and taking empirical frequencies. Mean TVD is then averaged over the 20% held-out test contexts, with standard deviations reported across three random seeds (Groot et al., 22 Sep 2025).

For GPT-2, the base model has mean TVD $\mu$1. Fine-tuning on the original Provo passage tokens does not help: FT (Orig.) gives $\mu$2. Fine-tuning on only the majority human continuation improves alignment to $\mu$3, and FT (Mul.) further improves it to $\mu$4. For Mistral-7B-IT, the corresponding values are Base $\mu$5, 1-Shot $\mu$6, FT (Orig.) $\mu$7, FT (Maj.) $\mu$8, and FT (Mul.) $\mu$9, with Oracle $Z\in H(n,k,d)$0 for both model columns.

These results support several specific claims. Fine-tuning on Provo with only one continuation does not reduce TVD relative to Base, ruling out mere OOD effects. Using the majority-vote continuation yields a substantial drop, especially in lower-variability contexts. Crucially, multi-label fine-tuning reduces mean TVD further and shifts the entire TVD distribution towards the oracle. A seed-wise standard deviation of $Z\in H(n,k,d)$1–$Z\in H(n,k,d)$2 indicates that these gains are robust. An ablation varying the number of human labels per context—1, 2, 4, 16, 32—shows that after about 16 annotations the performance plateaus, suggesting diminishing returns beyond that.

The paper also reports a lexical diversity analysis: the fine-tuned Mistral model covers a larger fraction of the set of unique human continuations, indicating that the model is better at generating rare human-preferred words, not just the modal ones. At the same time, a follow-up probe using 55 hand-crafted next-word QA items reveals a trade-off in single-answer tasks. Hit rates dropped for Mistral-7B-IT from $Z\in H(n,k,d)$3 to $Z\in H(n,k,d)$4 after multi-label fine-tuning, while GPT-2 rose marginally from $Z\in H(n,k,d)$5 to $Z\in H(n,k,d)$6. The stated implication is that optimizing for broad variability can impose a cost when a task admits no ambiguity.

The limitations are explicit. Provo is small—2.7 K contexts with about 40 annotations each—and only in English. The approach assumes that all annotators sample from a single underlying “true” distribution, an untested assumption. Only next-word distributional variation is addressed, not longer-span generation. Multi-label fine-tuning can degrade performance on deterministic tasks, highlighting a need to balance diversity with precision.

4. Reliability testing for geo-referenced language varieties

In computational dialectology, the variety hypothesis is framed as a reliability claim about corpora rather than a distribution-matching claim about generative models. Dunn studies 84 language varieties across nine languages: Arabic, German, English, French, Indonesian, Dutch, Portuguese, Russian, and Spanish. The corpora are drawn from two large geo-referenced data sources, web data derived from Common Crawl (CC) and Twitter (TW). For each variety and each register, the paper extracts up to 20 independent samples, each containing exactly 1 million word tokens. The total is 1,282 tweet samples and 1,530 web samples, and the number of varieties per language ranges from 2 for Dutch to 17 for Spanish (Dunn, 2021).

Preprocessing includes language identification, tokenization, and limiting the feature inventory to the top 100 k most frequent types per language. The study defines normalized frequencies

$Z\in H(n,k,d)$7

with $Z\in H(n,k,d)$8, and examines cosine similarity, Kullback–Leibler divergence, Jensen–Shannon divergence, and Spearman’s rank correlation. The primary similarity measure is Spearman’s $Z\in H(n,k,d)$9, applied to rank-ordered feature frequencies: $C=\{c_1,\dots,c_N\}$0 where $C=\{c_1,\dots,c_N\}$1.

The experimental protocol consists of five experiments. Experiment 1 measures within-variety consistency by drawing 50 random distinct pairs of samples from the same register. Experiment 2 evaluates feature types by comparing word unigrams and character trigrams. Experiment 3 measures cross-register similarity using 100 random TW–CC pairs for each variety. Experiment 4 compares within-variety and between-variety similarity, using paired $C=\{c_1,\dots,c_N\}$2-tests. Experiment 5 ranks varieties by distance from the global mean frequency vector and correlates the tweet-based and web-based rankings.

The quantitative findings are stable across languages. In Experiment 1, tweet-based character-trigram Spearman means range from $C=\{c_1,\dots,c_N\}$3 for Dutch to $C=\{c_1,\dots,c_N\}$4 for German with $C=\{c_1,\dots,c_N\}$5, while web-based values range from $C=\{c_1,\dots,c_N\}$6 for French and Spanish to $C=\{c_1,\dots,c_N\}$7 for Arabic with $C=\{c_1,\dots,c_N\}$8. Word-unigram consistency is substantially lower. In Experiment 2, character-trigram Spearman yields $C=\{c_1,\dots,c_N\}$9 correct classification for 7 of 9 languages, with range $W_i=\{w_{i1},\dots,w_{iM_i}\}$0–$W_i=\{w_{i1},\dots,w_{iM_i}\}$1, versus $W_i=\{w_{i1},\dots,w_{iM_i}\}$2–$W_i=\{w_{i1},\dots,w_{iM_i}\}$3 for word unigrams. In Experiment 3, mean cross-register Spearman falls in a narrow band: Arabic $W_i=\{w_{i1},\dots,w_{iM_i}\}$4–$W_i=\{w_{i1},\dots,w_{iM_i}\}$5, German $W_i=\{w_{i1},\dots,w_{iM_i}\}$6–$W_i=\{w_{i1},\dots,w_{iM_i}\}$7, English $W_i=\{w_{i1},\dots,w_{iM_i}\}$8–$W_i=\{w_{i1},\dots,w_{iM_i}\}$9, French $c_i$0–$c_i$1, Russian $c_i$2–$c_i$3, and Spanish $c_i$4–$c_i$5. The violin plots are described as tight, indicating small fluctuations across varieties.

Experiment 4 shows that within-variety similarity exceeds between-variety similarity in both registers. For tweets, same-variety $c_i$6–$c_i$7, different-variety $c_i$8–$c_i$9, difference $p(w\mid c_i)$0–$p(w\mid c_i)$1, all $p(w\mid c_i)$2. For web, same-variety $p(w\mid c_i)$3–$p(w\mid c_i)$4, different-variety $p(w\mid c_i)$5–$p(w\mid c_i)$6, difference $p(w\mid c_i)$7–$p(w\mid c_i)$8, all $p(w\mid c_i)$9. In Experiment 5, the joint Spearman rank correlation between tweet-based and web-based variety rankings is $q_\theta(w\mid c_i)$0 with $q_\theta(w\mid c_i)$1; individually significant cases include German $q_\theta(w\mid c_i)$2, French $q_\theta(w\mid c_i)$3, Portuguese $q_\theta(w\mid c_i)$4, and Russian $q_\theta(w\mid c_i)$5. Aggregating all languages by country, the average cross-register similarity is $q_\theta(w\mid c_i)$6: 62 of 66 countries fall in $q_\theta(w\mid c_i)$7, only Bahrain and Brazil fall below $q_\theta(w\mid c_i)$8, and only Nigeria and New Zealand exceed $q_\theta(w\mid c_i)$9.

The principal caveat is that reliability does not imply validity. A stable relationship between web and tweet data does not prove that either corpus reflects offline spoken usage. The paper also notes register effects, with tweets showing slightly higher internal consistency, and it treats lower-similarity cases such as Brazil and Bahrain as reminders of possible data-coverage issues or language-identification errors.

5. Statistical testing near real algebraic varieties

In “Testing the variety hypothesis,” the problem is formalized for a probability measure on the closed unit disk $c_i$00. The unknown object is a Borel probability measure $c_i$01 supported on $c_i$02. For integers $c_i$03 and $c_i$04, the hypothesis class is

$c_i$05

For any compact set $c_i$06, the squared-distance risk is

$c_i$07

The measure $c_i$08 is said to be “$c_i$09-supported” on $c_i$10 if $c_i$11.

The decision problem is to distinguish, with probability at least $c_i$12, between the alternatives

$c_i$13

The main uniform-convergence theorem states that there exist constants $c_i$14, depending only on $c_i$15, such that for all $c_i$16 and $c_i$17, if

$c_i$18

then with probability at least $c_i$19,

$c_i$20

where $c_i$21. A corollary gives a test that decides “yes” iff

$c_i$22

with the same sample bound. The asymptotic form is

$c_i$23

for some $c_i$24 (Lerario et al., 22 Jul 2025).

A second major contribution is the reduction to a semialgebraic decision problem. Any $c_i$25 can be defined by a single nonnegative polynomial

$c_i$26

with $c_i$27. The empirical risk constraint

$c_i$28

is encoded as a polynomial-inequality condition in the coefficients of $c_i$29, together with auxiliary variables for the distance computation. Dimension and nonemptiness are enforced by first-order conditions requiring rank deficiencies of the Jacobian $c_i$30 at boundary and interior points. The resulting coefficient set $c_i$31 is semialgebraic, and the existence set

$c_i$32

can be rewritten by effective quantifier elimination as a Boolean combination of sign-conditions on explicit polynomials $c_i$33.

The geometric backbone of the analysis is quantitative Hausdorff control over real algebraic varieties. For compact sets $c_i$34, the Hausdorff distance is

$c_i$35

On the polynomial coefficient space $c_i$36, the norm is the Bombieri–Weyl norm $c_i$37. The Thom-Lipschitz theorem bounds the Hausdorff distance between zero sets under perturbation by the distance to the discriminant $c_i$38. The quantitative closure theorem states that for every $c_i$39 and $c_i$40, there exists $c_i$41 such that

$c_i$42

A subsequent net-construction theorem gives $c_i$43 varieties $c_i$44 whose Hausdorff $c_i$45-balls cover $c_i$46.

Two algorithmic strategies follow. One is symbolic quantifier elimination, with worst-case runtime doubly-exponential in the number of variables. The other is a cover-and-test procedure over the finite $c_i$47-net, computing empirical risks $c_i$48 and accepting if $c_i$49. Its runtime is $c_i$50 trials, each requiring $c_i$51 distance computations, with distance evaluation reducible to Euclidean Distance Degree methods of cost $c_i$52. The overall cost is therefore bounded by $c_i$53.

6. Comparative interpretation, misconceptions, and open directions

The three uses of the variety hypothesis share neither a common ontology nor a common metric. In the language-model setting, the object is a conditional next-word distribution, and the key quantity is TVD to an empirical human distribution. In computational dialectology, the object is a geo-referenced language variety, and the key evidence is stable cross-register Spearman similarity. In algebraic geometry, the object is a real algebraic variety, and the key guarantee is a finite-sample test for small squared-distance risk. This suggests a common operational pattern—structured target classes, explicit metrics, and decision procedures based on sampled data—but not a single unified formalism.

One common misconception is that “variety hypothesis” has a fixed technical meaning across fields. The literature does not support that reading. Across these papers, “variety” denotes, respectively, empirical human continuation variability, national language varieties represented by digital corpora, and algebraic varieties in $c_i$54. A second misconception is that positive evidence in one setting automatically transfers to another. The support reported by Groot et al. concerns pluralistic next-word prediction and multi-label fine-tuning; the support reported by Dunn concerns corpus reliability across registers; the guarantees proved in the algebraic setting concern sample complexity and semialgebraic decidability.

The limitations are likewise domain-specific. For next-word prediction, the dataset is small, English-only, and restricted to local continuation choice; multi-label fine-tuning may reduce accuracy on deterministic tasks. For corpus reliability, the paper emphasizes that reliability is not validity, and that lower-agreement cases remain possible. For algebraic testing, decidability is established in principle, but one of the explicit computational routes has doubly-exponential worst-case runtime.

The open directions stated most explicitly arise from the language-model work: scaling to larger and more varied multi-reference datasets, exploring hybrid objectives that preserve factual accuracy while injecting human-like variability where appropriate, extending from word-level to phrase- or sentence-level generation, and investigating multilingual settings and variation across demographic or stylistic subgroups (Groot et al., 22 Sep 2025). A plausible implication of the broader literature is that future work on “testing the variety hypothesis” will continue to depend on making the notion of variety operational through explicit target classes, empirical approximations, and metrics whose stability can be quantified.