Heaps' Law: Type-Token Dynamics

• Heaps’ law is a statistical relationship that describes the sublinear growth of unique types as the total token count increases.
• It is closely related to Zipf’s law, where the observed power-law behavior and exponent depend on sampling methods, finite-size effects, and domain-specific constraints.
• Recent research generalizes the classical power law by introducing models that account for temporal correlations, dependency structures, and logarithmic corrections.

Heaps’ law, also called Herdan’s law, is the type–token relation in a growing sequence: as more tokens are observed, the number of distinct types increases in a regular way. In its classical form, one writes $V(N)\propto N^\beta$ or $V(N)=\alpha N^\beta$, where $N$ is token count, $V(N)$ is vocabulary size or type count, and $\beta$ is the growth exponent. In quantitative linguistics this is usually interpreted as sublinear vocabulary growth, but the modern literature treats the pure power law as a first-order description rather than an exact universal law: depending on the Zipf convention, finite-size regime, temporal ordering, and domain-specific constraints, the observed curve can be linear, logarithmically corrected, sublinear, saturating, or slightly curved even in log-log coordinates (Rosillo-Rodes et al., 3 Nov 2025, Font-Clos et al., 2014).

1. Definitions, observables, and notational variants

The basic objects are tokens and types. In text, tokens are word occurrences and types are distinct words; in other component systems, tokens are repeated instances of elementary components and types are distinct component classes. Several papers formalize the cumulative token count and vocabulary size explicitly. For a corpus of documents, if $n_i$ is the number of terms in document $i$, then after the first $i$ documents the cumulative corpus size is

$N_i=\sum_{\ell=1}^i n_\ell,$

and $V(N_i)$ is the number of distinct terms seen in those first $V(N)=\alpha N^\beta$0 documents. In generic type–token notation one may also write

$V(N)=\alpha N^\beta$1

where $V(N)=\alpha N^\beta$2 is the total number of tokens, $V(N)=\alpha N^\beta$3 the number of unique types, and $V(N)=\alpha N^\beta$4 the multiplicity of type $V(N)=\alpha N^\beta$5 (Lai et al., 2023, Li et al., 2024).

The literature distinguishes at least two empirically different uses of the law. One is a cross-sectional relation across complete datasets, such as the correlation between final vocabulary size $V(N)=\alpha N^\beta$6 and final text length $V(N)=\alpha N^\beta$7 across many books, composers, or corpora. The other is the sequential Heaps function within a single realization, often written $V(N)=\alpha N^\beta$8, the cumulative number of distinct types observed up to position $V(N)=\alpha N^\beta$9. These two uses are not interchangeable: a good power-law fit across datasets does not imply that within-sequence vocabulary growth is itself a clean power law (Chacoma et al., 2020, Serra-Peralta et al., 2021).

Notation is not standardized. Some papers use $N$0 for the Heaps exponent and $N$1 for the Zipf exponent, while others do the reverse. The same ambiguity occurs on the Zipf side, where one may define the exponent through a rank–frequency curve, a count–rank curve, or a frequency-distribution tail. Several papers emphasize that the mapping between Zipf and Heaps exponents depends on which quantity is assigned the exponent and on whether the statement concerns asymptotic or finite-size behavior (Rosillo-Rodes et al., 3 Nov 2025, Lu et al., 2010).

2. Zipf’s law as a source of Heaps-type growth

A central line of work treats Heaps’ law as derivative from Zipf’s law. In one deterministic formulation, types are ranked by count and assumed to satisfy an inverse power-law ranking

$N$2

with total token count $N$3 and number of observed types $N$4. Imposing that the last observed type has count $N$5 yields $N$6, and summing over ranks gives

$N$7

where $N$8 is the generalized harmonic number. The asymptotics of $N$9 then determine the type–token law (Rosillo-Rodes et al., 3 Nov 2025).

Under the convention $V(N)$0, the large-$V(N)$1 regimes are:

Zipf regime	Type–token asymptotics	Heaps interpretation
$V(N)$2	$V(N)$3	asymptotically linear
$V(N)$4	$V(N)$5	linear with logarithmic correction
$V(N)$6	$V(N)$7	sublinear power law with exponent $V(N)$8

This formulation corrects two common oversimplifications. First, the case $V(N)$9 is not a pure power law; it is the marginal regime $\beta$0. Second, for $\beta$1, vocabulary growth saturates at exponent $\beta$2 rather than remaining $\beta$3. The same work shows that replacing the harmonic sum by a crude integral fails in important special cases, especially at $\beta$4 and for steep rank decay $\beta$5, where the earlier approximation is wrong by a factor $\beta$6 (Rosillo-Rodes et al., 3 Nov 2025).

Earlier finite-size analysis had already shown that the classical reciprocal exponent relation is only asymptotically correct. Starting from a Zipf profile $\beta$7, one obtains the implicit relation

$\beta$8

with the special case

$\beta$9

at $n_i$0. In this framework the effective Heaps exponent depends strongly on system size $n_i$1, especially near $n_i$2, and numerical finite-size solutions match empirical data better than the infinite-size asymptotic formula in most tested datasets (Lu et al., 2010).

3. Generalizations and corrections to the classical power law

A large body of recent work argues that the pure power-law form is often only a local approximation. One route starts from the distribution of counts rather than the rank–frequency curve. If the number of types with count $n_i$3 obeys

$n_i$4

then the expected vocabulary growth in a random system is not generally a power law but

$n_i$5

with finite-size corrections involving the Lerch transcendent. This produces a log-log convex type–token curve rather than a straight line, and real books were found to follow this form surprisingly closely after rescaling by $n_i$6 and $n_i$7 (Font-Clos et al., 2014).

A second route treats vocabulary growth as a consequence of a two-variable scaling law for the frequency distribution. In that framework,

$n_i$8

Classical Heaps behavior $n_i$9 appears only when the scaling function is a pure power law down to the lowest frequencies. For lemmatized texts, however, the proposed double-power-law form

$i$0

implies

$i$1

so vocabulary growth is controlled by the low-frequency structure, not just by the Zipf tail (Font-Clos et al., 2013). A related first-principles sampling approach derives

$i$2

from the frequency spectrum $i$3, and under a particular “perfect Zipf distribution” obtains the logarithmic-type law

$i$4

together with equally explicit formulas for hapaxes and higher $i$5-legomena (Davis, 2018).

Other corrections target the residual curvature of empirical Heaps plots. Across twenty English books, a quadratic model in log-log space,

$i$6

fit the type–token data much better than a straight line, with the linear coefficient typically around $i$7 or slightly above and the quadratic coefficient typically around $i$8; this implies a scale-dependent effective exponent rather than a constant Heaps exponent (Fontanelli et al., 18 Nov 2025). A complementary hapax-based program makes the hapax rate

$i$9

the primary object and derives

$i$0

In that formulation the classical law $i$1 is exactly the special case of constant hapax rate, while logistic hapax-rate models gave the best empirical fits among the candidates tested (Dębowski, 2023).

4. Temporal ordering, dependence, and sequential Heaps functions

The classical Zipf–Heaps link usually assumes temporal independence or an equivalent random-sampling picture. Recent work makes explicit that this assumption is fragile. If a sequence is reshuffled, its empirical rank–frequency distribution is preserved but its temporal correlations are destroyed. Under such reshuffling, the same multiset of tokens can generate very different type–token trajectories $i$2, bounded by a maximally accelerated ordering and a maximally delayed ordering. Empirically, ordered and reshuffled sequences are close for written texts, but differ markedly for music listening and web browsing, where domain-specific temporal correlations make the observed Heaps exponent $i$3 weakly related to the reshuffled benchmark $i$4 (Zimmerlin et al., 24 Oct 2025).

Within literary texts, the same distinction appears in another form. Across many books, the global relation $i$5 can be well approximated by a power law, yet the within-text cumulative vocabulary curve $i$6 is generally not a power law. For tagged English literary texts, the observed $i$7 is on the whole well described by the exact expectation over random shufflings of the same text, but the deviations are statistically significant and predominantly negative, meaning that new words appear later than the shuffle baseline. Those deviations are grammatically structured: nouns follow the mean trend, verbs are more retarded than the mean, and “others” are less retarded (Chacoma et al., 2020).

In the strict IID infinite-dictionary model, Heaps’ law can be put on a rigorous asymptotic basis. If word probabilities satisfy

$i$8

then Bahadur’s theorem yields

$i$9

and Karlin’s strengthening gives

$N_i=\sum_{\ell=1}^i n_\ell,$0

This same framework supports goodness-of-fit tests for Zipf’s law based on deviations of the observed vocabulary-growth path from its Heaps-type scaling (Chebunin et al., 2017).

5. Empirical scope across language, biology, and other component systems

In large historical corpora, Heaps’ law is visible but not stationary. Using Google Books Ngram data, one study fit the English aggregate relation $N_i=\sum_{\ell=1}^i n_\ell,$1 with $N_i=\sum_{\ell=1}^i n_\ell,$2, while a function-word-adjusted probabilistic model gave $N_i=\sum_{\ell=1}^i n_\ell,$3. More importantly, the local Heaps exponent

$N_i=\sum_{\ell=1}^i n_\ell,$4

varied substantially across time, and the paper highlighted fluctuations over characteristic intervals of $N_i=\sum_{\ell=1}^i n_\ell,$5-$N_i=\sum_{\ell=1}^i n_\ell,$6 years in English, Russian, German, and French (Bochkarev et al., 2016).

Large-language-model corpora also exhibit the law. In PubMed-abstract emulations generated by GPT-Neo, the fitted Heaps exponents were $N_i=\sum_{\ell=1}^i n_\ell,$7 for GPT-Neo 125M, $N_i=\sum_{\ell=1}^i n_\ell,$8 for GPT-Neo 1.3B, and $N_i=\sum_{\ell=1}^i n_\ell,$9 for GPT-Neo 2.7B, compared with $V(N_i)$0 for the human PubMed corpus. All fits had very high log-log correlations, but the generated corpora grew vocabulary faster, largely because they produced many more singleton terms; increasing model size reduced that discrepancy (Lai et al., 2023).

Outside language, the law has been investigated in genomics and music. For “functional DNA words” defined as coding regions that code for protein domains, the human genome showed range-limited Heaps behavior, with estimated exponents $V(N_i)$1, $V(N_i)$2, and $V(N_i)$3 depending on whether the samples were chromosomes, chromosome arms, or both; when random token sampling reached the maximum level, the log-log type–token curve became distinctly quadratic rather than linear (Li et al., 2024). In classical music harmony, a composer-level fit over $V(N_i)$4 composers gave

$V(N_i)$5

while the same relation over $V(N_i)$6 individual pieces gave $V(N_i)$7, showing that the estimated Heaps exponent depends strongly on the level of aggregation (Serra-Peralta et al., 2021).

6. Mechanisms, interpretations, and controversies

One major interpretation is that Heaps’ law is not fundamental but derivative. Under a stable Zipfian frequency profile, both deterministic and probabilistic analyses derive vocabulary growth directly from frequency heterogeneity, leading several authors to describe Heaps’ law as a consequence or corollary of Zipf’s law rather than an independent empirical law (Lu et al., 2010, Rosillo-Rodes et al., 3 Nov 2025). A stronger version of this view places Heaps’ law inside larger scaling chains: one paper derives a differential Heaps law from Zipf-tail conditions, then uses it as the bridge from Zipf statistics to Hilberg-type entropy growth and ultimately to neural scaling bounds (Dębowski, 15 Dec 2025).

At the same time, several mechanisms generate Heaps behavior without reducing it to a fixed IID sampling model. Dependency-network models produce a mean-field Heaps curve with three regimes—linear, sublinear, and saturating—and an intermediate exponent $V(N_i)$8 controlled by the dependency out-degree $V(N_i)$9 (Mazzolini et al., 2018). Sample-space-reducing processes generate Zipf-like probabilities $V(N)=\alpha N^\beta$00 and an associated vocabulary curve

$V(N)=\alpha N^\beta$01

with $V(N)=\alpha N^\beta$02 for $V(N)=\alpha N^\beta$03, logarithmic corrections at $V(N)=\alpha N^\beta$04, and saturation at finite $V(N)=\alpha N^\beta$05 (Mazzolini et al., 2018). Modified Pólya-urn models based on the adjacent possible produce

$V(N)=\alpha N^\beta$06

with $V(N)=\alpha N^\beta$07 when $V(N)=\alpha N^\beta$08, linear growth when $V(N)=\alpha N^\beta$09, and $V(N)=\alpha N^\beta$10 at the boundary $V(N)=\alpha N^\beta$11, while simultaneously generating Zipf and Taylor laws (Tria et al., 2018).

The main controversy, therefore, is not whether type–token growth exists, but how it should be interpreted. One tradition treats it as a derived effect of Zipfian abundance structure; another shows that temporal correlations, finite dictionaries, hapax dynamics, and domain-specific organization can decouple the observed Heaps curve from any single static frequency law. A cautious synthesis is that Heaps’ law names a family of type–token regularities, not a single invariant formula: the classical power law is often a useful summary, but its exponent and even its functional form depend on conventions, scale, ordering, and mechanism (Zimmerlin et al., 24 Oct 2025, Font-Clos et al., 2014, Dębowski, 2023).