Allotaxonometry: Comparing Complex Systems

• Allotaxonometry is a quantitative method for comparing complex systems by analyzing heavy-tailed component distributions.
• It employs tunable divergences, such as rank-turbulence and probability-turbulence, to balance contributions from both dominant and rare types.
• Its applications span diverse fields including ecology, linguistics, economics, and large language model analysis.

Searching arXiv for the specified allotaxonometry papers to ground the article in the cited literature. Allotaxonometry is the quantitative, systematic, and information-rich comparison of two complex systems, or two states of the same system, through the structure of their component types. It was introduced for settings in which component sizes, frequencies, or ranks are heavy-tailed over many orders of magnitude, so that both dominant and rare types may matter substantively. In its canonical form, allotaxonometry couples a tunable divergence with an allotaxonograph: a two-part display that combines a map-like histogram of paired ranks or probabilities with a ranked list of per-type contributions. The original formulation centered on rank-turbulence divergence (RTD), a rank-based instrument for comparing ranked lists, and was later extended by probability-turbulence divergence (PTD), a probability-based instrument for comparing normalizable categorical distributions when reliable probability estimates are available (Dodds et al., 2020, Dodds et al., 2020).

1. Conceptual scope and heavy-tailed setting

Allotaxonometry addresses the comparison of complex systems whose components vary strongly in abundance or prominence. The motivating examples given in the literature include city populations, wealth distributions, species abundances, word frequencies, node degrees, n-grams, motifs, links, market capitalization, mortality causes, sports performance, job titles, and baby names. Across such systems, size–rank or frequency distributions are typically heavy-tailed or Zipf-like: a small number of types are very common, while a very large number are rare, with slow decay across rank (Dodds et al., 2020, Dodds et al., 2020).

This heavy-tailed structure creates a methodological difficulty. Rare types are numerous and noisy, common types can dominate conventional divergences, and comparisons often need to remain interpretable at both ends of the distribution. Allotaxonometry was proposed precisely for this regime. In the rank-based formulation, it is the quantitative comparison of the component structure, or “taxonomy,” of two systems through a tunable divergence over ranks. In the probability-based formulation, it becomes the systematic comparison of two categorical frequency distributions by directly operating on probabilities rather than ranks (Dodds et al., 2020, Dodds et al., 2020).

A central feature is tunability. The divergence has a single parameter $\alpha$ that shifts emphasis across the spectrum of types. Small $\alpha$ emphasizes tail behavior; large $\alpha$ emphasizes the head. This yields an instrument that is not merely a scalar distance but a controlled lens on compositional change. The method is therefore described as an instrument of “type calculus” in the RTD literature (Dodds et al., 2020).

2. Rank-turbulence divergence and the rank-based formulation

The original allotaxonometric instrument is rank-turbulence divergence. Let two systems be $A$ and $B$, with type sets $R_A$ and $R_B$, and let $U = R_A \cup R_B$ be the union of types. Each type $i$ has size $s_{i,A}$ and $\alpha$0, and the corresponding ranks $\alpha$1 and $\alpha$2 are obtained by sorting sizes in descending order, so rank $\alpha$3 is the largest. Ties are handled by fractional ranks. Types absent from one system are assigned tied last rank in that system: if $\alpha$4, then $\alpha$5, and symmetrically for $\alpha$6, where $\alpha$7 and $\alpha$8 (Dodds et al., 2020).

For $\alpha$9, RTD uses the rank transform $\alpha$0 and defines the per-component contribution

$\alpha$1

The overall divergence is the normalized sum

$\alpha$2

with normalization chosen so that $\alpha$3 and the maximally different case of disjoint type sets with the same within-system rank distributions attains $\alpha$4:

$\alpha$5

RTD is symmetric, non-negative, linearly separable into per-type contributions, and normalized to the unit interval. It is explicitly presented as a divergence rather than a metric; the triangle inequality is not guaranteed (Dodds et al., 2020).

Its limiting cases define the interpretation of $\alpha$6. In the tail-focused limit $\alpha$7,

$\alpha$8

with normalization

$\alpha$9

Here, relative rank ratios dominate, so large moves from rare to common, or the reverse, are emphasized. At $A$0, the contribution becomes

$A$1

which emphasizes the head and shoulder more than the far tail. In the head-focused limit $A$2, only rank mismatches contribute, weighted by inverse rank, so the highest-ranked types dominate (Dodds et al., 2020).

The motivation for the rank-based formulation is methodological as much as mathematical. Heavy tails generate severe subsampling and normalization problems for probability-based comparisons, and some systems are more naturally ranked by non-probabilistic attributes such as market capitalization or scores. RTD avoids estimating probabilities, handles exclusives through the tied-last convention, and preserves a directly interpretable per-type decomposition (Dodds et al., 2020).

3. Probability-turbulence divergence and direct comparison of frequencies

Probability-turbulence divergence extends allotaxonometry from ranks to probabilities. Let $A$3 and $A$4 be two normalizable categorical distributions indexed over the union $A$5 of their type sets, with each $A$6 and each distribution summing to $A$7. Types with $A$8 and $A$9, or vice versa, are “exclusive types” (Dodds et al., 2020).

For $B$0, PTD is constructed by transforming probabilities via $B$1, taking per-type absolute differences, reweighting by the exponent $B$2, and normalizing so that disjoint catalogs attain divergence $B$3. The per-type contribution is

$B$4

the normalization is

$B$5

and the overall divergence is

$B$6

PTD is symmetric, non-negative, normalized so that the maximum divergence is $B$7 when type sets are disjoint, and is likewise presented as a divergence rather than a metric (Dodds et al., 2020).

Its tuning role parallels RTD but now operates on frequency rather than position. Small $B$8 amplifies contrasts among rare types, while large $B$9 privileges contrasts among common types. For shared types, the limit $R_A$0 yields

$R_A$1

whereas exclusive types have per-type contributions diverging like $R_A$2. Because the normalization also diverges like $R_A$3, the total divergence remains finite, reducing to

$R_A$4

The consequence is sharp: if every type appears in both systems, then $R_A$5, even if probabilities differ. At the opposite limit,

$R_A$6

so the dominant terms are the most frequent types unless their probabilities are exactly equal (Dodds et al., 2020).

PTD is structurally modeled after RTD but differs in scope. RTD is more broadly applicable because it needs only ranks; PTD is more limited in application because it requires reliable normalization and robust probability estimates. When those estimates are available, PTD is more sensitive to actual changes in type frequency. The literature states this trade-off explicitly: RTD is more universal in settings lacking trustworthy probability estimates, whereas PTD better detects meaningful shifts in rate-of-use of types (Dodds et al., 2020).

PTD also establishes direct relations to other divergences and indices. At $R_A$7, it complements the Sørensen–Dice similarity and, equivalently, the $R_A$8 score; at $R_A$9, its per-type ordering agrees functionally with Hellinger-type squared-chord distances, including the Matusita distance; at $R_B$0, its contribution ordering aligns with many $R_B$1-norm salience orderings; and in the $R_B$2 limit it matches the structure of the Motyka distance up to a factor. The paper further draws conceptual links to Rényi entropy, Tsallis entropy, and Hill numbers, emphasizing that PTD shares their rare-versus-common tuning role but cannot be constructed from algebraic combinations of Rényi or Hill quantities (Dodds et al., 2020).

4. Allotaxonographs as analytic displays

The allotaxonograph is the visual counterpart of allotaxonometric divergence. In the RTD literature it is described as a dashboard combining a map-like, log-binned rank–rank histogram with a companion ordered list of components by descending contribution to RTD, together with balance bars that indicate system sizes and exclusivity. In the PTD literature the same logic is transferred to probability space, producing an allotaxonograph over paired probabilities rather than paired ranks (Dodds et al., 2020, Dodds et al., 2020).

For RTD, the left panel records all pairs $R_B$3 in a rank–rank histogram. The axes are log-binned, and the plot is rotated by $R_B$4 into a diamond so that the vertical axis is proportional to $R_B$5 and the horizontal axis to $R_B$6. Overlaid iso-contribution contours are defined by

$R_B$7

and visually reveal the geometry of the chosen $R_B$8. The right panel sorts types by $R_B$9, placing them left or right according to which system they favor. Types absent from one system appear on exclusive-type lines at the base of the histogram because of the tied-last rank convention (Dodds et al., 2020).

For PTD, the left panel is a two-dimensional histogram over $U = R_A \cup R_B$0, again rotated to avoid suggesting causality on either axis. The paper emphasizes a practical complication: zero probabilities correspond to $U = R_A \cup R_B$1. The solution is to append lighter panels at the bottom of each axis to accommodate $U = R_A \cup R_B$2 cases. Contour lines therefore “jump” discretely into these zero panels. On the right, the types are ranked by $U = R_A \cup R_B$3, and exclusive types are marked separately, for example with open triangles (Dodds et al., 2020).

A recurring interpretive rule is that edge types matter most. In PTD, the paper states that only types at the edges of the histogram can dominate any reasonable divergence. A “good fit” $U = R_A \cup R_B$4 is one whose contours track the outer envelope of the heavy-tailed cloud, producing a ranked list that is neither overwhelmed by exclusive rare types nor restricted to the very highest-frequency head. Very small $U = R_A \cup R_B$5 collapses interior contours and makes the list dominated by exclusives; very large $U = R_A \cup R_B$6 focuses the contours near the top of the histogram and foregrounds only the most frequent items (Dodds et al., 2020).

5. Empirical domains and applications

Allotaxonometry was introduced as a domain-general comparative framework and demonstrated on language use on Twitter and in books, species abundance, baby name popularity, market capitalization, sports performance, mortality causes, and job titles. The RTD paper’s examples show how the same rank-based instrument can compare corpora, ecological censuses, naming systems, and economic rankings without changing the underlying formalism (Dodds et al., 2020).

In the RTD examples, the day after the 2016 United States election was compared with the day after Charlottesville in Twitter language using $U = R_A \cup R_B$7, yielding $U = R_A \cup R_B$8. The contribution list foregrounded words such as “Trump,” “America,” “Donald,” “voters,” and “election” on one side, and “Charlottesville,” “Heyer,” “Nazis,” and “white” on the other, while an exclusive term such as “Cvjetanovic” was assigned tied last rank in the earlier corpus. In Barro Colorado Island tree species data, the 1985 and 2015 censuses were compared with $U = R_A \cup R_B$9, giving $i$0, far below a reported randomization baseline of $i$1; the leading contributor was Piper cordulatum, which dropped from rank $i$2 to $i$3. In United States baby names, the head-focused limit $i$4 highlighted top-name turnover, with reported values $i$5 for girls and $i$6 for boys (Dodds et al., 2020).

PTD was demonstrated on literature, social media, and ecology. In Jane Austen’s Pride and Prejudice, comparing the first and second halves of the text at the 2-gram level, $i$7 was reported as a good fit, and the top contributions mixed character names such as “Miss Bingley” and “Lady Catherine,” functional phrases such as “she had” and “in the,” and locale references such as the exclusive 2-gram “the Parsonage.” In Twitter data, comparing 2020/03/12 with 2020/05/30, the paper reports that 1-grams were well balanced by $i$8, 2-grams by $i$9, and 3-grams by $s_{i,A}$0, with exclusive lexicon fractions of approximately $s_{i,A}$1, $s_{i,A}$2, and $s_{i,A}$3 respectively. In Barro Colorado Island tree species abundances, comparing 1985 with 2015, $s_{i,A}$4 yielded a balanced list, again led by Piper cordulatum, and PTD and RTD were said to produce broadly consistent salient species lists (Dodds et al., 2020).

A later application brought allotaxonometry into the analysis of LLMs. In a study of “us versus them” bias, the method was used to compare 1-gram rank distributions in LLM outputs across ingroup versus outgroup prompts, across Conservative versus Liberal persona-conditioned outputs, and before versus after mitigation. The study used RTD with $s_{i,A}$5, 2,000 completions per condition, lowercase 1-gram tokenization as continuous sequences of non-whitespace characters, retained stopwords, and ranked words by their contribution to divergence. Reported examples included ingroup-overrepresented terms such as “champions,” “believers,” “advocates,” “reproductive,” and “equity,” outgroup-overrepresented terms such as “hypocrites,” “blame,” and “bubble,” and persona-specific contrasts such as Liberal-high “rights,” “social,” “justice,” “equity,” and “reproductive” versus Conservative-high “government,” “defense,” “freedom,” “brave,” “national,” and “enterprise.” In the post-mitigation comparison, affectively charged items reportedly lost prominence, while more neutral or institutional terms remained (Prama et al., 3 Dec 2025).

These applications show that allotaxonometry is not restricted to a single ontology of “types.” The compared entities may be words, species, names, firms, or any discrete ranked or countable components, provided that the comparison is meaningful at the level of shared and exclusive types.

6. Parameter choice, computation, and limitations

The central modeling choice in allotaxonometry is the tuning parameter $s_{i,A}$6. In RTD, small $s_{i,A}$7 emphasizes tail volatility and one-sided types; $s_{i,A}$8 to $s_{i,A}$9 often balances head, shoulder, and tail; and $\alpha$00 surfaces head changes. In PTD, small $\alpha$01 focuses on changes in the rare tail, whereas large $\alpha$02 privileges high-probability types. The recommended heuristic in both literatures is visual: choose $\alpha$03 so that the divergence contours match the observed envelope of the histogram and the resulting ranked list is balanced across the spectrum rather than dominated exclusively by rare or common types. The PTD paper states explicitly that there is no universal $\alpha$04 and treats automated parameter selection as an open problem, noting that simple regression is confounded by the overwhelming mass of rare types (Dodds et al., 2020, Dodds et al., 2020).

The computational workflow is straightforward. For RTD, one forms the union of types, computes ranks with fractional ties, assigns exclusives tied last rank, evaluates per-type contributions, computes the normalization, sums contributions, and constructs the allotaxonograph. The complexity is dominated by ranking at $\alpha$05, with divergence computation and histogram binning at $\alpha$06. For PTD, one normalizes counts into probabilities, forms the union of types, computes $\alpha$07 and $\alpha$08, aggregates to $\alpha$09, ranks types by $\alpha$10, and renders the histogram and ranked list. The PTD paper states that the method is linear in $\alpha$11, that sparse maps are memory-efficient for heavy-tailed catalogs, and that the instrument scales well to millions of types if implemented with streaming aggregation and sparse structures (Dodds et al., 2020, Dodds et al., 2020).

Several limitations are explicit in the literature. RTD and PTD are symmetric divergences, not metrics. RTD is preferred when probabilities are unreliable, when heavy tails are poorly sampled, or when the data are intrinsically rank-based. PTD requires reliable normalization and good probability estimates and is therefore more sensitive to sparse-tail uncertainty. PTD has a distinctive special case at $\alpha$12: it measures only catalog exclusivity, so changes in shared-type probabilities do not affect $\alpha$13. Conversely, RTD remains informative at $\alpha$14 because it depends on log-rank ratios. The literature also cautions that heavy tails amplify sampling noise, that truncation and coverage should be checked, that ties should be handled fractionally, and that exclusives should be treated with the prescribed conventions—$\alpha$15 in RTD and appended zero-probability panels in PTD—to preserve interpretability (Dodds et al., 2020, Dodds et al., 2020).

A common misconception is that allotaxonometry is simply another name for a standard scalar distance. In fact, its defining feature is the joint use of a tunable divergence and a decomposable visualization. Another misconception is that RTD and PTD are interchangeable. The papers present them as structurally related but methodologically distinct: RTD is the universal, rank-based instrument for comparing complex systems under heavy tails, while PTD is the more frequency-sensitive instrument for normalizable categorical distributions when probabilities are well estimated (Dodds et al., 2020, Dodds et al., 2020).