Causal Estimation of Tokenisation Bias

• The paper introduces a causal estimation framework employing regression discontinuity to quantify how tokeniser subword inclusion affects model output probabilities.
• It details experiments on LLaMA-style Transformers with various tokenisers and vocabulary sizes, showing significant probability jumps and aggregate bias metrics.
• The study highlights practical implications for multilingual fairness and economic costs while proposing algorithmic debiasing strategies like BPTree to mitigate tokenisation bias.

Causal estimation of tokenisation bias concerns quantifying the effect that a tokeniser’s subword vocabulary construction has on LLM (LM) behavior, specifically on the probabilities assigned to character-strings. While idealized LMs would be invariant to tokenisation choices, in practice, the presence or absence of specific subwords in the vocabulary exerts a causal influence on model output, resulting in systematic disparities—termed tokenisation bias—across contexts, languages, and model architectures. Recent work rigorously formalizes this bias, primarily via a regression discontinuity design leveraging the constructional properties of standard subword tokenisers (Lesci et al., 3 Jun 2025). Complementary research investigates the downstream consequences for multilingual fairness, inference efficiency, and economic costs (Lundin et al., 5 Sep 2025), while related studies detail general sampling biases arising from token-based autoregressive decoders (Phan et al., 2024).

1. Formal Definition of Tokenisation Bias

Tokenisation bias is defined as the change in the probability a model assigns to a character-string due solely to the inclusion or exclusion of its corresponding subword in the tokeniser's vocabulary. Let $i$ index subwords and $s_i$ represent the character-string spanned by subword $i$. Define the treatment indicator: $$W_i = 1\{\text{subword } i \text{ is in the vocabulary}\}$$ Potential outcomes are: $$Y_i(1) = \mathbb{E}_{c}\left[\log\,p_{\text{model trained with } i \in V}(s_i|c)\right]$$

$$Y_i(0) = \mathbb{E}_{c}\left[\log\,p_{\text{model trained without } i \in V}(s_i|c)\right]$$

where the expectation is over all preceding contexts $c$. The individual tokenisation bias (Individual Treatment Effect) is: $\tau_i = Y_i(1) - Y_i(0)$ with the average tokenisation bias given by: $\bar{\tau} = \mathbb{E}_i[Y_i(1)-Y_i(0)]$ This structure frames tokenisation bias explicitly as a causal estimand, focusing on the direct effect of subword presence in the vocabulary (Lesci et al., 3 Jun 2025).

2. Causal Estimation via Regression Discontinuity

Bottom-up tokenisation algorithms (e.g., BPE, WordPiece) construct a ranked sequence of candidate merges $m_1, m_2, \dots, m_K, \dots, m_{K_+}$; only the first $K$ merges constitute the active vocabulary. For each subword $i$ with rank $r_i$, the treatment assignment is $W_i = 1\{r_i \le K\}$. The cutoff $r = K$ (the vocabulary size limit) is exogenous, yielding a natural regression discontinuity (RD) framework.

The causal RD estimand is: $$\tau_{\rm RD} = \lim_{r \downarrow K}\mathbb{E}[Y_i|r_i=r] - \lim_{r \uparrow K}\mathbb{E}[Y_i|r_i=r]$$ In practice, a local linear regression is fitted within rank bandwidth $h$ about $K$: $$Y_i = \alpha + \beta\,(r_i - K) + \tau\,W_i + \varepsilon_i$$ The coefficient $\hat{\tau}$ estimates the local average treatment effect at the cutoff.

To ensure identification, the primary assumption is the continuity of expected potential outcomes in $r$ around $K$. Empirical robustness is checked by varying $h$, employing kernel-weighted fits, and applying LOESS curves on each cutoff side. Exclusion of nested subwords is required to satisfy non-interference (SUTVA).

3. Empirical Quantification and Key Findings

Experiments employ LLaMA-style Transformers (6–24 layers, 57M–850M non-embedding parameters) trained on an English corpus (“MiniPile”). Multiple tokenisers are used: vanilla BPE, WordPiece, and a composite “BPE2WP” (WordPiece applied over a BPE vocabulary), with vocabulary sizes $K \in \{8\text{k},32\text{k},128\text{k}\}$.

For each model, in-context log-probabilities $Y_i$ are computed for all vocabulary-eligible subwords on a held-out validation set, and the RD estimator is applied with $h=5000$.

Key results:

• For a 57M-parameter model with BPE at $K=32$k, $\hat{\tau} \approx 2.88$ nats, corresponding to a $\approx 17\times$ higher probability for subwords present in the vocabulary compared to those just excluded.
• Confidence intervals exclude zero by a wide margin (e.g., $[2.64, 3.12]$ nats).
• Smaller vocabularies ($K=8$k) yield somewhat lower bias ($\approx 2.3$ nats); larger vocabularies ($128$k) show similar effects.
• Across tokenisers, estimated jumps range from $2.5$ to $3.2$ nats.
• Increasing model size reduces the bias: $340$M $\rightarrow$ $\approx 2.1$ nats, $850$M $\rightarrow$ $\approx 1.0$ nat.
• Training dynamics: at initialization, bias reflects entropy ($\approx \log K$); sharp decline after initial steps, with steady rise to asymptotic values by step $50$k.

Subwords included in the vocabulary exhibit substantially higher stability in output log-probability across contexts than those marginally excluded (Lesci et al., 3 Jun 2025).

4. Downstream Impact: Length, Accuracy, and Economic Effects

Tokenisation bias manifests as a “length bias”: strings requiring multiple subwords to encode receive systematically lower probabilities than those mapped to a single subword, after controlling for frequency and context. This directly affects downstream metrics such as perplexity, generation stability, and, crucially, fairness in multilingual scenarios.

In multilingual LLM evaluation, token inflation—measured via fertility

$F = \frac{T}{W}$

(where $T$ is the total token count for $W$ whitespace-separated words)—serves as a proxy for tokenisation inefficiency. Lundin et al. show that higher fertility predicts lower MCQA accuracy across African languages, with OLS regression slopes $\beta_1$ ranging from $-0.08$ to $-0.18$ (i.e., each additional token/word correlates with an $8$–$18$ point drop in accuracy). Fertility explains $20$–$50$\% of accuracy variance per (model, subject) pair (Lundin et al., 5 Sep 2025).

Given transformers’ quadratic scaling in sequence length, doubling fertility quadruples training cost and time: $$\frac{\text{Cost}_2}{\text{Cost}_1} = \left(\frac{2F_1}{F_1}\right)^2 = 4$$ For instance, if English ($F\approx1$) requires \$105M in training cost, a language with$F=2$incurs \$420M for equivalent data. Inference latency and per-token cost also scale with fertility.

A plausible implication is that tokenisation-induced disparities disproportionately penalize morphologically rich, low-resource languages, both in model accuracy and in economic resources.

5. Causal Mechanisms and Algorithmic Mitigation

Tokenisation bias emerges via the interaction between tokeniser design and autoregressive decoding. Standard encoding schemes such as BPE and maximum prefix encoding (MPE) introduce an irreducible bias: the model’s estimated next-character distribution $P_{\text{model}}(\cdot \mid Z)$ (where $Z$ is the token sequence) can diverge systematically from the token-free ground truth $P_{\text{true}}(\cdot \mid X)$.

Explicitly, for prompt $x$ and character $c$: $$\operatorname{Bias}_T(x, c) = \sum_{t \in E(x, c)} P_{\text{model}}(t \mid z) - P_{\text{true}}(c \mid x)$$ where $E(x, c)$ is the set of subwords starting with $c$.

Algorithmic debiasing strategies are feasible. The Branch-and-Pass Tree (BPTree) estimator reconstructs unbiased character-level probabilities from any tokenized LM without retraining, operating via brute-force marginalization over subwords starting with specified character suffixes. This approach has been empirically validated: in controlled Markov chain settings, BPTree recovers character-level transition probabilities, whereas direct token prompting exhibits substantial systematic bias (Phan et al., 2024).

Mitigation via BPTree is practical for moderate vocabulary sizes and low-latency applications, but summation over large subword sets can present scalability challenges. The approach extends naturally to BPE as well as WordPiece tokenization.

6. Limitations, Assumptions, and Open Challenges

The causal estimation framework rests on several assumptions:

• The cutoff rank $K$ is exogenous, with no hidden confounding near the threshold.
• The potential outcomes are continuous in $r_i$ around $K$.
• Experiments are limited to English text; extension to other scripts and languages with higher fertility remains necessary.
• Results pertain to local effects: the RD estimator captures average treatment effects for subwords near the vocabulary boundary, not global properties of the vocabulary.
• SUTVA may be violated if inclusion/exclusion of one subword affects the likelihood of others (especially nested or overlapping subwords).
• In observational regression frameworks (e.g., (Lundin et al., 5 Sep 2025)), unobserved confounders (training data volume, domain divergence, script effects) may bias causal estimates.

A plausible implication is that tokenisation bias may be somewhat mitigated by increasing model size or adopting morphologically aware tokenisers, but cannot be entirely eliminated without fundamental architectural shifts or token-free modeling paradigms.

7. Practical Implications and Recommendations

Tokenisation bias is a major determinant of both the output probabilities and practical deployment characteristics of LMs:

• Tokeniser choice alters the log-probabilities of character-strings by orders of magnitude, explaining phenomena such as length bias and accuracy gaps for morphologically rich or low-resource languages.
• Regression discontinuity estimators, as proposed by Lesci et al., allow accurate measurement of vocabulary design impacts without retraining (Lesci et al., 3 Jun 2025).
• For multilingual and low-resource scenarios, token inflation directly depresses accuracy and multiplies computational cost, underscoring the need for morphologically aware tokenisation and fair resource allocation (Lundin et al., 5 Sep 2025).
• Algorithmic post-hoc debiasing enables token-free sampling from standard LMs, improving evaluation fairness and adaptability to diverse tokenisation schemes (Phan et al., 2024).

The selection of vocabulary size, tokeniser architecture, and downstream evaluation benchmarks should be guided by explicit measurement of local treatment effects (e.g., via $\hat{\tau}$). Extension of these methods to other languages, alternative model families, and study of global (rather than local) bias remains a key future direction.