---
title: Tokenization Inconsistency (TI) in NLP
url: https://www.emergentmind.com/topics/tokenization-inconsistency-ti
type: topic
---

# Tokenization Inconsistency (TI) in NLP

Searching arXiv for recent papers on tokenization inconsistency and related tokenizer effects to ground the article.
Tokenization inconsistency (TI) denotes a family of failures in which the relation between surface text and token sequences is not stable across contexts, tasks, encode–decode cycles, or equivalent internal representations. In recent work, TI has been used to describe at least four closely related phenomena: the same extractive span being tokenized differently in input and output contexts; detokenization–retokenization mismatches between communicating parties; multiple token ID sequences detokenizing to the same surface string; and formal failures of consistency or unbiasedness in statistical estimation and sampling. Across these senses, TI has been linked to performance degradation, hallucination, multilingual inequity, security fragility, and biases that scaling alone does not remove [2212.09912], [2508.20718], [2601.14658], [2407.11606], [2406.16829].

## 1. Conceptual scope and formal definitions

In extractive generation, TI arises when the same text segment is tokenized differently depending on context. A canonical example is extractive question answering: in BART-style BPE, the answer string `"1912"` may appear in the passage as `['G1912']` but as standalone output as `['19', '12']`, so the model is not simply copying a span from input to output [2212.09912]. In steganography and watermarking, TI is defined as the discrepancy between the token sequence generated by Alice and the sequence obtained by Bob after detokenizing and then retokenizing the transmitted text; the literature distinguishes **Source Inconsistent Tokens (SITs)** and **Consequential Inconsistent Tokens (CITs)** for the unmatched tokens on the two sides [2508.20718].

A more recent line of work locates TI in the representational non-uniqueness of modern subword tokenizers. Multiple distinct token ID sequences can decode to the same surface string, so a model can internally modify token IDs while leaving the visible text unchanged. The resulting **phantom edits** expose a tokenizer-level defect rather than a conventional knowledge failure: the model operates on different token sequences but produces an identical string after detokenization [2601.14658].

The formal theory of tokenization frames these issues in terms of stochastic maps. A tokenizer model is an encoder–decoder pair, with encoder $\tau: \Sigma^* \rightsquigarrow \Delta^*$ and decoder $\kappa: \Delta^* \rightsquigarrow \Sigma^*$. The basic consistency condition is

$$
\kappa \tau p = p,
$$

meaning that decoding the encoded distribution recovers the original distribution over character strings. Exactness strengthens this to $\kappa \tau = \mathrm{id}_{\Sigma^*}$ for every input [2407.11606]. A related result on next-character sampling shows that popular encoding schemes such as maximum prefix encoding and BPE induce a sampling bias that cannot be mitigated with more training or data; in that setting, TI is not merely an implementation artifact but a structural property of the encoding scheme [2406.16829].

## 2. Mechanisms that produce inconsistency

Several mechanisms recur across the literature. One is **context-sensitive subword merging**, especially where whitespace or affix boundaries participate in the token inventory. Space-prefixed and non-prefixed variants of the same string can lead to different tokenizations in otherwise equivalent contexts, which is why extractive tasks are particularly exposed [2212.09912]. Closely related mechanisms underlie the taxonomy of phantom edits: whitespace-boundary shifts, whitespace detachment or reattachment, intra-word resegmentation, morphological boundary surfacing, acronym splitting, and plural or possessive tail tokens [2601.14658].

A second mechanism is **retokenization non-invariance**. In security settings, a generated token sequence may detokenize to a string whose retokenization is different. The example `["_no", "body"]` detokenizing to `"nobody"` and then retokenizing as `["_nobody"]` captures the basic failure mode [2508.20718]. This is distinct from, but related to, the one-to-many token ID mappings that produce phantom edits in replacement tasks [2601.14658].

A third mechanism is **directional or domain-specific segmentation bias**. In arithmetic, GPT-3.5 and GPT-4 use up-to-3-digit tokens chunked left-to-right by default, whereas right-to-left chunking can be enforced with comma formatting at inference time. The resulting difference is not superficial formatting; it changes token boundaries and leads to a large accuracy gap on addition [2402.14903]. In a broader tokenizer-design study, pre-tokenization, vocabulary construction, and segmentation are treated as separate design phases, and the evidence shows that minimizing token count is not sufficient for downstream quality [2402.18376].

A fourth mechanism is **byte-level incompleteness**. Byte-level BPE vocabularies may contain undecodable or incomplete tokens consisting of stray bytes. When paired into out-of-distribution **improbable bigrams**, such tokens show a strong tendency toward hallucinatory behavior; alternative tokenizations of the same phrase can reduce hallucination sharply, indicating that the defect lies in incomplete-token handling rather than in the text itself [2410.23684].

A fifth mechanism is **semantic fragmentation of natural words**. Recent work defines “bad tokenization” as splitting natural words into multiple tokens, proposes penalty functions for quantifying the severity of such splits, and reports statistically significant links between higher penalties and model errors across multiple tasks and LLMs [2512.21933]. This suggests that TI is not restricted to explicit encoder–decoder mismatches; it also includes systematic degradations caused by subword decompositions that diverge from natural word structure.

## 3. Empirical manifestations across tasks

The clearest evidence for TI appears in extractive and arithmetic settings. In generative extractive QA, inconsistent tokenization damages the extractive nature of the task and increases hallucination. One study reports that with consistent tokenization, a BART model trained on SQuAD and evaluated on 8 QA datasets achieves a notable average of **+1.7 F2 gain**, converges faster, and becomes less likely to generate out-of-context answers [2212.09912]. The same line of work also reports that, with standard tokenization, the percentage of training instances where the tokenized gold answer does not appear verbatim in the tokenized context can reach **99.6%** [2212.09912].

In arithmetic, TI appears as a token-boundary effect rather than a semantic misunderstanding. For 8-shot addition with `gpt-3.5-turbo-0301`, standard left-to-right number chunking yields about **76%** accuracy, while right-to-left chunking yields about **98%**, a gain of up to **22 percentage points** [2402.14903]. The error profile is stereotyped rather than noisy: when the sum is longer in digits than either addend, left-to-right accuracy can drop to **8%**, and the dominant pattern is a **digit 4 error** in which the first three digits are always correct [2402.14903]. This is accompanied by off-by-one errors at token boundaries and a finding that errors are not correlated with the number of carries, indicating that the relevant inductive bias is tokenization-dependent rather than a direct reflection of human arithmetic difficulty [2402.14903].

Machine translation studies show that the phenomenon long predates current frontier LLMs. In neural machine translation, testing five tokenizers over ten language pairs led to the conclusion that tokenization significantly affects final translation quality and that the best tokenizer differs for different language pairs [1812.08621]. Reported differences reach **up to 12 BLEU points and 15 TER points**, with language-specific segmenters such as Mecab or the Stanford Word Segmenter outperforming generic alternatives on Japanese and Chinese respectively [1812.08621].

TI can also be induced adversarially. The **ADT (Adversarial Dataset for Tokenizer)** was designed specifically to challenge token segmentation. On **ADT-Human (Chinese)**, reported overall error rates include **61.36%** for GPT-4o and **50.00%** for GPT-4; on **ADT-Human (English)**, GPT-4o reaches **64.29%** [2405.17067]. The same work argues that tokenization inconsistency is a universal issue across open-source and closed-source LLMs and is especially visible in Chinese [2405.17067].

## 4. Bias, multilingual disparity, and security implications

TI has a pronounced multilingual and fairness dimension. One strand of work measures **fertility**, defined as

$$
F = \frac{T}{W},
$$

the average number of tokens per word, and shows on AfriMMLU that higher fertility reliably predicts lower accuracy across 10 LLMs and 16 African languages [2509.05486]. The reported regression slopes range from **$-0.08$ to $-0.18$**, interpreted as **8–18 percentage-points lower accuracy** per unit increase in tokens per word, with fertility explaining **20–50%** of the variance in accuracy [2509.05486]. The same study translates token inflation into cost: because transformer cost scales quadratically with sequence length, a doubling in tokens leads to quadrupled training cost and time, and a 5× token inflation yields a 25× cost or time increase [2509.05486].

A larger cross-linguistic evaluation over more than 200 languages uses **Tokens Per Sentence (TPS)**, **Characters Per Token (CPT)**, and **Relative Tokenization Cost (RTC)**. English is reported at about **50.2 TPS**, while Myanmar reaches about **357.2 TPS**, giving an **RTC** of approximately **7.12** [2510.12389]. Non-Latin and morphologically complex languages often incur **3–5 times higher RTC ratios**, which directly reduces effective context utilization and increases token-based costs [2510.12389]. Related work on dialectal tasks distinguishes **Tokenization Parity (TP)** and **Information Parity (IP)**, finding that TP is a better predictor on tasks reliant on syntactic and morphological cues such as extractive QA, whereas IP better predicts semantic tasks such as topic classification [2509.20045].

Bias can also enter through the tokenizer vocabulary itself. A study of GPT-4 and GPT-4o reports that tokenizer vocabularies constructed independently of actual model training data can inadequately represent non-English languages, propagating under-trained or untrained tokens and raising security and ethical concerns [2406.11214]. For long Chinese tokens, **Token Retention Accuracy (TRA)** is reported as **80.7%** for GPT-4 but **45.2%** for GPT-4o; human raters also found that GPT-4o produced completely irrelevant sentences in about **35%** of cases for such tokens [2406.11214]. Another paper argues more generally that tokens and pretraining can act as a backdoor for bias and other unwanted content, and that current alignment practices may not remediate this pathway [2412.10924].

Security-sensitive applications make TI operational. In LLM-based steganography and watermarking, inconsistent tokens are rare at the token level—reported as mostly **<0.5%** of total tokens—but TI can affect up to **50%** of long messages at the text level [2508.20718]. The same work shows that some TIs are temporary and some persistent, and that the distribution differs sharply by model, with only **8.76%** temporary TIs in Llama-2-7b but **81.98%** in Swallow-7b and **87.93%** in Qwen2.5-7b [2508.20718].

## 5. Measurement and analytical frameworks

TI is measured through several partly overlapping lenses. Some are formal, some are task-specific, and some are explicitly fairness-oriented.

| Measure | Definition | Main use |
|---|---|---|
| Statistical consistency | $\kappa \tau p = p$ | Whether tokenization preserves estimator consistency [2407.11606] |
| Fertility | $F = \frac{T}{W}$ | Tokens-per-word inflation and its link to accuracy and cost [2509.05486] |
| Relative Tokenization Cost | $RTC(L) = \frac{TPS(L)}{TPS(English)}$ | Cross-linguistic efficiency relative to English [2510.12389] |
| Tokenization Parity | $\frac{|t(s_A)|}{|t(s_B)|}$ | Parallel-sentence fragmentation across languages or dialects [2509.20045] |

Task-level diagnostics extend these measures. The penalty-function framework for “bad tokenization” proposes **Token Anomaly Score (AS)**, **Similarity to Under-trained Tokens (UT)**, **Pairwise Distance Between Tokens (PD)**, and **Contextual Penalty (CP)**, then compares penalties on correct versus incorrect predictions using one-sided Student’s $t$-tests and Mann–Whitney $U$ tests [2512.21933]. Among these, **CP** is reported as the strongest and most consistent correlate of model errors, with statistically significant differences in at least **17 out of 28** dataset–LLM settings [2512.21933]. The same study reports that these penalties always outperform simple baselines such as number of tokens or number of splits [2512.21933].

A different analytic tradition focuses on unbiased inference rather than downstream accuracy. For tokenized autoregressive models, the bias in next-character sampling under MPE and BPE is formalized and corrected algorithmically: an unbiased estimation procedure can simulate token-free behavior from any language model trained on tokenized data, without finetuning, and with complexity linear in sequence length in the MPE case [2406.16829]. This line of work implies that some observed TI is not merely empirical brittleness but an identifiable distortion in the probability calculus induced by the tokenizer.

## 6. Mitigation strategies and research directions

The simplest mitigation is to enforce consistency where the task semantics demand it. In extractive QA, the recommended procedure is to identify the answer span in the input, tokenize the whole input, extract the answer’s tokenization from the tokenized context, and train the model to output that exact token sequence rather than a separately tokenized answer string [2212.09912]. This restores the equality condition $T(x)_{I'} = T(y)_{J'}$ for the answer span and reduces out-of-context generation [2212.09912].

Arithmetic TI can be mitigated either by changing the input tokenization or by making the model convert tokenizations internally. Right-to-left chunking improves performance directly, and a prompting-based conversion in which the model repeats the problem and answer with a different tokenization nearly matches direct right-to-left input; simply asking for right-to-left output without repeating the problem does not suffice [2402.14903]. This indicates that alignment of input and output token boundaries is itself a computational aid [2402.14903].

Robustness to segmentation variation can also be trained. A study of the “curse of tokenization” reports that moderate **BPE-dropout**, specifically rates around **0.2–0.4**, improves generalization and robustness, whereas high dropout rates above **0.6** hurt performance [2406.11687]. For incomplete tokens in byte-level BPE, forcing alternative tokenizations that avoid incomplete-token pairings can dramatically reduce hallucination: in Llama3.1, the reported reduction is **93%** on the same phrases [2410.23684].

Security applications require stronger guarantees. In steganography, **stepwise verification** removes candidate-level inconsistent tokens during generation and guarantees exact token-sequence alignment, yielding **100% correct extraction** [2508.20718]. In watermarking, a **post-hoc rollback** method monitors temporary versus persistent TIs and corrects only the persistent cases, with reported runtime overhead **<5%** [2508.20718].

At the tokenizer-design level, multiple studies converge on similar recommendations: use linguistically informed or morphology-aware tokenization; treat pre-tokenization, vocabulary construction, and segmentation as separate design choices; avoid treating token count minimization as the sole objective; and adopt broader multilingual evaluation rather than English-centric compression metrics [2402.18376], [2502.15343], [2510.12389]. Formal work adds two further directions: tokenizer-aware objectives that account for equivalent encodings, and post-hoc algorithms that recover unbiased character-level behavior from tokenized models [2601.14658], [2406.16829]. Taken together, these results suggest that TI is not a marginal preprocessing issue but a representational problem at the interface between strings, vocabularies, and model computation.

Source: https://www.emergentmind.com/topics/tokenization-inconsistency-ti