Tokenization Inconsistency (TI) in NLP

• Tokenization Inconsistency (TI) is defined as failures in maintaining stable mappings between surface text and token sequences across different contexts and tasks.
• TI manifests through phenomena such as extractive QA mismatches, phantom edits, and retokenization non-invariance that compromise model output integrity.
• Empirical findings show that addressing TI can improve task accuracy (e.g., up to 22% gain in arithmetic tasks) while reducing bias and security risks.

Searching arXiv for papers on tokenization inconsistency and related tokenizer effects to ground the article. Tokenization inconsistency (TI) denotes a family of failures in which the relation between surface text and token sequences is not stable across contexts, tasks, encode–decode cycles, or equivalent internal representations. In recent work, TI has been used to describe at least four closely related phenomena: the same extractive span being tokenized differently in input and output contexts; detokenization–retokenization mismatches between communicating parties; multiple token ID sequences detokenizing to the same surface string; and formal failures of consistency or unbiasedness in statistical estimation and sampling. Across these senses, TI has been linked to performance degradation, hallucination, multilingual inequity, security fragility, and biases that scaling alone does not remove (Sun et al., 2022, Yan et al., 28 Aug 2025, Ayoobi et al., 21 Jan 2026, Gastaldi et al., 2024, Phan et al., 2024).

1. Conceptual scope and formal definitions

In extractive generation, TI arises when the same text segment is tokenized differently depending on context. A canonical example is extractive question answering: in BART-style BPE, the answer string "1912" may appear in the passage as ['G1912'] but as standalone output as ['19', '12'], so the model is not simply copying a span from input to output (Sun et al., 2022). In steganography and watermarking, TI is defined as the discrepancy between the token sequence generated by Alice and the sequence obtained by Bob after detokenizing and then retokenizing the transmitted text; the literature distinguishes Source Inconsistent Tokens (SITs) and Consequential Inconsistent Tokens (CITs) for the unmatched tokens on the two sides (Yan et al., 28 Aug 2025).

A more recent line of work locates TI in the representational non-uniqueness of modern subword tokenizers. Multiple distinct token ID sequences can decode to the same surface string, so a model can internally modify token IDs while leaving the visible text unchanged. The resulting phantom edits expose a tokenizer-level defect rather than a conventional knowledge failure: the model operates on different token sequences but produces an identical string after detokenization (Ayoobi et al., 21 Jan 2026).

The formal theory of tokenization frames these issues in terms of stochastic maps. A tokenizer model is an encoder–decoder pair, with encoder $\tau: \Sigma^* \rightsquigarrow \Delta^*$ and decoder $\kappa: \Delta^* \rightsquigarrow \Sigma^*$. The basic consistency condition is

$\kappa \tau p = p,$

meaning that decoding the encoded distribution recovers the original distribution over character strings. Exactness strengthens this to $\kappa \tau = \mathrm{id}_{\Sigma^*}$ for every input (Gastaldi et al., 2024). A related result on next-character sampling shows that popular encoding schemes such as maximum prefix encoding and BPE induce a sampling bias that cannot be mitigated with more training or data; in that setting, TI is not merely an implementation artifact but a structural property of the encoding scheme (Phan et al., 2024).

2. Mechanisms that produce inconsistency

Several mechanisms recur across the literature. One is context-sensitive subword merging, especially where whitespace or affix boundaries participate in the token inventory. Space-prefixed and non-prefixed variants of the same string can lead to different tokenizations in otherwise equivalent contexts, which is why extractive tasks are particularly exposed (Sun et al., 2022). Closely related mechanisms underlie the taxonomy of phantom edits: whitespace-boundary shifts, whitespace detachment or reattachment, intra-word resegmentation, morphological boundary surfacing, acronym splitting, and plural or possessive tail tokens (Ayoobi et al., 21 Jan 2026).

A second mechanism is retokenization non-invariance. In security settings, a generated token sequence may detokenize to a string whose retokenization is different. The example ["_no", "body"] detokenizing to "nobody" and then retokenizing as ["_nobody"] captures the basic failure mode (Yan et al., 28 Aug 2025). This is distinct from, but related to, the one-to-many token ID mappings that produce phantom edits in replacement tasks (Ayoobi et al., 21 Jan 2026).

A third mechanism is directional or domain-specific segmentation bias. In arithmetic, GPT-3.5 and GPT-4 use up-to-3-digit tokens chunked left-to-right by default, whereas right-to-left chunking can be enforced with comma formatting at inference time. The resulting difference is not superficial formatting; it changes token boundaries and leads to a large accuracy gap on addition (Singh et al., 2024). In a broader tokenizer-design study, pre-tokenization, vocabulary construction, and segmentation are treated as separate design phases, and the evidence shows that minimizing token count is not sufficient for downstream quality (Schmidt et al., 2024).

A fourth mechanism is byte-level incompleteness. Byte-level BPE vocabularies may contain undecodable or incomplete tokens consisting of stray bytes. When paired into out-of-distribution improbable bigrams, such tokens show a strong tendency toward hallucinatory behavior; alternative tokenizations of the same phrase can reduce hallucination sharply, indicating that the defect lies in incomplete-token handling rather than in the text itself (Jang et al., 2024).

A fifth mechanism is semantic fragmentation of natural words. Recent work defines “bad tokenization” as splitting natural words into multiple tokens, proposes penalty functions for quantifying the severity of such splits, and reports statistically significant links between higher penalties and model errors across multiple tasks and LLMs (Pawar et al., 26 Dec 2025). This suggests that TI is not restricted to explicit encoder–decoder mismatches; it also includes systematic degradations caused by subword decompositions that diverge from natural word structure.

3. Empirical manifestations across tasks

The clearest evidence for TI appears in extractive and arithmetic settings. In generative extractive QA, inconsistent tokenization damages the extractive nature of the task and increases hallucination. One study reports that with consistent tokenization, a BART model trained on SQuAD and evaluated on 8 QA datasets achieves a notable average of +1.7 F2 gain, converges faster, and becomes less likely to generate out-of-context answers (Sun et al., 2022). The same line of work also reports that, with standard tokenization, the percentage of training instances where the tokenized gold answer does not appear verbatim in the tokenized context can reach 99.6% (Sun et al., 2022).

In arithmetic, TI appears as a token-boundary effect rather than a semantic misunderstanding. For 8-shot addition with gpt-3.5-turbo-0301, standard left-to-right number chunking yields about 76% accuracy, while right-to-left chunking yields about 98%, a gain of up to 22 percentage points (Singh et al., 2024). The error profile is stereotyped rather than noisy: when the sum is longer in digits than either addend, left-to-right accuracy can drop to 8%, and the dominant pattern is a digit 4 error in which the first three digits are always correct (Singh et al., 2024). This is accompanied by off-by-one errors at token boundaries and a finding that errors are not correlated with the number of carries, indicating that the relevant inductive bias is tokenization-dependent rather than a direct reflection of human arithmetic difficulty (Singh et al., 2024).

Machine translation studies show that the phenomenon long predates current frontier LLMs. In neural machine translation, testing five tokenizers over ten language pairs led to the conclusion that tokenization significantly affects final translation quality and that the best tokenizer differs for different language pairs (Domingo et al., 2018). Reported differences reach up to 12 BLEU points and 15 TER points, with language-specific segmenters such as Mecab or the Stanford Word Segmenter outperforming generic alternatives on Japanese and Chinese respectively (Domingo et al., 2018).

TI can also be induced adversarially. The ADT (Adversarial Dataset for Tokenizer) was designed specifically to challenge token segmentation. On ADT-Human (Chinese), reported overall error rates include 61.36% for GPT-4o and 50.00% for GPT-4; on ADT-Human (English), GPT-4o reaches 64.29% (Wang et al., 2024). The same work argues that tokenization inconsistency is a universal issue across open-source and closed-source LLMs and is especially visible in Chinese (Wang et al., 2024).

4. Bias, multilingual disparity, and security implications

TI has a pronounced multilingual and fairness dimension. One strand of work measures fertility, defined as

$F = \frac{T}{W},$

the average number of tokens per word, and shows on AfriMMLU that higher fertility reliably predicts lower accuracy across 10 LLMs and 16 African languages (Lundin et al., 5 Sep 2025). The reported regression slopes range from $-0.08$ to $-0.18$, interpreted as 8–18 percentage-points lower accuracy per unit increase in tokens per word, with fertility explaining 20–50% of the variance in accuracy (Lundin et al., 5 Sep 2025). The same study translates token inflation into cost: because transformer cost scales quadratically with sequence length, a doubling in tokens leads to quadrupled training cost and time, and a 5× token inflation yields a 25× cost or time increase (Lundin et al., 5 Sep 2025).

A larger cross-linguistic evaluation over more than 200 languages uses Tokens Per Sentence (TPS), Characters Per Token (CPT), and Relative Tokenization Cost (RTC). English is reported at about 50.2 TPS, while Myanmar reaches about 357.2 TPS, giving an RTC of approximately 7.12 (Teklehaymanot et al., 14 Oct 2025). Non-Latin and morphologically complex languages often incur 3–5 times higher RTC ratios, which directly reduces effective context utilization and increases token-based costs (Teklehaymanot et al., 14 Oct 2025). Related work on dialectal tasks distinguishes Tokenization Parity (TP) and Information Parity (IP), finding that TP is a better predictor on tasks reliant on syntactic and morphological cues such as extractive QA, whereas IP better predicts semantic tasks such as topic classification (Kanjirangat et al., 24 Sep 2025).

Bias can also enter through the tokenizer vocabulary itself. A study of GPT-4 and GPT-4o reports that tokenizer vocabularies constructed independently of actual model training data can inadequately represent non-English languages, propagating under-trained or untrained tokens and raising security and ethical concerns (Yang et al., 2024). For long Chinese tokens, Token Retention Accuracy (TRA) is reported as 80.7% for GPT-4 but 45.2% for GPT-4o; human raters also found that GPT-4o produced completely irrelevant sentences in about 35% of cases for such tokens (Yang et al., 2024). Another paper argues more generally that tokens and pretraining can act as a backdoor for bias and other unwanted content, and that current alignment practices may not remediate this pathway (Zimmerman et al., 2024).

Security-sensitive applications make TI operational. In LLM-based steganography and watermarking, inconsistent tokens are rare at the token level—reported as mostly <0.5% of total tokens—but TI can affect up to 50% of long messages at the text level (Yan et al., 28 Aug 2025). The same work shows that some TIs are temporary and some persistent, and that the distribution differs sharply by model, with only 8.76% temporary TIs in Llama-2-7b but 81.98% in Swallow-7b and 87.93% in Qwen2.5-7b (Yan et al., 28 Aug 2025).

5. Measurement and analytical frameworks

TI is measured through several partly overlapping lenses. Some are formal, some are task-specific, and some are explicitly fairness-oriented.

Measure	Definition	Main use
Statistical consistency	$\kappa \tau p = p$	Whether tokenization preserves estimator consistency (Gastaldi et al., 2024)
Fertility	$F = \frac{T}{W}$	Tokens-per-word inflation and its link to accuracy and cost (Lundin et al., 5 Sep 2025)
Relative Tokenization Cost	$RTC(L) = \frac{TPS(L)}{TPS(English)}$	Cross-linguistic efficiency relative to English (Teklehaymanot et al., 14 Oct 2025)
Tokenization Parity	$\kappa: \Delta^* \rightsquigarrow \Sigma^*$0	Parallel-sentence fragmentation across languages or dialects (Kanjirangat et al., 24 Sep 2025)

Task-level diagnostics extend these measures. The penalty-function framework for “bad tokenization” proposes Token Anomaly Score (AS), Similarity to Under-trained Tokens (UT), Pairwise Distance Between Tokens (PD), and Contextual Penalty (CP), then compares penalties on correct versus incorrect predictions using one-sided Student’s $\kappa: \Delta^* \rightsquigarrow \Sigma^*$1-tests and Mann–Whitney $\kappa: \Delta^* \rightsquigarrow \Sigma^*$2 tests (Pawar et al., 26 Dec 2025). Among these, CP is reported as the strongest and most consistent correlate of model errors, with statistically significant differences in at least 17 out of 28 dataset–LLM settings (Pawar et al., 26 Dec 2025). The same study reports that these penalties always outperform simple baselines such as number of tokens or number of splits (Pawar et al., 26 Dec 2025).

A different analytic tradition focuses on unbiased inference rather than downstream accuracy. For tokenized autoregressive models, the bias in next-character sampling under MPE and BPE is formalized and corrected algorithmically: an unbiased estimation procedure can simulate token-free behavior from any LLM trained on tokenized data, without finetuning, and with complexity linear in sequence length in the MPE case (Phan et al., 2024). This line of work implies that some observed TI is not merely empirical brittleness but an identifiable distortion in the probability calculus induced by the tokenizer.

6. Mitigation strategies and research directions

The simplest mitigation is to enforce consistency where the task semantics demand it. In extractive QA, the recommended procedure is to identify the answer span in the input, tokenize the whole input, extract the answer’s tokenization from the tokenized context, and train the model to output that exact token sequence rather than a separately tokenized answer string (Sun et al., 2022). This restores the equality condition $\kappa: \Delta^* \rightsquigarrow \Sigma^*$3 for the answer span and reduces out-of-context generation (Sun et al., 2022).

Arithmetic TI can be mitigated either by changing the input tokenization or by making the model convert tokenizations internally. Right-to-left chunking improves performance directly, and a prompting-based conversion in which the model repeats the problem and answer with a different tokenization nearly matches direct right-to-left input; simply asking for right-to-left output without repeating the problem does not suffice (Singh et al., 2024). This indicates that alignment of input and output token boundaries is itself a computational aid (Singh et al., 2024).

Robustness to segmentation variation can also be trained. A study of the “curse of tokenization” reports that moderate BPE-dropout, specifically rates around 0.2–0.4, improves generalization and robustness, whereas high dropout rates above 0.6 hurt performance (Chai et al., 2024). For incomplete tokens in byte-level BPE, forcing alternative tokenizations that avoid incomplete-token pairings can dramatically reduce hallucination: in Llama3.1, the reported reduction is 93% on the same phrases (Jang et al., 2024).

Security applications require stronger guarantees. In steganography, stepwise verification removes candidate-level inconsistent tokens during generation and guarantees exact token-sequence alignment, yielding 100% correct extraction (Yan et al., 28 Aug 2025). In watermarking, a post-hoc rollback method monitors temporary versus persistent TIs and corrects only the persistent cases, with reported runtime overhead <5% (Yan et al., 28 Aug 2025).

At the tokenizer-design level, multiple studies converge on similar recommendations: use linguistically informed or morphology-aware tokenization; treat pre-tokenization, vocabulary construction, and segmentation as separate design choices; avoid treating token count minimization as the sole objective; and adopt broader multilingual evaluation rather than English-centric compression metrics (Schmidt et al., 2024, Wegmann et al., 21 Feb 2025, Teklehaymanot et al., 14 Oct 2025). Formal work adds two further directions: tokenizer-aware objectives that account for equivalent encodings, and post-hoc algorithms that recover unbiased character-level behavior from tokenized models (Ayoobi et al., 21 Jan 2026, Phan et al., 2024). Taken together, these results suggest that TI is not a marginal preprocessing issue but a representational problem at the interface between strings, vocabularies, and model computation.