Tokenization Techniques, Methods, and Impacts

• Tokenization is the process of converting character or byte sequences into discrete tokens drawn from a finite vocabulary. This process is crucial for determining sequence length, vocabulary size, embedding dimensions, and contributes significantly to the performance of language models by encountering issues such as Pretraining-to-Finetuning Domain Disparity.
• Different tokenization methods include subword, word-level, byte-level, BPE, WordPiece, SentencePiece, and morphological tokenization. Each method has trade-offs between vocabulary size, sequence length, and handling of unknown words (like out-of-vocabulary terms and Non-English languages), and the best choice depends on the specific language and application.
• Tokenization techniques not only affect computational efficiency but also influence language-model performance, robustness, capability to handle informal content, and suitability for different tasks across various languages and dialects.

Tokenization is the conversion of character or byte strings into sequences of discrete tokens drawn from a finite vocabulary. A token may be a character, byte, subword, word, morpheme, multiword expression, punctuation mark, structured computer-character sequence, or special symbol. Tokenization determines sequence length, vocabulary size, embedding and output-layer dimensions, information coverage, linguistic granularity, and the computational units available to a LLM. It is therefore simultaneously a representation, compression, statistical-estimation, and computational interface rather than a neutral preprocessing operation.

1. Fundamental concepts and tokenization objectives

Let $\Sigma$ denote a character or byte alphabet and $\Delta$ a token vocabulary. A tokenizer maps a string $x\in\Sigma^*$ to a sequence $(t_1,\ldots,t_K)$ with $t_i\in\Delta$. The number of token occurrences in a corpus is distinct from the number of token types: the former measures sequence length, whereas the latter is the vocabulary cardinality. Character-level representations minimize vocabulary size and largely eliminate unknown-word problems, but produce long sequences. Word-level representations produce shorter sequences but require large vocabularies and suffer from sparsity and OOV failures. Subword tokenization occupies an intermediate position.

A tokenizer typically involves three distinct decisions:

1. Pre-tokenization: rules that impose boundaries before vocabulary construction and segmentation.
2. Vocabulary construction: selection of strings that become token types.
3. Segmentation: selection of a particular token sequence for an input under the fixed vocabulary.

These stages are not interchangeable. “Tokenization Is More Than Compression” demonstrates that changing pre-tokenization, vocabulary construction, or segmentation can materially change language-model performance even when token counts are similar (Schmidt et al., 2024). In particular, unconstrained minimum-token segmentation can create units crossing word boundaries, such as ation_is, _to_b, and e_$$</code>, while using more tokens with linguistically aligned boundaries can yield better performance.</p> <p>The central trade-off can be expressed schematically as:</p> <p>$$\text{small vocabulary} \leftrightarrow \text{long sequences},$</p> <p>$\text{large vocabulary} \leftrightarrow \text{short sequences and greater vocabulary cost}.$$</p> <p>Tokenization also determines whether punctuation is a delimiter or an internal component of a structured unit. An information-retrieval tokenizer may first recognize IP addresses, URLs, dates, and email addresses so that punctuation internal to these sequences does not destroy their identity (<a href="/papers/1303.0407" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Badawi et al., 2013</a>). In this setting, the intended pipeline is:</p> <p>$$\text{document} \rightarrow \text{special-sequence pattern matching} \rightarrow \text{filtration or protection} \rightarrow \text{ordinary tokenization} \rightarrow \text{indexing or statistics}.$

The treatment of recognized sequences may be removal, preservation as unified terms, or another filtration policy; the cited implementation does not specify a single default behavior.

2. Principal tokenization families

Character and byte tokenization

Character-level tokenization represents each character as a separate unit. Byte-level tokenization instead represents UTF-8 or other byte encodings and can process arbitrary text through a fixed byte inventory. The Turkish language-model study evaluates a ByT5-style byte-level vocabulary of 384 symbols (Toraman et al., 2022). Character and byte methods offer language independence and open-vocabulary behavior, but their sequences are long and require the model to construct words, morphology, and larger patterns compositionally.

Byte-level BPE operates over byte strings rather than necessarily complete Unicode characters. A Unicode character may occupy multiple UTF-8 bytes, and a token may represent only part of a character’s byte sequence. “Byte BPE Tokenization as an Inverse string Homomorphism” formalizes this by treating detokenization as a concatenative map from token sequences to byte strings (Geng et al., 2024).

Word-level tokenization

Word tokenization generally splits at whitespace and separates punctuation. It preserves complete surface words when they are present in the vocabulary but does not generalize compositionally across inflection, derivation, compounds, names, or unseen forms. In Korean translation experiments, word tokenization produced a translation OOV rate of 26.20% and was substantially weaker than smaller-granularity alternatives (Park et al., 2020). In Turkish, word-level systems require large vocabularies because agglutinative morphology generates many surface forms (Toraman et al., 2022).

BPE

Byte-Pair Encoding begins with small symbols and repeatedly merges frequent adjacent pairs until a target vocabulary size is reached. If $(a,b)$ is the selected pair, it is replaced with a new symbol $ab$. BPE is compact and usually provides strong open-vocabulary coverage, but its units are frequency-driven rather than explicitly morphological.

BPE can create entire words, word fragments, punctuation units, or strings crossing linguistic boundaries. Its behavior depends strongly on the fitting corpus and pre-tokenizer. A corpus containing informal or dialectal forms can make those forms available as token types, whereas a standardized corpus may represent them through smaller fragments. In experiments on English language variation, pre-tokenizer choice had the largest average effect on downstream performance, followed by vocabulary size and then fitting corpus (Wegmann et al., 21 Feb 2025).

WordPiece

WordPiece is a subword method used in BERT-like models. For a word, standard WordPiece applies a greedy longest-match-first, or MaxMatch, procedure: it repeatedly selects the longest prefix of the remaining string that belongs to the vocabulary. If a complete segmentation cannot be found, the word becomes the unknown token.

The computational cost of conventional MaxMatch can be $\Delta$0 or $\Delta$1, where $\Delta$2 is input length and $\Delta$3 is the maximum vocabulary-token length. LinMaxMatch augments a vocabulary trie with failure links and failure-pop lists, preserving exactly the original greedy output while reducing inference complexity to $\Delta$4 (Song et al., 2020). Its end-to-end extension combines pre-tokenization and WordPiece segmentation in a single linear pass.

SentencePiece and Unigram

SentencePiece operates directly on raw text and does not require whitespace-defined words. Its unigram mode assigns probabilities to candidate pieces and selects or marginalizes over segmentations. It can handle languages without conventional word boundaries, including Chinese and Japanese, and supports subword segmentation across punctuation and unusual character sequences (Domingo et al., 2018).

SentencePiece is not equivalent to BPE. BPE uses an ordered sequence of pair merges, whereas unigram segmentation uses a probabilistic objective over possible segmentations. In Korean translation and natural-language understanding, ordinary BPE was generally surpassed by a hybrid method that applied MeCab-ko morphological segmentation followed by BPE; ordinary 64K BPE was nevertheless best for KorQuAD extractive reading comprehension (Park et al., 2020).

Morphological and hybrid tokenization

Morphological tokenization uses an analyzer to expose roots, stems, affixes, particles, and inflectional endings. In Korean, MeCab-ko trained on the Sejong corpus supplies linguistically motivated boundaries. Pure morphological segmentation reduces linguistic ambiguity but suffers from high OOV and under-trained-token rates because complete morphemes remain atomic.

The hybrid pipeline

$\Delta$5

prevents BPE from crossing morpheme boundaries while retaining subword decomposition for rare morphemes. In the Korean experiments, morphology-aware BPE with a 32K Korean vocabulary achieved 40.34 BLEU for Korean-to-English and 40.41 BLEU for English-to-Korean (Park et al., 2020).

A Turkish hybrid system similarly combines root and affix dictionaries, phonological normalization, shared identifiers for allomorphic variants, explicit whitespace and capitalization tokens, and BPE fallback for unknown material. It reports 90.29% Turkish Token Percentage and 85.8% Pure Token Percentage on TR-MMLU (Bayram et al., 19 Aug 2025). These results establish improved morphological alignment and interpretability for the evaluated setting, but do not by themselves establish superior language-model performance under matched retraining.

3. Linguistic granularity, morphology, and language variation

Tokenization effects are language-dependent because orthographic boundaries, morphology, scripts, and lexical productivity differ substantially across languages.

In Japanese and Chinese, spaces do not normally delimit words. Japanese MeCab uses conditional random fields and morphological information, while the Stanford Word Segmenter uses conditional random fields with Chinese Treebank segmentation and character-based features. In neural machine translation, MeCab was especially effective for English-to-Japanese, achieving 45.8 BLEU compared with 33.6 for Moses; Stanford segmentation was especially effective for English-to-Chinese, achieving 29.9 BLEU compared with 21.4 for Moses (Domingo et al., 2018). These directional asymmetries show that target-side segmentation can have a disproportionately large effect.

Korean tokenization exhibits a similar interaction between linguistic structure and statistical coverage. Pure morphology had OOV rates of 7.51%, 4.67%, 2.72%, and 1.40% for 8K, 16K, 32K, and 64K vocabularies, respectively. Ordinary BPE had OOV rates around 0.07–0.12%, while morphology-aware BPE remained around 0.06–0.07% (Park et al., 2020). The hybrid method therefore combines morpheme boundaries with open-vocabulary decomposition.

For Turkish, WordPiece and BPE were strongest overall at approximately 16.6K vocabulary items, while morphological tokenization remained competitive and consistently exceeded character and word-level tokenization (Toraman et al., 2022). Increasing vocabulary size benefited morphological and word-level tokenizers more than BPE and WordPiece because larger inventories reduce unknown stems and surface forms. The study recommends allocating approximately 20% of total model parameters to vocabulary embeddings for BPE and WordPiece and approximately 40% for morphological and word-level tokenizers, treating these as empirical guidelines rather than universal laws.

Language variation introduces another dimension. In English experiments distinguishing tasks requiring robustness from tasks requiring sensitivity to variation, GPT-2-style pre-tokenization was strongest on robust semantic tasks, while a Twitter-fitted tokenizer and a 64K vocabulary were stronger on variation-sensitive tasks such as authorship, register, error, and dialect classification (Wegmann et al., 21 Feb 2025). The same tokenizer is therefore not necessarily optimal when variation should be ignored and when it should be preserved as a predictive signal.

For low-resource Indic NER, SentencePiece outperformed BPE in zero-shot transfer to Assamese, Oriya/Odia, Santali, Manipuri, Marathi, and Sindhi, although BPE was slightly better on Hindi fine-tuning and SentencePiece did not solve the absolute difficulty of Sindhi (Pattnayak et al., 23 Apr 2025). In this context, SentencePiece’s flexibility and reduced dependence on source-language frequency patterns were associated with improved entity consistency.

4. Statistical and formal foundations

A tokenizer can be modeled as a pair of stochastic maps:

$\Delta$6

where $\Delta$7 encodes strings into token sequences and $\Delta$8 decodes token sequences into strings. If $\Delta$9 is the true character-level distribution, the induced token distribution is

$x\in\Sigma^*$0

A token-level estimator $x\in\Sigma^*$1 decoded through $x\in\Sigma^*$2 converges to $x\in\Sigma^*$3. Consequently, it estimates the original distribution if and only if

$x\in\Sigma^*$4

This is the fundamental consistency condition proposed in “The Foundations of Tokenization: Statistical and Computational Concerns” (Gastaldi et al., 2024). Exactness is the stronger condition

$x\in\Sigma^*$5

which guarantees preservation for every distribution. Tokenizer inconsistency is distinct from estimator inconsistency: a token model may converge perfectly in token space while the encode–decode composition converges to the wrong character-level distribution.

Ambiguity arises when multiple token sequences decode to the same string. If the model assigns probability to token sequences outside the canonical tokenizer image, decoded probability must marginalize over all token preimages:

$x\in\Sigma^*$6

One-best tokenization is not generally equivalent to this marginalization. Empty-output tokens can create infinitely many preimages, whereas multiplicativity and a trivial decoder kernel guarantee that each fixed text has finitely many token preimages, although their number may still grow exponentially.

Finite-state methods provide a complementary formal perspective. “Tokenization as Finite-State Transduction” constructs a transducer representing all vocabulary-compatible segmentations and then composes it with a character-level regular constraint (Cognetta et al., 2024). MaxMatch is implemented through trie states, failure transitions, and output pops. BPE is represented by an ordered sequence of finite-state merge gadgets. The resulting construction promotes regular character-level constraints into token-level constraints while enforcing canonical tokenization.

This framework supports guided generation. A regular expression or finite automaton over characters can be composed with a tokenizer transducer and projected onto token sequences. The resulting token-level automaton constrains generation both by the surface-language condition and by the tokenizer’s canonical segmentation. The approach is especially relevant when a character-level constraint otherwise permits token histories that the model would not encounter during training.

A related formal-language result treats detokenization as a string homomorphism. If $x\in\Sigma^*$7 concatenates the string represented by each token, then the set of all tokenizations of a language $x\in\Sigma^*$8 is the inverse homomorphic image

$x\in\Sigma^*$9

Regular and context-free languages remain regular or context-free under this inverse-homomorphic construction, respectively (Geng et al., 2024). The result concerns all valid tokenizations; the language of one tokenizer’s selected canonical outputs may be a stricter subset whose formal properties are not automatically inherited.

5. Compression, vocabulary construction, and computational objectives

Compression is an important but incomplete description of tokenization. PathPiece explicitly minimizes the number of tokens in a document for a fixed vocabulary by solving a shortest-path problem. For a document of byte length $(t_1,\ldots,t_K)$0 and maximum token width $(t_1,\ldots,t_K)$1, its segmentation complexity is $(t_1,\ldots,t_K)$2 (Schmidt et al., 2024). Yet shortest token sequences did not consistently yield better language-model performance. Pre-tokenization and vocabulary initialization were more consequential than corpus token count alone, and BPE initialization improved PathPiece and SaGe despite their different final objectives.

A direct optimization formulation treats the vocabulary itself as the object of optimization. Given a corpus and a budget of at most $(t_1,\ldots,t_K)$3 non-singleton tokens, the objective is to minimize the total number of output tokens:

$(t_1,\ldots,t_K)$4

where $(t_1,\ldots,t_K)$5 is the singleton vocabulary (Lim et al., 8 Jan 2025). Equivalently, the objective maximizes weighted coverage of adjacent character positions. The exact optimization problem is NP-hard by reduction from Vertex Cover.

GreedTok selects arbitrary candidate substrings directly rather than constructing them through pairwise merges. Its relaxation to Weighted Maximum Coverage admits the classical $(t_1,\ldots,t_K)$6 approximation guarantee for GreedWMC, but that guarantee does not directly apply to GreedTok because the tokenization objective is neither generally submodular nor supermodular. On four corpora, GreedTok used approximately 3% fewer tokens per word than BPE on average and required approximately 13% fewer learned tokens to reach the same target compression (Lim et al., 8 Jan 2025). These are corpus-compression results, not controlled evidence of improved language-model quality.

Evaluation metrics themselves can encode tokenizer preferences. Bits-per-byte is decomposed as

$(t_1,\ldots,t_K)$7

where $(t_1,\ldots,t_K)$8 is tokens per byte and $(t_1,\ldots,t_K)$9 is loss per token (Schaeffer et al., 3 Nov 2026). BPB therefore does not make tokenizers neutral: an untrained uniform model can prefer some vocabulary sizes because the tokenizer has already incorporated corpus statistics. Lower BPB also does not necessarily imply better downstream performance. On trained models, correlations between BPB and downstream performance were weak for BLiMP, MBPP, HumanEval, and GSM8K.

The same issue affects other purportedly tokenizer-agnostic metrics. Unigram-normalized loss subtracts a tokenizer-specific unigram baseline, and its apparent agreement with BPB can result from confounding factors. The broader implication is that every evaluation metric answers a particular question: compression, predictive modeling, exact-string likelihood, answer discrimination, or task performance. No single metric is neutral with respect to all of these objectives.

6. Tokenization as a computational and systems constraint

Tokenization determines not only what information is represented but also what computations are practically accessible. In symbolic tasks such as counting, sorting, reversal, arithmetic, and pattern matching, the relevant units may be individual characters or digits. BPE can merge these units into opaque multi-character tokens. “Tokenization Constraints in LLMs: A Study of Symbolic and Arithmetic Reasoning Limits” introduces Token Awareness as a criterion for whether a token representation exposes a task-relevant property (Zhang et al., 20 May 2025).

The study reports large differences between raw BPE strings and atomically aligned formats. For GPT-4o-mini counting, pure-BPE Chain-of-Thought accuracy on 30–40-character inputs was 2.0% for one target and 2.7% for another, whereas precise-item formats reached 56.1% and 70.8%. List formatting also improved sorting and reversal. These experiments associate token-boundary alignment with symbolic performance, although delimiters and list formats may change prompting, visual structure, sequence length, and task interpretation in addition to tokenization.

Tokenization can also become a serving bottleneck. In agentic workloads with long, mostly cached contexts, prefix caching reduces model prefill but does not normally eliminate full tokenization before the cache can be consulted. For a 130K-token Llama 3.1 prompt, tokenization accounted for approximately 67% of time-to-first-token after a prefix-cache hit (Siavashi et al., 9 Oct 2026). GraphTok moves normalization-related frontend operations, pre-tokenization, BPE merging, and token-ID assembly to a replayable CUDA graph while preserving reference token IDs bit-for-bit for supported GPT-2, Llama 3, Qwen 2.5, DeepSeek-V3, and Gemma 3 configurations.

GraphTok reports 1.7–3.2 ms to encode one million GPT-2 tokens including transfers and is 4.7–6.0 times faster than gigatoken on Llama 3. In vLLM with cached prefixes, it lowers mean TTFT by 2.1–2.4 times. In a simulated NVIDIA Dynamo frontend, it sustains 518 requests/s of 121K-token traffic within a 2-second P99 target, compared with 296 requests/s for gigatoken and 18.5 for the default tokenizer. The results are specific to evaluated configurations and hardware, but demonstrate that exact tokenization can be a production systems workload rather than negligible CPU preprocessing.

Tokenization also affects resource allocation. Larger vocabularies increase embedding and output-layer parameters, while finer segmentation increases sequence length and attention cost. In Turkish language-model experiments, vocabulary-size effects differed by tokenizer: BPE and WordPiece saturated at moderate sizes, whereas morphological and word-level tokenizers benefited more from expansion (Toraman et al., 2022). In language variation experiments, 32K vocabularies were strongest for robustness tasks, while 64K was strongest for sensitivity tasks (Wegmann et al., 21 Feb 2025). In symbolic reasoning, atomically aligned representations improved exact computation at the cost of longer sequences (Zhang et al., 20 May 2025).

These results support a task- and language-dependent view of tokenizer design. A tokenizer for information retrieval may need special-sequence protection; a translation tokenizer may need language-specific morphology; an NER tokenizer may need entity stability and cross-script generalization; an LLM tokenizer may need a balance among vocabulary cost, sequence length, semantic units, and multilingual coverage; and an inference system may need exact, accelerated execution. No universal ordering of character, subword, word, morphological, or phrase-level units follows from compression alone.

7. Evaluation principles, limitations, and future directions

Tokenization research commonly evaluates compression, OOV rate, vocabulary size, sequence length, BLEU, TER, accuracy, F1, exact match, language-model loss, BPB, processing time, and downstream task performance. These measures are not interchangeable. A tokenizer may reduce token count while producing semantically fragmented units; preserve morphology while increasing OOV; improve NER transfer while increasing sequence length; or lower BPB without improving answer discrimination.

Evaluation should therefore include controlled comparisons in which model architecture, training data, optimization, vocabulary budget, and compute are held constant where possible. Translation experiments show that tokenizer choice can produce gains of up to approximately 12 BLEU points and 15 TER points, but also show strong source–target asymmetry (Domingo et al., 2018). Korean experiments demonstrate that one tokenizer can win translation while another wins extractive reading comprehension (Park et al., 2020). Turkish and English-variation studies show that vocabulary size and pre-tokenizer should be tuned jointly with task requirements (Toraman et al., 2022, Wegmann et al., 21 Feb 2025).

Several limitations recur across the literature:

• Morphological tokenizers depend on analyzer quality, dictionaries, allomorph mappings, and ambiguity resolution.
• BPE and WordPiece may create statistically convenient but linguistically heterogeneous units.
• Character-level methods provide coverage but may impose severe sequence-length costs.
• Special-token and whitespace conventions can invalidate simple homomorphism assumptions.
• Canonical segmentation differs from the set of all vocabulary-compatible segmentations.
• Tokenization metrics may be confounded with vocabulary size, corpus composition, model scale, and task difficulty.
• Small comparisons among models or tokenizers make correlations unstable.
• Observational tokenization penalties do not establish causal degradation.
• Exact probability evaluation may require marginalization over multiple tokenizations.
• Faster tokenization does not necessarily imply lower total cost, since CPU contention, GPU occupancy, transfers, and batching determine serving behavior.

Current research directions include morphology-aware and hybrid tokenizers, mixed-granularity vocabularies containing subwords, words, and multiword expressions, direct combinatorial vocabulary optimization, finite-state representations of canonical tokenization, task-aware evaluation, and hardware acceleration. The Less-is-Better proposal frames tokenization as a balance between token occurrences and vocabulary types, using Memorizer and Forgetter mechanisms to learn units of variable size (Yang, 2024). The partition-cover formulation removes the requirement that useful tokens be constructible through BPE-style pairwise merges (Lim et al., 8 Jan 2025). PathPiece shows that minimizing token count alone is insufficient (Schmidt et al., 2024). Formal work establishes consistency, finite-state realizability, and inverse-homomorphic preservation under specified assumptions (Gastaldi et al., 2024, Cognetta et al., 2024, Geng et al., 2024).

The general conclusion is that tokenization is a joint linguistic, statistical, computational, and systems design problem. Its effects arise from the interaction of vocabulary construction, boundary constraints, segmentation policy, model architecture, task structure, language variation, corpus distribution, and deployment hardware. Token count and vocabulary size remain important, but effective tokenization also requires information preservation, morphological and semantic alignment, canonical and lossless decoding, suitable statistical estimators, and implementation strategies that respect the exact tokenizer semantics.