Split Tokenization in NLP

Updated 6 July 2026

Split tokenization is the process of dividing raw text into discrete, linguistically meaningful tokens using a two-stage pipeline of pre-tokenization followed by subword segmentation.
It incorporates methods like WordPiece, BPE, and UnigramLM, and has evolved to include linguistically grounded and script-aware techniques for improved accuracy.
This approach optimizes token boundaries to reduce vocabulary size and sequence length, enhancing performance in tasks ranging from machine translation to speech and video processing.

Split tokenization is the segmentation of an input sequence into discrete units before modeling. In the mainstream NLP literature, it most often denotes a two-stage pipeline in which raw text is first pre-tokenized into typographic or word-like units and then decomposed into subwords such as WordPiece, BPE, or UnigramLM segments; in more recent work, the same label also covers linguistically constrained syllable or morphological segmentation, script-aware separation before compression, optimization-based token selection, and modality-specific split schemes for speech and video (Mielke et al., 2021, Song et al., 2020, Lim et al., 8 Jan 2025, Argaw et al., 6 Jul 2025). Across these usages, the central technical question is constant: where boundaries should be placed so that token sequences preserve useful structure while controlling vocabulary size, sequence length, and computational cost.

1. Conceptual scope and historical position

In the survey literature, split tokenization is aligned with deterministic pre-tokenization: whitespace splitting, punctuation separation, Unicode-category rules, and regex-based heuristics for items such as URLs, emails, hashtags, and mixed alphanumeric strings. This form of segmentation produces “typographic units” or “pre-tokens,” which may then be passed to a learned subword method or to a word-level model. The same literature contrasts these deterministic splits with learned segmenters such as BPE, WordPiece, and UnigramLM, as well as with character- and byte-level modeling, which remove the tokenization design problem by operating at maximal granularity (Mielke et al., 2021).

The comparative literature consistently rejects the idea of a universal best tokenizer. In neural machine translation, the best tokenizer differs by language pair, with MeCab and Stanford Word Segmenter excelling on Japanese and Chinese targets, and Moses performing best on several whitespace-delimited but morphologically rich languages such as German, Russian, and Arabic (Domingo et al., 2018). In Arabic text classification, none of the six evaluated tokenization techniques is the best choice overall; performance depends on dataset size, task type, and the amount of morphology in the data (Alyafeai et al., 2021). This task dependence is reinforced by later work on language variation, where tokenizers that are best for semantic robustness are not the same as those that are best for form-sensitive tasks (Wegmann et al., 21 Feb 2025).

2. Canonical split tokenization: pre-tokenization plus WordPiece

A particularly influential formulation defines split tokenization as the composition of pre-tokenization and WordPiece subword segmentation. Pre-tokenization splits raw text on whitespace and punctuation into coarse tokens, typically words. WordPiece then applies longest-match-first, or MaxMatch: at each position, it emits the longest vocabulary token that is a prefix of the remaining string, continuing until the word is fully covered or returning <unk> if no valid prefix exists. In the formalization used for Fast WordPiece Tokenization, the vocabulary contains a suffix indicator $\sigma$ for tokens that may only start inside a word, and classic end-to-end tokenization simply applies WordPiece independently to each pre-tokenized word (Song et al., 2020).

Before linear-time methods, the best-known WordPiece algorithms had complexity $O(n^2)$ or $O(nm)$ for a word of length $n$ and maximum token length $m$ . Fast WordPiece Tokenization replaces rescanning with a trie augmented by failure links $f(v)$ and failure pops $F(v)$ , producing LinMaxMatch, which is strictly $O(n)$ for a single word. The same work merges pre-tokenization and WordPiece into a single streaming pass over general text by recognizing punctuation directly at the root and treating punctuation as stand-alone tokens. The result preserves original WordPiece outputs while eliminating vocabulary-dependent multiplicative factors during tokenization; precomputation is $O(M)$ for trie and failure links, with naive storage of all failure pops costing $O(Mm)$ time and space offline (Song et al., 2020).

Setting	Earlier methods	LinMaxMatch
Single-word WordPiece	$O(n^2)$ 0 or $O(n^2)$ 1	$O(n^2)$ 2
End-to-end text tokenization	Pre-tokenize, then tokenize each word	Single-pass $O(n^2)$ 3
Precomputation	Not the main bottleneck discussed	Trie + failure links $O(n^2)$ 4; naive failure pops $O(n^2)$ 5

The empirical results show that the single-pass method is 8.2x faster than HuggingFace Tokenizers and 5.1x faster than TensorFlow Text on average for general text tokenization, while single-word tokenization is about $O(n^2)$ 6 faster on average and improves further on longer words (Song et al., 2020).

3. Linguistically grounded split boundaries

A major line of work treats split tokenization not as purely statistical segmentation but as a way to expose linguistically meaningful units before or instead of standard subword learning. In Swahili, syllable tokenization is defined over a fixed inventory of 219 syllables with a maximal-onset, single-vowel-nucleus rule. In Hindi and Marathi, MorphTok introduces morphology-aware pre-tokenization, including sandhi splitting, before BPE, and adds Constrained BPE to keep dependent vowels attached to their bases. In Turkish, tokenization is compared across character-, subword-, word-, and morphological-level granularity, with a Zemberek-based analyzer producing stem-and-suffix units (Atuhurra et al., 2024, Brahma et al., 14 Apr 2025, Toraman et al., 2022).

Approach	Languages	Primary unit
Syllable tokenization	Swahili	219 syllables
MorphTok + CBPE	Hindi, Marathi	Stems, affixes, sandhi-resolved components
Morphological-level tokenization	Turkish	Stems and morphemes

For Swahili, the syllable tokenizer replaces GPT-2’s default byte-level BPE with a syllable vocabulary of size $O(n^2)$ 7. The embedding table drops from approximately $O(n^2)$ 8 for the default 50,257-token GPT-2 vocabulary to approximately $O(n^2)$ 9 for syllable embeddings in FP32, while human evaluation improves over both BPE and WordPiece on structure, meaning, word order, and fluency. The reported averages are 4.60, 4.59, 4.63, and 4.76 for the syllable tokenizer, compared with 4.49, 4.53, 4.52, and 4.69 for WordPiece, and 3.27, 3.43, 3.28, and 3.46 for BPE (Atuhurra et al., 2024).

MorphTok makes the same general point for Indic morphology. Its lookup-based and model-driven pre-tokenizers explicitly split stems, affixes, and sandhi-resolved compounds before BPE. CBPE then modifies initialization so that dependent vowels remain attached to their preceding bases, avoiding tokens that begin with a dependent vowel. On Hindi IN22-Gen, fertility at 8k merges drops from 1.2708 for BPE to 1.2495 for CBPE, an approximately 1.68% reduction. Human EvalTok scores rise from 2.56 for BPE to 3.16 for Lookup+Pre-tokenization+BPE, and language-model perplexity improves from approximately 8.45 for BPE to 7.24 for Lookup+CBPE (Brahma et al., 14 Apr 2025).

The Turkish study reaches a more qualified conclusion. WordPiece and BPE remain strong defaults, but the morphological tokenizer is competitive, especially as vocabulary size grows. The paper further proposes an empirical rule for medium-sized models: the ratio of vocabulary parameters to total parameters can be chosen as 20% for de facto tokenizers such as BPE and WordPiece and 40% for other tokenizers, including morphological- and word-level schemes (Toraman et al., 2022).

4. Script-aware and nonconcatenative split tokenization

For scripts in which orthographic units are not well approximated by Unicode code points or byte sequences, split tokenization becomes a structural constraint. The WWHO architecture separates tokenization into three layers: a router that identifies script regions, a DFA-based syllabifier called LinguisTrie that emits atomic orthographic syllables, and SGPE, a syllable-aware grapheme pair encoding stage that compresses only after syllable boundaries are fixed. The system is designed for complex Abugida scripts such as Sinhala and Devanagari and guarantees that no valid syllable is ever split across multiple tokens, a property stated as the Linguistic Zero-Breakage Guarantee (Darshana, 26 Mar 2026).

The quantitative effects are large. On the test set reported for WWHO, SGPE achieves a Token to Word Ratio of 1.274 for Sinhala and 1.181 for Hindi. Relative to OpenAI’s o200k base tokenizer, token counts fall by 61.7% for Sinhala and 27.0% for Hindi; on the mixed-script Sinhala–Devanagari–English dataset, overall TWR is 1.240, corresponding to a 36.7% reduction versus o200k base. The reported context-window extension reaches up to $O(nm)$ 0 for Sinhala relative to DeepSeek V3 (Darshana, 26 Mar 2026).

A related but distinct strategy is SPLINTER, which targets nonconcatenative morphology by relinearizing words before standard tokenization. SPLINTER iteratively deletes template characters, recording each deletion as a composite symbol of the form “position:character,” until only the root letters remain contiguously. This makes root material linearly accessible to BPE or UnigramLM without requiring a morphological analyzer. Intrinsically, SPLINTER improves cognitive plausibility scores for Hebrew under both BPE and UnigramLM, though it also increases tokens per word and the proportion of one-character tokens. Downstream, the Hebrew BERT experiments report gains in question answering from 72.9 to 74.4 F1 and from 63.6 to 65.4 exact match, while prefix segmentation accuracy rises from 99.1 to 99.3 (Gazit et al., 18 Mar 2025).

5. Robustness, fairness, and reasoning under split-induced variation

Recent work treats split tokenization not only as a design choice but as a measurable source of model failure. In the study of natural-word fragmentation in LLMs, a “split word” is a natural word that becomes multiple subword tokens, and four penalty functions are proposed to quantify the effect: Token Anomaly Score, Under-trained Token similarity, Pairwise Distance, and Contextual Penalty. Across 28 dataset–LLM settings, the contextual penalty aggregated by top-3 words is the strongest predictor of error, reaching significance in 20/28 settings at the 10% level and 17/28 at the 5% level. The same study reports that larger vocabularies produce fewer splits and lower penalties (Pawar et al., 26 Dec 2025).

The fairness literature shows an analogous failure mode for gender-diverse pronouns. Under standard BPE, binary pronouns are usually intact tokens, whereas neopronouns are overfragmented; the reported fertility is approximately 1.20 for binary pronouns and 1.87 for neopronouns. Pronoun Tokenization Parity repairs this by adding full-token neopronoun case forms to the tokenizer and resizing the embedding matrix. Combined with lexical-layer finetuning, this raises neopronoun accuracy from 14.1% to 58.4% and reduces adversarial injection error sharply in the reported experiments (Ovalle et al., 2023).

Arithmetic provides a different demonstration of split sensitivity. In GPT-3.5 and GPT-4, numbers are ordinarily tokenized left-to-right in 3-digit chunks; enforcing right-to-left chunking at inference time by separators yields large improvements on addition. With 1 shot, reported accuracy rises from 68.5% under left-to-right chunking to 95.6% under right-to-left chunking; with 8 shots, it rises from 75.6% to 97.8%. The dominant left-to-right failure mode occurs when the sum has more digits than either addend, where accuracy collapses to 8.25% (Singh et al., 2024).

Variation across dialects and orthographies shows the same pattern at the corpus level. For zero-shot cross-lingual POS tagging on non-standardized varieties, the strongest predictor of target accuracy is the split word ratio difference

$O(nm)$ 1

where $O(nm)$ 2 is the proportion of words in dataset $O(nm)$ 3 that split into at least two subwords. Character-level noise injection on the source side often improves transfer by increasing source split rates toward target split rates; the paper reports many strong negative Spearman correlations between $O(nm)$ 4 and target accuracy, often between about $O(nm)$ 5 and $O(nm)$ 6 (Blaschke et al., 2023). A related study on English language variation reaches a compatible conclusion from another direction: the pre-tokenizer has the largest effect on downstream performance, with GPT-2-style pre-tokenization best on robust semantic tasks and larger vocabularies around 64k best on form-sensitive tasks (Wegmann et al., 21 Feb 2025).

6. Optimization, compression, and systems-level formulations

Another branch of the literature treats split tokenization as an explicit optimization problem. In the partition-cover formulation, a tokenizer selects a vocabulary so that each word is partitioned into contiguous substrings from that vocabulary plus singleton base tokens. The objective is to minimize total tokens, equivalently to maximize a cover score over adjacent singleton pairs. The decision version is NP-hard via reduction from vertex cover, and the problem relaxes naturally to weighted maximum coverage. The practical algorithm GreedTok greedily selects substrings by marginal cover gain and, on large corpora, uses on average approximately 3% fewer tokens per word than BPE at comparable vocabulary sizes; conversely, BPE requires on average approximately 13% more tokens than GreedTok to reach the same tokens-per-word target (Lim et al., 8 Jan 2025).

ToaST goes further by decoupling segmentation structure from vocabulary selection. It first builds full binary split trees over pretokens using byte $O(nm)$ 7-gram counts, independent of any vocabulary. Inference then recursively descends each tree and emits the first in-vocabulary node reached on each path. Vocabulary selection is cast as an integer program minimizing total token count under this inference procedure; the LP relaxation is near-integral in practice, with a worst reported relative gap of $O(nm)$ 8. On English text, ToaST reduces token counts by more than 11% compared to BPE, WordPiece, and UnigramLM at vocabulary sizes of 40,960 and above. In 1.5B-parameter language-model experiments, it achieves the highest CORE score, 0.2632±0.0076, outperforming BPE by 7.6%, UnigramLM by 5.3%, and WordPiece by 2.6%, while scoring best on 13 of 22 individual tasks (Schmidt et al., 21 May 2026).

At the systems level, split tokenization also becomes a parallelization problem. LoPT divides a long string into overlapping chunks, tokenizes them concurrently, and then merges the results using character-position-based overlap matching rather than token-ID matching. Dynamic chunk-length adjustment ensures sufficient overlap to recover boundary-spanning merges. For BPE and WordPiece tokenizers, the framework guarantees output identical to standard sequential tokenization. Across long-text datasets in English and Chinese, LoPT achieves accuracy 1.0 and reported speedups ranging from about 3.5x to 5.8x over standard fast tokenizers, depending on dataset and model family (Shao et al., 7 Nov 2025).

7. Extensions beyond text

Outside text NLP, “split tokenization” is sometimes redefined for other discrete representation problems. In video generation, channel-split quantization increases representational capacity without increasing the total number of autoregressive tokens. The latent channels are partitioned into $O(nm)$ 9 groups, each group is quantized independently, and the encoder’s spatiotemporal compression is increased by the same factor so that overall sequence length stays constant. The paper formalizes a packing function whose effective capacity exceeds $n$ 0 for per-group budget $n$ 1, while preserving the token count seen by the generator. In reconstruction, the best reported model, a Mamba-based tokenizer with CS-FSQ, reaches 32.54 dB PSNR, SSIM 0.747, and LPIPS 0.151 on Xiph-2K, and 32.36 dB, 0.691, and 0.206 on DAVIS; in VideoGPT-based generation it achieves FVD 55.4 on SkyTimelapse and 266.2 on UCF-101 (Argaw et al., 6 Jul 2025).

In speech, TokenSplit performs source separation directly in the token domain. It uses SoundStream acoustic tokens, discretized w2v-BERT semantic tokens, and character-level transcript tokens inside a single encoder–decoder Transformer, with masking used to switch among separation, transcript-conditioned separation, recognition, and text-to-speech generation. The tokenization is “split” in the sense that the model emits ordered, source-specific discrete streams rather than waveform masks. Transcript conditioning substantially improves recognition-oriented metrics: on 3-second Libri2Mix segments, direct TokenSplit improves differential word error rate from 26.6% to 12.1% when the target transcript is provided, and TokenSplitRefine improves it from 17.2% to 10.9% (Erdogan et al., 2023).

Taken together, these strands define split tokenization as a broad technical domain rather than a single algorithm. It spans pre-tokenization and subword segmentation, language- and script-specific boundary constraints, explicit optimization over token inventories, systems techniques for exact parallel tokenization, and modality-specific decompositions in speech and video. The recurrent design tension is the same in every setting: splitting must expose the units that matter for modeling without destroying compression, efficiency, or downstream generalization.