TokSuite: Benchmarking Tokenizer Effects on LMs

• TokSuite is a systematic framework that isolates, quantifies, and benchmarks tokenizer effects on language model behavior under diverse real-world perturbations.
• It compares fourteen transformer-based models differing only in tokenization, using a high-coverage benchmark across multiple languages and domains.
• Empirical findings reveal that robust tokenizers like TokenMonster minimize accuracy drops in noisy settings while balancing efficiency and cross-lingual performance.

TokSuite is a systematically designed framework for isolating, quantifying, and benchmarking the effects of tokenizer choice on LLM (LM) behavior. It comprises a suite of fourteen LMs—each using a distinct tokenizer but otherwise held constant in architecture, dataset, training budget, and initialization—and a high-coverage benchmark to assess LM robustness under realistic input perturbations. This dataset and methodology enables unconfounded attribution of behavioral differences to tokenization, providing a foundation for empirical and analytical study of tokenization, efficiency, and robustness in modern subword and byte-level tokenizers (Altıntaş et al., 23 Dec 2025).

1. Experimental Design and Model Suite

TokSuite compares fourteen transformer-based LMs, each with approximately one billion non-embedding parameters, sharing the Meta Lingua Llama-3.2-1B configuration. All models are pretrained for 100,000 steps on a 100 billion-token corpus comprising English (FineWeb-Edu, 40B) and balanced high-quality data in ZH, TR, IT, and FA (FineWeb-2 HQ, 15B each). A central methodological feature is the use of a shared "super vocabulary" for embedding initialization, ensuring that vocabulary overlaps across tokenizers reflect identical initial representations.

The tokenizers compared cover a wide range of algorithms and vocabulary sizes:

Tokenizer	Type/Algorithm	Vocabulary Size
ByT5	Byte-level (fixed 256 bytes)	259
TokenMonster	Ungreedy subword lookahead	32,000
Phi-3	BPE + byte fallback	32,064
GPT-2	BPE + byte fallback	50,257
Comma	BPE + byte fallback	64,000
mBERT	WordPiece + [UNK] fallback	110,000
Llama-3.2	BPE + byte fallback	128,256
Tekken	BPE + byte fallback	130,000
Qwen-3	BPE + byte fallback	151,646
GPT-4o	BPE + byte fallback	200,000
BLOOM	BPE + byte fallback	250,680
Aya	BPE + byte fallback	255,029
Gemma-2	Unigram LM + byte fallback	256,128
XGLM	Unigram LM + byte fallback	256,008

The core differences among the algorithms are: BPE (greedy symbol-pair merging), WordPiece (likelihood maximization), Unigram (candidate pruning by perplexity), TokenMonster (lookahead "ungreedy" merges), and byte-level (fixed, unlearned vocabulary).

2. Benchmarking Methodology and Evaluation Metrics

TokSuite introduces a perturbation-oriented benchmark comprising approximately 5,000 test cases distributed across five languages (EN, IT, TR, FA, ZH) and three domains (general, elementary math, STEM). Each canonical question is accompanied by systematically perturbed variants reflecting:

• Input medium and script transformation: romanizations, non-native keyboard layouts, orthographic standards
• Diacritics and accent variation
• Orthographic and grammatical errors: typos, homoglyphs, morphosyntactic errors
• Morphological challenges (strongly present in Turkish)
• Noise artifacts: zero-width chars, OCR errors, space omissions
• Informal registers and web-style input: abbreviations, colloquial forms, emojis
• Linguistic variety: code-switching, dialects, paraphrases
• STEM-specific perturbations: LaTeX, ASCII diagrams, chemical formulas
• Unicode styling effects: decorative, fullwidth, or transformed characters

Evaluation is based on canonical accuracy $(A_{\mathrm{can}})$, perturbed accuracy $(A_{p})$, and relative accuracy drop $(\Delta A)$, as well as per-example normalized log-likelihood. Additional intrinsic metrics include subword fertility $(SF)$, proportion of continued words $(PCW)$, and cross-lingual parity for parallel sentences.

3. Empirical Results and Comparative Analysis

Empirical results demonstrate marked differences in robustness and efficiency attributable to tokenizer choice:

• Multilingual robustness: TokenMonster achieves the lowest average $\Delta A$ (0.17), followed by ByT5 and Comma (0.22), with the highest vulnerability in Tekken (0.27). Unicode styling constitutes the most difficult perturbation category, yielding an average $\Delta A$ of 0.53.
• Noise sensitivity: Perturbation-induced accuracy losses are higher in non-English languages ($\Delta A = 0.22$) than in English ($\Delta A = 0.15$).
• STEM applicability: LaTeX and STEM formatting perturbations induce substantial accuracy drops (0.23 and 0.29, respectively).

Notable quantitative insights:

• TokenMonster delivers strong robustness (lowest $\Delta A$) despite being trained only on English data, attributed to its "ungreedy" subword segmentation.
• ByT5 is nearly immune to certain categories (grammatical/morphological errors, $\Delta A \approx 0$ in EN), but at the cost of significant inefficiency (high $SF_{EN}=4.40$, $SF_{FA}=7.72$, $PCW_{EN}=0.87$).
• Unigram tokenizers (Gemma-2, XGLM) demonstrate superior cross-lingual parity (average 1.18) and low fertility ($\approx$1.6), supporting balanced efficiency and robustness.
• Vocabulary size is not a strong predictor of robustness; for example, Qwen-3 (151k) and Gemma-2 (256k) underperform mBERT (110k) in some error categories.

4. Failure Modes and Analytical Characterization

TokSuite enables fine-grained failure analysis:

• Subword fragmentation: Minor orthographic errors (e.g., a typo in "doctor" $\rightarrow$ "doctro") can induce radical fragmentation in BPE-based tokenizers, mimicking out-of-vocabulary effects and causing up to 0.28 loss in perturbed accuracy.
• Agglutinative morphology: In Turkish, small surface changes (e.g., /t/ $\rightarrow$ /d/) reconfigure subword boundaries except for byte-level and TokenMonster, which maintain stable segmentation.
• LaTeX/STEM tokens: Changes in brace spacing disrupt token boundaries, often impeding model performance. Only TokenMonster and Qwen-3 exhibit resilience to these formatting perturbations.
• Unicode styling: NFKC-normalizing tokenizers (e.g., XGLM) limit the $\Delta A$ (≈0.09) but cannot recover stylized input; others experience much higher drops ($\Delta A > 0.50$).

Intrinsic analytical metrics formalized include subword distribution entropy

$H = - \sum_{t \in V} p(t) \log p(t)$

and expected token length per character

$$\bar{L} = \frac{1}{|C|}\sum_{c \in \text{chars}} |T(c)|$$

These metrics are practically predictive: higher $SF$ and $PCW$ correspond to greater brittleness under input perturbation.

5. Conclusions and Practical Recommendations

Empirical and analytical evidence from TokSuite leads to several primary conclusions:

• Tokenizer design exerts a dominant influence on robustness to real-world perturbations—vocabulary size and model scale are secondary.
• Byte-level (ByT5) and lookahead (TokenMonster) algorithms are most robust, at the tradeoff of less efficient (longer) token sequences.
• Unigram LM tokenizers provide a practical compromise between efficiency (low $SF$/PCW) and perturbation tolerance.
• Conventional BPE/WordPiece tokenizers exhibit high sensitivity to Unicode and morphological perturbations unless normalization or byte fallback is integrated.
• Scaling model size beyond the studied ∼1B parameter regime does not substantially increase tokenization robustness.

For production and research settings:

• When high tolerance to typos, domain inconsistencies, or multilingual noise is critical, employ byte-level or lookahead subword tokenizers despite increased sequence lengths.
• For balanced efficiency and robustness, select Unigram LM tokenizers with vocabulary sizes in the 200–300k range.
• Align normalization strategies—Unicode NFKC, whitespace treatment—with the target application’s text domain.
• For off-the-shelf LMs, mitigation methods such as Exact-Byte-Level probability conversion or token-healing should be considered to address inference-stage tokenization mismatches.
• Intrinsic tokenizer metrics (fertility, PCW, parity) in conjunction with real-world perturbation benchmarks should be used to assess candidate tokenizers prior to large-scale LM training.

TokSuite establishes a robust empirical and methodological baseline for future studies on tokenizer effects, supporting reproducible and unconfounded benchmarking of LM performance under diverse input scenarios (Altıntaş et al., 23 Dec 2025).