Multilingual Tokenizers

• Multilingual tokenizers are vocabulary induction and subword segmentation algorithms that can process diverse languages, scripts, and morphologies.
• They utilize methods like BPE, Unigram LM, and WordPiece to optimize vocabulary allocation and ensure fairness across multiple language domains.
• Practical construction involves joint training, script-aware merging, and evaluation with metrics such as fertility, parity, and STRR to enhance LLM performance.

Multilingual tokenizers are vocabulary induction and subword segmentation algorithms designed to process and encode text from multiple languages—often across diverse scripts, morphologies, and domain contexts—into discrete token sequences for LLMs. They represent a foundational component in multilingual NLP systems, dictating model efficiency, representation fairness, and downstream transfer performance. This article critically reviews the underlying algorithmic principles, evaluation metrics, overlap trade-offs, and practical design considerations as developed in leading research, with special attention to technical depth and empirical findings.

1. Algorithmic Foundations: Subword Modeling in Multilingual Contexts

Multilingual tokenizers commonly build upon subword algorithms such as Byte-Pair Encoding (BPE), Unigram LLM (ULM), and WordPiece, with significant variants in pre-tokenization strategy, normalization, and vocabulary allocation.

• BPE (Byte-Pair Encoding) iteratively merges the most frequent adjacent symbol pairs in the corpus until the vocabulary size reaches a preset threshold. The merge sequence encodes corpus frequency information, making it possible to recover the underlying language/domain data proportions from the merge list itself (Hayase et al., 23 Jul 2024). The segmentation objective is to minimize token count (maximize compression) rather than any explicit ML objective.

$f = \frac{\text{total tokens}}{\text{total words}}$

• Unigram LM (ULM) is a probabilistic model that generates segmentations of words into subwords, then prunes the vocabulary using EM to maximize log-likelihood over all possible segmentations (Karthika et al., 21 Jun 2025).

$$L(\theta) = \sum_{x \in D} \log \left( \sum_{s \in \mathcal{S}(x)} P_\theta(s) \right)$$

• Vocabulary size and script coverage are typically increased in multilingual setups, with empirically supported factors of 3-8× compared to monolingual English models to maintain comparable fertility and OOV/UNK rates (Ali et al., 2023, Kiulian et al., 24 Oct 2024).

Algorithmic extensions include mixture-of-experts routing for script-aware subword allocation (SUTRA (Tamang et al., 19 Nov 2024)), two-stage curricula combining subword and superword segmentation for morphologically complex languages (IndicSuperTokenizer (Rana et al., 5 Nov 2025)), and entirely vocabulary-free neural models that learn segmentation end-to-end via differentiable BiLSTM architectures (Islam et al., 2022).

2. Formal Evaluation Metrics for Multilingual Tokenization

Tokenizer quality is measured via both corpus-level and vocabulary-level statistics:

• Fertility: Average tokens per word; lower is better for efficiency.
• Parity: Ratio of token sequence length between language pairs; parity ≈ 1 indicates fairness (Petrov et al., 2023, Ali et al., 2023).
• Normalized Sequence Length (NSL):

$$\text{NSL}(T_\lambda, T_\beta) = \frac{\sum_i \text{len}(T_\lambda(D_i))}{\sum_i \text{len}(T_\beta(D_i))}$$

Captures the compression gain or penalty relative to a baseline tokenizer (Tamang et al., 28 Sep 2024, Tamang et al., 19 Nov 2024).

• Single Token Retention Rate (STRR): Proportion of words in a reference list encoded as a single token, measuring vocabulary allocation to high-frequency words and cross-lingual fairness (Nayeem et al., 11 Oct 2025).
• UNK Rate, Closeness to Character-level: Early-warning for poor tokenization (UNK > 3.7%, Closeness > 0.87 signals performance degradation) (Zhang et al., 2022).
• Zipfian metrics (Cardinality, Power-law Deviation ε, AUC, Slope β₁): Capture rank-frequency properties of token distributions for intrinsic evaluation and correlate with downstream translation accuracy (Lotz et al., 3 Jun 2025).
• Script and Language Coverage, Core-Token Ratio: Frameworks such as Qtok define script- and class-specific metrics, compute coverage across up to 430k unified tokens, and reveal that non-Latin scripts are generally underrepresented in most production tokenizers (Chelombitko et al., 16 Oct 2024).

Extrinsic metrics assess downstream effects: perplexity, F1/accuracy on classification/translation tasks, throughput (OTPT), and time-to-first-token (TTFT).

3. Vocabulary Allocation, Overlap, and Cross-Lingual Transfer

Vocabulary allocation—the number and rank of tokens assigned per language—directly affects segmentation granularity, representation equity, and downstream performance. Vocabulary overlap—the sharing of token types across languages—enables or interferes with cross-lingual transfer depending on the semantic alignment of the shared tokens (Kallini et al., 23 Sep 2025, Limisiewicz et al., 2023).

• Overlap Ratio:

$$\text{overlap\_ratio} = \frac{|V_1 \cap V_2|}{|V_1 \cup V_2|}$$

Modest overlap (IoU ≈ 0.1–0.2) yields >30-point gains in zero-shot transfer as measured on XNLI/XQuAD; full or high-similarity overlap matches performance at lower vocabulary size (Kallini et al., 23 Sep 2025).

• Semantic Filtering: Sharing semantically unrelated tokens ("false friends") can distort hidden representation spaces and degrade transfer, especially across typologically distant languages.
• Allocation-Fairness Tradeoff: High allocation (long tokens per language) is optimal for word-level tasks (POS, dependency), but high overlap is optimal for sentence-level/NLI/NER tasks. Designers tune this tradeoff using allocation (CPT, ARI) and overlap (JSD) metrics to match their application needs (Limisiewicz et al., 2023).

Parallel tokenizer frameworks enforce direct alignment of embedding indices for semantically equivalent words using bilingual dictionaries, leading to improved fertility and transfer, especially in low-resource settings (Kautsar et al., 7 Oct 2025).

4. Practical Construction, Scaling, and Adaptation Strategies

Tokenizer Construction

• Joint or Clustered Training: Joint tokenizers are trained on corpus mixtures, possibly stratified by language family (e.g., Indic cluster-based approaches) (Karthika et al., 21 Jun 2025, Stollenwerk, 2023).
• Script- and Language-tagged Merging: Script-aware merges and separation of conceptual vs. surface tokens reduce fragmentation for complex scripts (Mixture-of-Experts setups, e.g. SUTRA (Tamang et al., 19 Nov 2024)).
• Universal Tokenizers: Pretraining on an expanded set of languages (including those unseen during primary model pretraining) materially improves adaptation, with gains of up to 20 points in LLM-judge win rate, and enhances plasticity even for completely unseen scripts (Abagyan et al., 12 Jun 2025).
• Vocabulary Expansion and Merging: Carefully merging monolingual vocabularies (e.g., via copying English tokens, reusing IDs, and topping up with monolingual frequent types) improves coverage for underrepresented languages while maintaining English performance (Kiulian et al., 24 Oct 2024).

Scaling Laws

• Optimal vocabulary size increases sublinearly with the number of languages. For 5 European languages, |V|≈100k; for Indic tokenizers, 180–200k is common (Ali et al., 2023, Rana et al., 5 Nov 2025).
• Gains in efficiency (fertility, NSL) plateau past certain data (10 GB) or vocabulary thresholds (Rana et al., 5 Nov 2025).
• Tokenizer selection is most critical for languages that are underrepresented in training or morphologically divergent (Tamang et al., 19 Nov 2024, Tamang et al., 28 Sep 2024).

Adaptation and Robustness

• Adaptation to new languages is best accomplished via a universal tokenizer plus continued pretraining in the new language, rather than attempting post hoc vocabulary swaps (Abagyan et al., 12 Jun 2025).
• Vocabulary-free or neural tokenization (BiLSTM over character sequences) is robust to adversarial noise, misspellings, and code-switching, improving downstream NLI and sentiment accuracy in low-resource regimes (Islam et al., 2022).
• Balanced temperature sampling in data stream selection is less critical for the tokenizer than for model training; extremely skewed corpora mainly harm the rare script languages once severe imbalance is reached (Zhang et al., 2022).

5. Fairness, Bias, and Cost Disparities

Intrinsic unfairness arises when tokenizers allocate a disproportionate share of their vocabulary to high-resource or dominant-script languages (e.g., English, Chinese), penalizing low-resource communities with higher average sequence lengths, cost per token, latency, and context compression (Petrov et al., 2023, Tamang et al., 19 Nov 2024).

• Tokenization Premium:

$$p_L = \mathbb{E}_s \left[ \frac{|t(S_L)|}{|t(S_{\text{en}})|} \right]$$

Premiums for some minority languages can reach ⨉15 over English, directly translating to higher financial and computational costs (Petrov et al., 2023).

• STRR and NSL reveal chronic under-allocation for languages such as Hindi (STRR ≈30–40%) and Dravidian/Indic scripts (NSL ≫ 1 vs. English baseline) (Nayeem et al., 11 Oct 2025, Tamang et al., 19 Nov 2024).
• Best Practices for fairness include explicit per-language pretokenization, allocation of "core vocabulary" (top ~2,000 words) as single tokens, script-balancing in vocabulary sampling, and iterative evaluation of parity in NSL/fertility/premium metrics (Tamang et al., 19 Nov 2024, Nayeem et al., 11 Oct 2025, Kiulian et al., 24 Oct 2024).

Notable methods for enhancing fairness involve explicit objectives to minimize length disparity variance during vocabulary merging and corpus-wide coverage validation on diverse parallel datasets (Petrov et al., 2023).

6. Diagnostic, Benchmarking, and Transparency Tools

Frameworks such as Qtok (Chelombitko et al., 16 Oct 2024) propose an extensive suite of diagnostic metrics, including:

Metric	Purpose	Typical Range/Interpretation
STRR (Single-Token Rate)	Type-level fairness, word bias	98-100% (EN), 30-40% (Hindi)
NSL (Norm. Seq. Length)	Relative sequence length	<1 (better than baseline), >1 worse
Core-Token Ratio (CTR)	Completeness across group	29–59% (group-level)
Script-Coverage (SCₐ)	Unicode script representation	Biased to Latin in most tokenizers
Overlap Ratio	Cross-lingual sharing	0 (none), 0.1–0.9 (scalable)

Open-source benchmarks (e.g., FLORES-200) and "data mixture inference" attacks (Hayase et al., 23 Jul 2024) provide empirical transparency, revealing the actual proportion of languages and domains in commercial tokenizer training.

7. Future Directions and Open Challenges

• Comprehensive Vocabulary Coverage: Evidence suggests robust multilingual coverage may require vocabularies exceeding 300–400k tokens (for 300+ languages) (Chelombitko et al., 16 Oct 2024).
• Dynamic Vocabulary and Morphological Informativity: Ongoing research targets the inclusion of morphological and named-entity awareness into merge rules, especially for agglutinative or polysynthetic languages (Rana et al., 5 Nov 2025).
• Sampling and Data Transparency: Auditability of tokenizer, pretraining corpora, and merge logs is critical for detecting bias, overrepresentation, or intellectual property infractions (Hayase et al., 23 Jul 2024).
• Application-Specific Trade-offs: Word-level vs. sentence-level task optimization requires design-time selection of allocation and overlap strategies (Limisiewicz et al., 2023).
• Domain and Script Adaptation: Universal tokenizers exhibit greater "plasticity" but may still penalize low-resource or unseen scripts without careful byte-level fallback and byte/character coverage (Abagyan et al., 12 Jun 2025).

• (Petrov et al., 2023) LLM Tokenizers Introduce Unfairness Between Languages
• (Ali et al., 2023) Tokenizer Choice For LLM Training: Negligible or Crucial?
• (Lotz et al., 3 Jun 2025) Beyond Text Compression: Evaluating Tokenizers Across Scales
• (Rana et al., 5 Nov 2025) IndicSuperTokenizer: An Optimized Tokenizer for Indic Multilingual LLMs
• (Tamang et al., 19 Nov 2024) Evaluating Tokenizer Performance of LLMs Across Official Indian Languages
• (Kallini et al., 23 Sep 2025) False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual LLMs
• (Stollenwerk, 2023) Training and Evaluation of a Multilingual Tokenizer for GPT-SW3
• (Kiulian et al., 24 Oct 2024) From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages
• (Chelombitko et al., 16 Oct 2024) Qtok: A Comprehensive Framework for Evaluating Multilingual Tokenizer Quality in LLMs
• (Tamang et al., 28 Sep 2024) Performance Evaluation of Tokenizers in LLMs for the Assamese Language
• (Zhang et al., 2022) How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training?
• (Islam et al., 2022) A Vocabulary-Free Multilingual Neural Tokenizer for End-to-End Task Learning
• (Limisiewicz et al., 2023) Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages
• (Abagyan et al., 12 Jun 2025) One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers
• (Karthika et al., 21 Jun 2025) Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights
• (Nayeem et al., 11 Oct 2025) Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation
• (Kautsar et al., 7 Oct 2025) Parallel Tokenizers: Rethinking Vocabulary Design for Cross-Lingual Transfer
• (Hayase et al., 23 Jul 2024) Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?

---

Careful algorithmic design, coverage-aware vocabulary allocation, and rigorous evaluation remain critical for advancing equitable, efficient, and broadly capable multilingual tokenization in state-of-the-art LLMs.