IndicSuperTokenizer: Optimized Indic Tokenization

• IndicSuperTokenizer is a specialized tokenizer for Indic multilingual LLMs that addresses script diversity, rich morphology, and large-scale efficiency.
• It utilizes a two-stage, curriculum-style BPE pipeline combining language-agnostic pre-tokenization and script-aware merging to optimize token-to-word compression.
• The system achieves remarkable compression and throughput gains, reducing token counts by up to 39.5% and improving inference speeds by 44% over comparable tokenizers.

IndicSuperTokenizer refers to a class of tokenizers specifically engineered for Indic multilingual LLMs, addressing challenges spanning script diversity, rich morphological structure, and efficiency for large-scale deployment. The major instantiations—"IndicSuperTokenizer" for 22+ Indian languages (Rana et al., 5 Nov 2025) and its drop-in variant BrahmicTokenizer-131K (Shravan, 28 May 2026), as well as its central use in Krutrim LLM (Kumar et al., 2024)—constitute the principal advances in this family. These systems establish state-of-the-art token-to-word compression (“fertility”), high linguistic alignment with Indic scripts, and robust trade-offs between multilingual compression and inference cost.

1. Architectural Principles and Algorithms

IndicSuperTokenizer (IST) utilizes a two-stage curriculum-style Byte Pair Encoding (BPE) pipeline that alternates language-agnostic and script-aware phases to optimize both subword and multi-word token formation (Rana et al., 5 Nov 2025). The procedure is:

• Stage 1 (Subword Learning): Text is first pre-tokenized using whitespace and a language-agnostic regex. BPE merges apply strictly within word boundaries, allowing only fine-grained, script-controlled subword units. Merging continues until an intermediate vocabulary threshold $t < |V|$ is achieved.
• Stage 2 (Superword Learning): Word boundary constraints are removed, BPE merges are allowed across word boundaries (but not across sentences), targeting multi-word tokens. Vocabulary expands from $t$ to $|V|$.

Pseudocode, defined in (Rana et al., 5 Nov 2025), formalizes these stages, ensuring reproducibility. Stage 1 and 2 are governed by a merge rule selection maximizing the frequency of adjacent symbol pairs: $(x^*, y^*) = \argmax_{(x, y)} \mathrm{freq}(x, y)$

Pre-tokenization incorporates comprehensive Unicode normalization (NFKC), with regex rules engineered per script (e.g., Devanagari vowel sign boundaries), and includes punctuation and digit separation. Although morphology-aware segmentation was explored (splitting root/affix using external analyzers), the final design prioritizes general regex-driven splitting (Rana et al., 5 Nov 2025).

BrahmicTokenizer-131K (Shravan, 28 May 2026), a prominent instance, is constructed through a two-stage retrofit of OpenAI’s o200k_base. Stage 1 prunes out nine non-Brahmic scripts; Stage 2 fills dead slots with corpus-trained, script-specific Brahmic tokens, governed by a linear-programming allocation over nine blocks, filtered to avoid cross-script merges.

Krutrim LLM (Kumar et al., 2024) uses a SentencePiece BPE tokenizer, with vocabulary optimized after eliminating 70% data redundancy using a multi-stage deduplication and language-aware text cleaning pipeline. The approach ensures maximal coverage of Indic word forms, minimize over-fragmentation of consonant clusters.

2. Vocabulary Construction and Allocation

IST adopts a large shared vocabulary (e.g., 200,000 tokens in (Rana et al., 5 Nov 2025)), distributed proportionally to the training corpus size for each script. The BPE algorithm uses UTF-8 byte fallback for an open-vocabulary design. In contrast, BrahmicTokenizer-131K (Shravan, 28 May 2026) restricts the vocabulary to 131,072 tokens, with an explicit script-prune-and-retrofit mechanism. Here, slots for Brahmic scripts (e.g., Devanagari, Bengali, Oriya) are allocated so as to maximize cumulative corpus token savings, computed via a discrete concave maximization: $$\max_{x_s \in \mathbb{Z}_{\ge 0}} \sum_{s \in S} c_s(x_s) \quad \text{s.t.} \sum_{s \in S} x_s = 2{,}215, \; 0 \le x_s \le K_s$$ where $c_s(k)$ gives corpus token-savings for script $s$ with $k$ slots.

In Krutrim LLM’s usage (Kumar et al., 2024), a 100,000-token vocabulary is sampled from stratified, cleaned corpora, tuning for 99.7% script-character coverage and disabling byte fallback to maintain subword consistency.

Example script-wise enhancements in BrahmicTokenizer-131K post-surgery:

Script	New Slots Added
Oriya (Odia)	663
Tamil	138
Devanagari	173
... (others)	...

3. Tokenization Efficiency and Fertility Metrics

Tokenization efficiency is measured via the fertility (token-to-word) ratio on multilingual benchmarks:

$$F = \frac{\text{total \# tokens}}{\text{total \# words}} = \frac{1}{N} \sum_{i=1}^N \frac{|\mathrm{tokens}(L_i)|}{|\mathrm{words}(L_i)|}$$

IST achieves substantial fertility improvements:

• LLaMA-4 tokenizer: $F_{\text{LLaMA4}} \approx 2.80$
• Sutra tokenizer: $t$0
• IST: $t$1 Reductions: 39.5% vs LLaMA4, 18% vs Sutra (Rana et al., 5 Nov 2025).

BrahmicTokenizer-131K produces, on a 2.84B-word/27M-doc Indic corpus, 26.7% fewer tokens than baseline Tekken/Sarvam-m at the same vocabulary size (Shravan, 28 May 2026), with language-specific savings as high as 76.79% on Odia (due to addition of 725 Oriya-block tokens, otherwise absent).

Krutrim’s IndicSuperTokenizer shows similar advances against OpenAI Tiktoken: for Hindi R=1.39 vs Tiktoken R=1.62, for Odia R=1.80 vs 6.39 (Kumar et al., 2024).

4. Benchmarking and Cross-Language Performance

Comprehensive evaluations span downstream and intrinsic metrics:

• Downstream tasks: English (zero/few-shot, e.g., HellaSwag, MMLU, GSM8K) and Indic (ARC Challenge, COPA, XNLI, Sentiment, etc.) accuracy are unaffected or slightly improved; for example, English accuracy matched LLaMA-4 (0.279), and Indic averaged slightly higher (0.394 vs. 0.388) in (Rana et al., 5 Nov 2025).
• Intrinsic metrics: IST exhibited optimal fertility and normalized sequence length in almost all benchmarked languages. BrahmicTokenizer-131K matches or exceeds o200k_base’s compression on English (1.235–1.232 tokens/word; 96.8% of FLORES sentences bit-identical), and achieves the best code and math compression among 131K-vocab tokenizers (Shravan, 28 May 2026).
• Compression and memory: The substantial token-count reductions achieved directly decrease memory footprint and training/inference cost.

5. Throughput and Practical Deployment

Adopting IST results in marked inference speed gains. In a controlled throughput study using LLaMA-3.2 1B models (identical except tokenizer), running on 8×H100 GPUs, IST delivered:

• Time-to-first-token: 18.98 ms (vs. 19.17 ms for LLaMA-4)
• Output tokens/sec: 169.42 (vs. 117.99)
• Relative throughput improvement: 44% (Rana et al., 5 Nov 2025)

Practical deployment is facilitated by preserving all surface interfaces: BrahmicTokenizer-131K is a strict drop-in for o200k_base—merge rules, pre-tokenization, decoder logic, JSON schema, and special tokens are unchanged except for the appended Brahmic tokens (Shravan, 28 May 2026). Transitioning involves only replacing the tokenizer.json and resizing the vocabulary embedding matrix.

Example usage in Python (HuggingFace Transformers): $t$2

6. Ablation Analyses and Robustness

IST’s design space is characterized by rigorous ablation studies (Rana et al., 5 Nov 2025):

• Data Size: Fertility gains plateau after 10GB in Stage 1.
• Transition Point: 90% of final vocabulary allocated in Stage 1 yields best results.
• Vocabulary size: No meaningful fertility improvement beyond 200K tokens.
• One-stage vs. Two-stage: Two-stage process achieves lower fertility (1.69 vs. 1.86).
• Regex Pre-tokenization: LLaMA-4’s regex rules improve fertility by 38–40% over GPT-2’s.
• Unicode Normalization Forms: Minimal differences; NFKC adopted.
• Glitch Tokens: IST’s multi-word tail truncates over-fragmented and under-trained tokens present in pure BPE schemes.

7. Implementation Notes and Release

IST is implemented using HuggingFace’s SuperBPE for maximum flexibility and reproducibility; SentencePiece’s priority-BPE and TikToken are referenced instruments for comparison and metrics (Rana et al., 5 Nov 2025). Preprocessing relies on platforms such as warcio and trafilatura for large-scale text cleaning, as in Krutrim LLM (Kumar et al., 2024). Downstream task evaluation adapts the lm-eval-harness.

BrahmicTokenizer-131K is released under Apache 2.0 at HuggingFace, with engineering smoke tests for verifying structural constraints (e.g., no cross-script merges, no tokens exceeding 32 UTF-8 bytes) (Shravan, 28 May 2026).

8. Context, Limitations, and Comparative Position

IST establishes a new state-of-the-art for Indic-focused tokenization, delivering up to 44% inference throughput gains and robust intrinsic metric improvements while incurring negligible trade-offs for English, European, and code content (Rana et al., 5 Nov 2025, Shravan, 28 May 2026). In contrast to monolithic or specialist Indic tokenizers such as Sarvam-1 or MUTANT-Indic, IST and BrahmicTokenizer-131K achieve strong Indic compression without sacrificing non-Indic utility. Existing specialist alternatives forgo code and EU coverage, and/or increase English fertility by up to 15.9% and code/math by 26–33%. A plausible implication is that the two-stage, cross-word, and regex-driven architectural approaches in IST represent a critical advance for tokenization in diverse, morphologically rich languages.

The artifact is considered directly usable and extensible for future LLM development targeting Indic and code-mixed corpora.